Code Correctness Evaluation on DB
68.6F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 68.6 | 58.5 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 67.6 | 52.8 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 63 | 56.3 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 62.2 | 50 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 60.4 | 58.2 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 59.6 | 58.4 | |
| BCEval Model=Llama-3.1-8B2024.10 | 57.2 | 55.4 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 53 | 59.1 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 45.6 | 59.4 | |
| BCEval Model=GPT-4o-mini2024.10 | 45.5 | 62.2 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 44 | 62 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 40.1 | 59.1 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 38.8 | 60.3 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 33.8 | 56.7 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 31.8 | 53.9 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 29.4 | 55.7 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 28.1 | 55 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 27.7 | 52.5 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 27.5 | 55 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 25.8 | 55 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 20.2 | 52.7 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 15.9 | 54.3 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 13.2 | 52.4 | |
| 2PREval Model=GPT-4o-mini2024.10 | 13 | 53.2 |