Code Correctness Evaluation on BCB
69.6F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 69.6 | 62.1 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 66.9 | 54.1 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 64.8 | 49.7 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 62.3 | 61.9 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 57.5 | 58 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 57.4 | 55.1 | |
| BCEval Model=Llama-3.1-8B2024.10 | 55.2 | 57.8 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 45.7 | 51.2 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 42.5 | 58 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 42 | 48.8 | |
| BCEval Model=GPT-4o-mini2024.10 | 39.1 | 56 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 35.6 | 47.1 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 35.5 | 51.2 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 33.3 | 54.4 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 32.5 | 47.8 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 27.9 | 43.9 | |
| 2PREval Model=GPT-4o-mini2024.10 | 24.3 | 54.4 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 20.8 | 44.4 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 19.9 | 49.5 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 18 | 50.3 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 17 | 33.7 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 16.6 | 50.3 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 12.4 | 37.6 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 12.3 | 49 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 4.9 | 47.1 |