Code Correctness Evaluation on HE-PY
72.6F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 72.6 | 71.5 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 67.2 | 58.4 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 59.5 | 58.2 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 57.8 | 58.7 | |
| BCEval Model=Llama-3.1-8B2024.10 | 55.2 | 60 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 48.4 | 60.5 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 47.8 | 58.7 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 47.4 | 58.9 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 47.3 | 58.7 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 46.4 | 61.2 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 44.6 | 58.4 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 43.9 | 58.4 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 43.5 | 61 | |
| BCEval Model=GPT-4o-mini2024.10 | 43.2 | 61.5 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 42.9 | 57.4 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 42.2 | 58.9 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 40.3 | 60.8 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 37.4 | 60 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 35.2 | 57.3 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 33.9 | 56.6 | |
| 2PREval Model=GPT-4o-mini2024.10 | 27.1 | 56.6 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 24.1 | 54.2 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 23.3 | 55.3 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 12.7 | 51.3 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 6 | 66.5 |