Code Correctness Evaluation on HE-JA
72.9F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 72.9 | 70.8 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 65.2 | 56.6 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 59.9 | 58.6 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 59.1 | 59.4 | |
| BCEval Model=Llama-3.1-8B2024.10 | 55.9 | 58.6 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 52.9 | 60.2 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 49.3 | 60.6 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 45.8 | 57.6 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 44.6 | 58.2 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 43.2 | 57.7 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 39.9 | 56.8 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 37.5 | 55.7 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 37.4 | 56.2 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 36.8 | 56 | |
| BCEval Model=GPT-4o-mini2024.10 | 36.6 | 59.4 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 35.1 | 59 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 35 | 58.8 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 34.5 | 58.3 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 31.9 | 55.1 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 30 | 57.6 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 29.6 | 53.7 | |
| 2PREval Model=GPT-4o-mini2024.10 | 24.6 | 55.6 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 21.6 | 51.7 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 16.6 | 52 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 8.4 | 49.2 |