Code Correctness Evaluation on HE-Go
68.2F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 68.2 | 66.4 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 64.7 | 54.5 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 55.4 | 55.7 | |
| BCEval Model=Llama-3.1-8B2024.10 | 54.2 | 58.3 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 51.7 | 54.8 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 50.6 | 60.8 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 48.3 | 57.7 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 45.8 | 58.8 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 45.8 | 55.9 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 45.7 | 56 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 45.2 | 57.7 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 42.2 | 56 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 39.9 | 55.4 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 38.4 | 58.4 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 37.4 | 55.1 | |
| BCEval Model=GPT-4o-mini2024.10 | 33.3 | 57.3 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 31.7 | 57.4 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 31.5 | 57.1 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 30.5 | 53.5 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 23.7 | 55.4 | |
| 2PREval Model=GPT-4o-mini2024.10 | 22.7 | 54.8 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 21.7 | 53.2 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 17.2 | 52.5 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 11.8 | 51.5 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 9.7 | 51.2 |