Code Correctness Evaluation on HE-JS
67.6F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 67.6 | 66.5 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 65.4 | 57.8 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 57 | 58.3 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 53.7 | 55.4 | |
| BCEval Model=Llama-3.1-8B2024.10 | 46.2 | 54.3 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 44 | 55.7 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 42.2 | 57.2 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 41.5 | 57.2 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 40.9 | 58.5 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 39.2 | 55.4 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 38.9 | 57 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 38.7 | 56.5 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 37.4 | 56.3 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 35.2 | 57.2 | |
| BCEval Model=GPT-4o-mini2024.10 | 35 | 58 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 32.1 | 57.6 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 31.6 | 54.8 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 30.7 | 54.8 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 27.2 | 55.9 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 25.5 | 55.7 | |
| 2PREval Model=GPT-4o-mini2024.10 | 24.2 | 55 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 18.5 | 53.9 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 15.6 | 52.8 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 12.6 | 51.7 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 5.1 | 51.1 |