Code Correctness Evaluation on HE-CPP
66.3F1 ScoreCoCoA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 66.3 | 67.4 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 62.9 | 56.8 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 54.9 | 57.6 | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 53.4 | 54.6 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 47.3 | 57.8 | |
| BCEval Model=Llama-3.1-8B2024.10 | 46.5 | 54.9 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 44.9 | 60.6 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 42.5 | 56.6 | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 38.9 | 57.1 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 35.7 | 56.2 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 35.3 | 55.4 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 34.9 | 55.9 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 33.1 | 55.6 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 30 | 55.2 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 27.7 | 55.1 | |
| BCEval Model=GPT-4o-mini2024.10 | 26.7 | 55.6 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 24.5 | 53.2 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 23.4 | 54.7 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 22.2 | 54.9 | |
| 2PREval Model=GPT-4o-mini2024.10 | 19.8 | 54.9 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 18.1 | 54.1 | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 11.9 | 52.7 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 11.2 | 51.9 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 3.9 | 50.7 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 3.9 | 50.5 |