Multilingual Code Evaluation on HumanEval-X
95.45Python ScoreGPT-o1-mini
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| GPT-o1-miniApproach=System-22025.02 | 95.45 | 92.42 | 94.7 | 90.91 | 88.64 | 92.42 | |
| MCTS-JudgeApproach=System-2, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 90.15 | 85.61 | 84.09 | 84.85 | 81.06 | 85.15 | |
| MCTS-JudgeApproach=System-2, Base Model=GPT-4o-mini2025.02 | 87.88 | 86.36 | 88.64 | 88.64 | 85.61 | 87.43 | |
| MCTS-JudgeApproach=System-2, Base Model=Mistralai-Codestral-22B2025.02 | 87.78 | 75.76 | 77.27 | 73.48 | 75.76 | 78.01 | |
| CodeJudgeApproach=System-1, Base Model=GPT-4o-mini2025.02 | 87.12 | 83.33 | 87.88 | 86.36 | 84.09 | 85.76 | |
| CodeJudgeApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 86.36 | 81.06 | 79.55 | 82.58 | 75.75 | 81.06 | |
| VanillaApproach=System-1, Base Model=GPT-4o-mini2025.02 | 86.36 | 82.58 | 85.61 | 86.36 | 84.85 | 85.15 | |
| CodeJudgeApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 85.61 | 69.7 | 68.94 | 71.21 | 66.67 | 72.43 | |
| ICE-ScoreApproach=System-1, Base Model=GPT-4o-mini2025.02 | 84.85 | 78.79 | 86.36 | 83.33 | 85.61 | 83.79 | |
| GPT-o1-previewApproach=System-22025.02 | 82.58 | 89.39 | 87.12 | 86.36 | 83.33 | 85.76 | |
| VanillaApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 82.58 | 68.18 | 70.45 | 62.88 | 66.67 | 70.15 | |
| ICE-ScoreApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 82.58 | 68.18 | 60.61 | 63.64 | 61.36 | 67.27 | |
| MCTS-JudgeApproach=System-2, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 80.3 | 77.27 | 80.3 | 78.79 | 82.58 | 79.85 | |
| MCTS-JudgeApproach=System-2, Base Model=Llama-3.1-8B-Instruct2025.02 | 74.24 | 79.55 | 77.27 | 70.45 | 71.97 | 74.7 | |
| VanillaApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 73.48 | 64.39 | 63.64 | 66.67 | 60.61 | 65.76 | |
| CodeJudgeApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 73.48 | 69.7 | 67.42 | 69.7 | 66.67 | 69.39 | |
| CodeJudgeApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 73.48 | 73.48 | 75.76 | 70.45 | 67.42 | 72.12 | |
| Qwen-QwQ-32BApproach=System-22025.02 | 72.73 | 75 | 75 | 64.39 | 78.03 | 73.03 | |
| ICE-ScoreApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 72.73 | 74.24 | 71.97 | 72.73 | 78.79 | 74.09 | |
| ICE-ScoreApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 72.73 | 64.39 | 62.88 | 56.82 | 54.55 | 62.27 | |
| ICE-ScoreApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 71.21 | 76.52 | 69.7 | 70.45 | 74.24 | 72.42 | |
| VanillaApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 65.91 | 63.64 | 64.39 | 62.12 | 70.45 | 65.3 | |
| VanillaApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 62.12 | 64.39 | 68.94 | 64.39 | 73.48 | 66.66 | |
| GPT-4oApproach=System-12025.02 | 59.09 | 53.79 | 60.61 | 62.12 | 65.91 | 60.3 |