Code Correctness Evaluation on APPS
80AccuracyMCTS-Judge
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| MCTS-JudgeApproach=System-2, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 80 | — | |
| MCTS-JudgeModel Size=16B, # Reasoning Tokens=2065+412, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 80 | — | |
| MCTS-JudgeApproach=System-2, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 79 | — | |
| GPT-o1-miniApproach=System-22025.02 | 78 | — | |
| o1-miniModel Size=~100B, # Reasoning Tokens=37552025.02 | 78 | — | |
| MCTS-JudgeApproach=System-2, Base Model=GPT-4o-mini2025.02 | 76 | — | |
| GPT-o1-previewApproach=System-22025.02 | 75 | — | |
| o1-previewModel Size=~300B, # Reasoning Tokens=56312025.02 | 75 | — | |
| MCTS-JudgeApproach=System-2, Base Model=Mistralai-Codestral-22B2025.02 | 72 | — | |
| ICE-ScoreApproach=System-1, Base Model=GPT-4o-mini2025.02 | 72 | — | |
| CodeJudgeApproach=System-1, Base Model=GPT-4o-mini2025.02 | 72 | — | |
| CodeJudgeApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 68 | — | |
| ICE-ScoreApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 65 | — | |
| VanillaApproach=System-1, Base Model=GPT-4o-mini2025.02 | 65 | — | |
| VanillaApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.02 | 62 | — | |
| CodeJudgeApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 62 | — | |
| VanillaApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 62 | — | |
| MCTS-JudgeApproach=System-2, Base Model=Llama-3.1-8B-Instruct2025.02 | 62 | — | |
| Qwen-QwQ-32BApproach=System-22025.02 | 60 | — | |
| Qwen-QwQ-32BModel Size=32B, # Reasoning Tokens=25592025.02 | 60 | — | |
| ICE-ScoreApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 56 | — | |
| VanillaApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 56 | — | |
| GPT-4oApproach=System-12025.02 | 55 | — | |
| CodeJudgeApproach=System-1, Base Model=Mistralai-Codestral-22B2025.02 | 54 | — | |
| CodeJudgeEval Model=GPT-4o-mini2024.10 | 53.2 | 18.6 | |
| BCEval Model=Llama-3.1-8B2024.10 | 53.2 | 61.1 | |
| CodeJudgeApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 53 | — | |
| CA-SPEval Model=Llama-3.1-8B2024.10 | 52.5 | 60.1 | |
| BCEval Model=GPT-4o-mini2024.10 | 51.4 | 13.4 | |
| ICE-ScoreEval Model=GPT-4o-mini2024.10 | 51.2 | 15.5 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Best2024.10 | 51.2 | 66.7 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.10 | 51.2 | 17.5 | |
| ICE-ScoreEval Model=Llama-3.1-8B2024.10 | 51.2 | 53.7 | |
| CodeJudgeEval Model=Qwen-2.5-7B2024.10 | 51.1 | 18.9 | |
| VanillaEval Model=GPT-4o-mini2024.10 | 50.9 | 5.5 | |
| 2PREval Model=Llama-3.1-8B2024.10 | 50.9 | 11 | |
| MCTS-JudgeEval Model=GPT-4o-mini2024.10 | 50.6 | 42.8 | |
| 2PREval Model=GPT-4o-mini2024.10 | 50.5 | 6.7 | |
| CA-SPEval Model=GPT-4o-mini2024.10 | 50.5 | 12.6 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.10 | 50.2 | 64.8 | |
| CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.10 | 50 | 44 | |
| VanillaEval Model=Qwen-2.5-7B2024.10 | 50 | 3.4 | |
| BCEval Model=Qwen-2.5-7B2024.10 | 49.6 | 1.4 | |
| CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.10 | 49.5 | 61.1 | |
| ICE-ScoreEval Model=Qwen-2.5-7B2024.10 | 49.1 | 13.9 | |
| VanillaEval Model=Llama-3.1-8B2024.10 | 49.1 | 28.6 | |
| 2PREval Model=Qwen-2.5-7B2024.10 | 48.2 | 6.5 | |
| CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.10 | 48 | 52.8 | |
| ICE-ScoreApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 48 | — | |
| CA-SPEval Model=Qwen-2.5-7B2024.10 | 47.9 | 14.6 | |
| CodeJudgeEval Model=Llama-3.1-8B2024.10 | 47.5 | 26.9 | |
| ICE-ScoreApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.02 | 42 | — | |
| VanillaApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.02 | 41 | — |