List your company
ResearchFormerly WizWand

Code Correctness Evaluation on APPS

80AccuracyMCTS-Judge
39.4449.9760.571.03Oct 4, 2024Oct 26, 2024Nov 18, 2024Dec 11, 2024Jan 3, 2025Jan 26, 2025Feb 18, 2025
Updated 5mo ago

Evaluation results

MethodLinks
MCTS-JudgeApproach=System-2, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.0280—
MCTS-JudgeModel Size=16B, # Reasoning Tokens=2065+412, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.0280—
MCTS-JudgeApproach=System-2, Base Model=Qwen2.5-Coder-14B-Instruct2025.0279—
GPT-o1-miniApproach=System-22025.0278—
o1-miniModel Size=~100B, # Reasoning Tokens=37552025.0278—
MCTS-JudgeApproach=System-2, Base Model=GPT-4o-mini2025.0276—
GPT-o1-previewApproach=System-22025.0275—
o1-previewModel Size=~300B, # Reasoning Tokens=56312025.0275—
MCTS-JudgeApproach=System-2, Base Model=Mistralai-Codestral-22B2025.0272—
ICE-ScoreApproach=System-1, Base Model=GPT-4o-mini2025.0272—
CodeJudgeApproach=System-1, Base Model=GPT-4o-mini2025.0272—
CodeJudgeApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.0268—
ICE-ScoreApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.0265—
VanillaApproach=System-1, Base Model=GPT-4o-mini2025.0265—
VanillaApproach=System-1, Base Model=Qwen2.5-Coder-14B-Instruct2025.0262—
CodeJudgeApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.0262—
VanillaApproach=System-1, Base Model=Mistralai-Codestral-22B2025.0262—
MCTS-JudgeApproach=System-2, Base Model=Llama-3.1-8B-Instruct2025.0262—
Qwen-QwQ-32BApproach=System-22025.0260—
Qwen-QwQ-32BModel Size=32B, # Reasoning Tokens=25592025.0260—
ICE-ScoreApproach=System-1, Base Model=Mistralai-Codestral-22B2025.0256—
VanillaApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.0256—
GPT-4oApproach=System-12025.0255—
CodeJudgeApproach=System-1, Base Model=Mistralai-Codestral-22B2025.0254—
CodeJudgeEval Model=GPT-4o-mini2024.1053.218.6
BCEval Model=Llama-3.1-8B2024.1053.261.1
CodeJudgeApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.0253—
CA-SPEval Model=Llama-3.1-8B2024.1052.560.1
BCEval Model=GPT-4o-mini2024.1051.413.4
ICE-ScoreEval Model=GPT-4o-mini2024.1051.215.5
CoCoAEval Model=GPT-4o-mini, lambda=Best2024.1051.266.7
CoCoAEval Model=Qwen-2.5-7B, lambda=Worst2024.1051.217.5
ICE-ScoreEval Model=Llama-3.1-8B2024.1051.253.7
CodeJudgeEval Model=Qwen-2.5-7B2024.1051.118.9
VanillaEval Model=GPT-4o-mini2024.1050.95.5
2PREval Model=Llama-3.1-8B2024.1050.911
MCTS-JudgeEval Model=GPT-4o-mini2024.1050.642.8
2PREval Model=GPT-4o-mini2024.1050.56.7
CA-SPEval Model=GPT-4o-mini2024.1050.512.6
CoCoAEval Model=Llama-3.1-8B, lambda=Best2024.1050.264.8
CoCoAEval Model=GPT-4o-mini, lambda=Worst2024.105044
VanillaEval Model=Qwen-2.5-7B2024.10503.4
BCEval Model=Qwen-2.5-7B2024.1049.61.4
CoCoAEval Model=Llama-3.1-8B, lambda=Worst2024.1049.561.1
ICE-ScoreEval Model=Qwen-2.5-7B2024.1049.113.9
VanillaEval Model=Llama-3.1-8B2024.1049.128.6
2PREval Model=Qwen-2.5-7B2024.1048.26.5
CoCoAEval Model=Qwen-2.5-7B, lambda=Best2024.104852.8
ICE-ScoreApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.0248—
CA-SPEval Model=Qwen-2.5-7B2024.1047.914.6
CodeJudgeEval Model=Llama-3.1-8B2024.1047.526.9
ICE-ScoreApproach=System-1, Base Model=Llama-3.1-8B-Instruct2025.0242—
VanillaApproach=System-1, Base Model=DeepSeek-Coder-V2-16B-Instruct2025.0241—