Code Generation on MBPP 2021 (test)
94.94AccuracyPoT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| PoTBase Model=Qwen3-4B-Instruct-25072026.01 | 94.94 | — | |
| LATSBase Model=Qwen3-4B-Instruct-2507, Method Category=Search-Based Reasoning2026.01 | 90.66 | — | |
| Deepseek-V3Inference Strategy=Zero-shot2026.01 | 87.6 | — | |
| Gpt-4oInference Strategy=Zero-shot2026.01 | 87.6 | — | |
| RethinkMCTSBase Model=Qwen3-4B-Instruct-2507, Method Category=Search-Based Reasoning2026.01 | 87.34 | — | |
| ReflexionBase Model=Qwen3-4B-Instruct-2507, Method Category=Self-Refinement / Debugging2026.01 | 85.6 | — | |
| CODETBase Model=Qwen3-4B-Instruct-2507, Method Category=Self-Refinement / Debugging2026.01 | 83.79 | — | |
| PG-TDBase Model=Qwen3-4B-Instruct-2507, Method Category=Search-Based Reasoning2026.01 | 83.72 | — | |
| AB-MCTSBase Model=Qwen3-4B-Instruct-2507, Method Category=Search-Based Reasoning2026.01 | 83.66 | — | |
| Claude-Opus-4Inference Strategy=Zero-shot2026.01 | 83.2 | — | |
| Few-shot + CoTBase Model=Qwen3-4B-Instruct-2507, Method Category=Standard Inference2026.01 | 82.88 | — | |
| ToTBase Model=Qwen3-4B-Instruct-2507, Method Category=Search-Based Reasoning2026.01 | 82.37 | — | |
| Best of N (N=20)Base Model=Qwen3-4B-Instruct-2507, Method Category=Standard Inference2026.01 | 82.1 | — | |
| RAPBase Model=Qwen3-4B-Instruct-2507, Method Category=Self-Refinement / Debugging2026.01 | 81.71 | — | |
| Gemini-2.5-flashInference Strategy=Zero-shot2026.01 | 81.7 | — | |
| Self-RefineBase Model=Qwen3-4B-Instruct-2507, Method Category=Self-Refinement / Debugging2026.01 | 80.13 | — | |
| Zero-shotBase Model=Qwen3-4B-Instruct-2507, Method Category=Standard Inference2026.01 | 79.77 | — | |
| Qwen3-235B-A22BInference Strategy=Zero-shot2026.01 | 75.1 | — | |
| Merged (α=0.2)Model variant=Merged (α=0.2), Interpolation Weight=0.22026.04 | — | 78.2 | |
| Qwen3-4B-Instruct-2507Model variant=Base Model2026.04 | — | 75.2 | |
| SFT No-TestModel variant=SFT No-Test2026.04 | — | 70.2 |