Mathematical Reasoning on SVAMP (Pass@1 Accuracy)
94.2Pass@1 AccuracyBaseline
Evaluation Results
| Method | Links | |
|---|---|---|
| BaselineBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 94.2 | |
| SFTBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 94.1 | |
| BaselineBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 93.9 | |
| SFTBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 93.5 | |
| GPT-4Zero-shot=true2026.04 | 93.1 | |
| LightRBase Model=Qwen2.5-Math-7B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 93.1 | |
| LightRBase Model=Qwen2.5-Math-1.5B-Instruct, Evaluation Protocol=Zero-shot2025.10 | 93 | |
| IRISBackbone=Qwen2.5, Zero-shot=true2026.04 | 90.6 | |
| LightRBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.10 | 83.5 | |
| Qwen2.5 BaseZero-shot=true2026.04 | 82.1 | |
| SFTBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.10 | 81.5 | |
| BaselineBase Model=DeepSeek-R1-Distill-Qwen-1.5B, Evaluation Protocol=Zero-shot2025.10 | 79.9 | |
| LightRBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.10 | 77.2 | |
| SFTBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.10 | 76.2 | |
| LightRBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.10 | 76 | |
| DeepSeek-MathZero-shot=true2026.04 | 73.2 | |
| IRISBackbone=WizMath, Zero-shot=true2026.04 | 70.4 | |
| MetaMathZero-shot=true2026.04 | 68.8 | |
| BaselineBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.10 | 68.8 | |
| BaselineBase Model=Qwen2.5-Math-7B, Evaluation Protocol=Zero-shot2025.10 | 67.9 | |
| SFTBase Model=Qwen2.5-Math-1.5B, Evaluation Protocol=Zero-shot2025.10 | 64.1 | |
| WizMath BaseZero-shot=true2026.04 | 57.3 | |
| Coconut + dropout-GRPOseeds=32026.06 | 44 | |
| Qwen2.5-1.5Bmode=zero-shot2026.06 | 43 | |
| Coconut SFTbaseline=true, seeds=12026.06 | 41.67 | |
| Coconut + GRPOrollout=deterministic, seeds=32026.06 | 41.67 | |
| Coconut + REINFORCE + dropoutbaseline_type=EMA, seeds=32026.06 | 38.78 |