Mathematical Reasoning on GSM-Hard (pass@1 accuracy)
69.6GSM-Hard pass@1 AccRV-Syn
Evaluation results
| Method | Links | ||
|---|---|---|---|
| RV-SynBackbone=Qwen2.5-7B-Instruct, Problem Crafter=7B / 72B model, Chain-of-Thought=true, Zero-shot=true2025.04 | 69.6 | — | |
| Mammoth2Backbone=Qwen2.5-7B-Instruct, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 68.2 | — | |
| PromptCoTBackbone=Qwen2.5-7B-Instruct, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.7 | — | |
| MetaMathBackbone=Qwen2.5-7B-Instruct, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.6 | — | |
| MathScaleBackbone=Qwen2.5-7B-Instruct, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.6 | — | |
| ScaleQuestBackbone=Qwen2.5-7B-Instruct, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.6 | — | |
| Orca-MathBackbone=Qwen2.5-7B-Instruct, Problem Crafter=GPT-4, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.4 | — | |
| Jiuzhang3.0Backbone=Qwen2.5-7B-Instruct, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 67.4 | — | |
| Orca-MathBackbone=Phi-3-mini, Problem Crafter=GPT-4, Chain-of-Thought=true, Zero-shot=true2025.04 | 65.3 | — | |
| RV-SynBackbone=Phi-3-mini, Problem Crafter=7B / 72B model, Chain-of-Thought=true, Zero-shot=true2025.04 | 65 | — | |
| MetaMathBackbone=Phi-3-mini, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 64.5 | — | |
| Jiuzhang3.0Backbone=Phi-3-mini, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 63.8 | — | |
| ScaleQuestBackbone=Phi-3-mini, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 63.8 | — | |
| MathScaleBackbone=Phi-3-mini, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 63.5 | — | |
| PromptCoTBackbone=Phi-3-mini, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 63.5 | — | |
| Mammoth2Backbone=Phi-3-mini, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 62.9 | — | |
| Official ModelBackbone=Qwen2.5-7B-Instruct, Problem Crafter=-, Chain-of-Thought=true, Zero-shot=true2025.04 | 59.9 | — | |
| Official ModelBackbone=Phi-3-mini, Problem Crafter=-, Chain-of-Thought=true, Zero-shot=true2025.04 | 54.1 | — | |
| SFT-CoTBackbone=Qwen3-4B2026.05 | 50.23 | 117.3 | |
| RV-SynBackbone=LLaMA-3-8B-Instruct, Problem Crafter=7B / 72B model, Chain-of-Thought=true, Zero-shot=true2025.04 | 44.5 | — | |
| Orca-MathBackbone=LLaMA-3-8B-Instruct, Problem Crafter=GPT-4, Chain-of-Thought=true, Zero-shot=true2025.04 | 42.7 | — | |
| ScaleQuestBackbone=LLaMA-3-8B-Instruct, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 41.9 | — | |
| PromptCoTBackbone=LLaMA-3-8B-Instruct, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 41.5 | — | |
| Jiuzhang3.0Backbone=LLaMA-3-8B-Instruct, Problem Crafter=7B Math Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 40.5 | — | |
| Mammoth2Backbone=LLaMA-3-8B-Instruct, Problem Crafter=72B Model, Chain-of-Thought=true, Zero-shot=true2025.04 | 40.3 | — | |
| MathScaleBackbone=LLaMA-3-8B-Instruct, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 38.4 | — | |
| MetaMathBackbone=LLaMA-3-8B-Instruct, Problem Crafter=ChatGPT, Chain-of-Thought=true, Zero-shot=true2025.04 | 36.2 | — | |
| Official ModelBackbone=LLaMA-3-8B-Instruct, Problem Crafter=-, Chain-of-Thought=true, Zero-shot=true2025.04 | 35.6 | — | |
| Ours-RLBackbone=Qwen3-4B2026.05 | 35.1 | 63.9 | |
| Ours-SFTBackbone=Qwen3-4B2026.05 | 29.72 | 64.49 | |
| SFT-w/o CoTBackbone=Qwen3-4B2026.05 | 14.18 | 0 | |
| RoTBackbone=Qwen3-4B2026.05 | 14.1 | 32 | |
| SFT-CoTBackbone=Llama-3.2-1B2026.05 | 13.4 | 89.25 | |
| Ours-RLBackbone=Llama-3.2-1B2026.05 | 11.52 | 43.35 | |
| Ours-SFTBackbone=Llama-3.2-1B2026.05 | 10.91 | 42.07 | |
| CoLaR-2Backbone=Qwen3-4B2026.05 | 9.17 | 47.4 | |
| CoconutBackbone=Llama-3.2-1B2026.05 | 5.49 | 6 | |
| CoconutBackbone=Qwen3-4B2026.05 | 5.42 | 6 | |
| SFT-w/o CoTBackbone=Llama-3.2-1B2026.05 | 5 | 0 | |
| CoLaR-2Backbone=Llama-3.2-1B2026.05 | 3.86 | 42 |