Mathematical Reasoning on SVAMP (Pass@1)
93.1Pass@1GPT-4 (0613)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| GPT-4 (0613)Zero-shot without demonstrations=false2024.03 | 93.1 | — | |
| SFT-CoTModel=Qwen3-VL-4B-Instruct2026.01 | 84.3 | 55.9 | |
| ChatGPTZero-shot without demonstrations=false2024.03 | 83 | — | |
| LEMMA (w/ MetaMath)Backbone=LLaMA3-8B, # Samples=403.59k2025.03 | 82.8 | — | |
| LEMMABackbone=LLaMA3-8B, # Samples=88.90k2025.03 | 82.6 | — | |
| KPMath-PlusBase=Qwen1.5, Size=72B, Zero-shot without demonstrations=true2024.03 | 82.1 | — | |
| LEMMA (w/ MetaMath)Backbone=DeepSeekMath-7B, # Samples=403.59k2025.03 | 82.1 | — | |
| RefAug-90kBackbone=LLaMA3-8B, # Samples=89.92k2025.03 | 81.8 | — | |
| LEMMABackbone=DeepSeekMath-7B, # Samples=88.90k2025.03 | 81.6 | — | |
| KPMath-PlusBase=DSMath, Size=7B, Zero-shot without demonstrations=true2024.03 | 81.5 | — | |
| RefAugBackbone=LLaMA3-8B, # Samples=29.94k2025.03 | 81.5 | — | |
| KPMath-PlusBase=Llemma, Size=34B, Zero-shot without demonstrations=true2024.03 | 81.2 | — | |
| KPMath-PlusBase=Llama-2, Size=70B, Zero-shot without demonstrations=true2024.03 | 81.2 | — | |
| GPTAugBackbone=LLaMA3-8B, # Samples=88.62k2025.03 | 79.7 | — | |
| ISCBackbone=LLaMA3-8B, # Samples=86.78k2025.03 | 79.7 | — | |
| MetaMathBackbone=DeepSeekMath-7B, # Samples=394.99k2025.03 | 79.4 | — | |
| KPMath-PlusBase=Llama-2, Size=13B, Zero-shot without demonstrations=true2024.03 | 76.7 | — | |
| KPMath-PlusBase=Mistral, Size=7B, Zero-shot without demonstrations=true2024.03 | 76.4 | — | |
| MetaMathBackbone=LLaMA3-8B, # Samples=394.99k2025.03 | 76.1 | — | |
| RefAug-90kBackbone=DeepSeekMath-7B, # Samples=89.92k2025.03 | 74.1 | — | |
| RefAugBackbone=DeepSeekMath-7B, # Samples=29.94k2025.03 | 72.8 | — | |
| RoT (Ours)Model=Qwen3-VL-4B-Instruct2026.01 | 72.7 | 32 | |
| ISCBackbone=DeepSeekMath-7B, # Samples=86.78k2025.03 | 71.2 | — | |
| GPTAugBackbone=DeepSeekMath-7B, # Samples=88.62k2025.03 | 71 | — | |
| RFTBackbone=DeepSeekMath-7B, # Samples=86.52k2025.03 | 70.9 | — | |
| SFT-w/o CoTModel=Qwen3-VL-4B-Instruct2026.01 | 70 | 0 | |
| RFTBackbone=LLaMA3-8B, # Samples=86.52k2025.03 | 69.9 | — | |
| SFTBackbone=DeepSeekMath-7B, # Samples=14.97k2025.03 | 68.1 | — | |
| SFT-CoTModel=Qwen3-VL-2B-Instruct2026.01 | 67.3 | 63.4 | |
| SFTBackbone=LLaMA3-8B, # Samples=14.97k2025.03 | 67.1 | — | |
| SFT-CoTModel=LLaVa-V1.6-Mistral-7B2026.01 | 56.3 | 79.9 | |
| RoT (Ours)Model=Qwen3-VL-2B-Instruct2026.01 | 53.7 | 32 | |
| SFT-w/o CoTModel=Qwen3-VL-2B-Instruct2026.01 | 52.3 | 0 | |
| RoT (Ours)Model=LLaVa-V1.6-Mistral-7B2026.01 | 49 | 32 | |
| SFT-w/o CoTModel=LLaVa-V1.6-Mistral-7B2026.01 | 40.7 | 0 |