# Mathematical Reasoning on SVAMP (Pass@1 Accuracy) Source: [Mathematical Reasoning on SVAMP (Pass@1 Accuracy)]() Collection index: [llms.txt]() Data updated: 2026-06-11T07:09:35.195Z Evaluation of model performance on the SVAMP mathematical reasoning dataset, specifically reporting Pass@1 Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Mathematical Reasoning | | Dataset (including any stated split) | SVAMP | | Metric | Pass@1 Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 27 of 27 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Pass@1 Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Pass@1 Accuracy | Result date | Report paper | | --- | --- | --- | --- | | Baseline | 94.2 | Not recorded | 68fa7c061e209131cea8bf2f | | SFT | 94.1 | Not recorded | 68fa7c061e209131cea8bf2f | | Baseline | 93.9 | Not recorded | 68fa7c061e209131cea8bf2f | | SFT | 93.5 | Not recorded | 68fa7c061e209131cea8bf2f | | GPT-4 | 93.1 | Not recorded | 69f03f3355a3c32af89fa21d | | LightR | 93.1 | Not recorded | 68fa7c061e209131cea8bf2f | | LightR | 93 | Not recorded | 68fa7c061e209131cea8bf2f | | IRIS | 90.6 | Not recorded | 69f03f3355a3c32af89fa21d | | LightR | 83.5 | Not recorded | 68fa7c061e209131cea8bf2f | | Qwen2.5 Base | 82.1 | Not recorded | 69f03f3355a3c32af89fa21d | | SFT | 81.5 | Not recorded | 68fa7c061e209131cea8bf2f | | Baseline | 79.9 | Not recorded | 68fa7c061e209131cea8bf2f | | LightR | 77.2 | Not recorded | 68fa7c061e209131cea8bf2f | | SFT | 76.2 | Not recorded | 68fa7c061e209131cea8bf2f | | LightR | 76 | Not recorded | 68fa7c061e209131cea8bf2f | | DeepSeek-Math | 73.2 | Not recorded | 69f03f3355a3c32af89fa21d | | IRIS | 70.4 | Not recorded | 69f03f3355a3c32af89fa21d | | MetaMath | 68.8 | Not recorded | 69f03f3355a3c32af89fa21d | | Baseline | 68.8 | Not recorded | 68fa7c061e209131cea8bf2f | | Baseline | 67.9 | Not recorded | 68fa7c061e209131cea8bf2f | | SFT | 64.1 | Not recorded | 68fa7c061e209131cea8bf2f | | WizMath Base | 57.3 | Not recorded | 69f03f3355a3c32af89fa21d | | Coconut + dropout-GRPO | 44 | Not recorded | 6a28f35090966ade919c1829 | | Qwen2.5-1.5B | 43 | Not recorded | 6a28f35090966ade919c1829 | | Coconut SFT | 41.67 | Not recorded | 6a28f35090966ade919c1829 | | Coconut + GRPO | 41.67 | Not recorded | 6a28f35090966ade919c1829 | | Coconut + REINFORCE + dropout | 38.78 | Not recorded | 6a28f35090966ade919c1829 | ### Baseline Pass@1 Accuracy: 94.2. Result ID: 6a100628fbc373592c089b73. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### SFT Pass@1 Accuracy: 94.1. Result ID: 6a100628fbc373592c089b7d. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### Baseline Pass@1 Accuracy: 93.9. Result ID: 6a100628fbc373592c089b7c. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### SFT Pass@1 Accuracy: 93.5. Result ID: 6a100628fbc373592c089b74. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### GPT-4 Pass@1 Accuracy: 93.1. Result ID: 69f0810b47f34ce924d6d78d. [Reporting paper]() Recorded evaluation configuration: {"Zero-shot":"true"}. This record does not assert independent reproduction. ### LightR Pass@1 Accuracy: 93.1. Result ID: 6a100628fbc373592c089b7e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### LightR Pass@1 Accuracy: 93. Result ID: 6a100628fbc373592c089b75. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B-Instruct","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### IRIS Pass@1 Accuracy: 90.6. Result ID: 69f0810b47f34ce924d6d78b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"Qwen2.5","Zero-shot":"true"}. This record does not assert independent reproduction. ### LightR Pass@1 Accuracy: 83.5. Result ID: 6a100628fbc373592c089b78. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"DeepSeek-R1-Distill-Qwen-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### Qwen2.5 Base Pass@1 Accuracy: 82.1. Result ID: 69f0810b47f34ce924d6d78e. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Zero-shot":"true"}. This record does not assert independent reproduction. ### SFT Pass@1 Accuracy: 81.5. Result ID: 6a100628fbc373592c089b77. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"DeepSeek-R1-Distill-Qwen-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### Baseline Pass@1 Accuracy: 79.9. Result ID: 6a100628fbc373592c089b76. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"DeepSeek-R1-Distill-Qwen-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### LightR Pass@1 Accuracy: 77.2. Result ID: 6a100628fbc373592c089b7b. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### SFT Pass@1 Accuracy: 76.2. Result ID: 6a100628fbc373592c089b7a. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### LightR Pass@1 Accuracy: 76. Result ID: 6a100628fbc373592c089b72. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### DeepSeek-Math Pass@1 Accuracy: 73.2. Result ID: 69f0810b47f34ce924d6d791. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Zero-shot":"true"}. This record does not assert independent reproduction. ### IRIS Pass@1 Accuracy: 70.4. Result ID: 69f0810b47f34ce924d6d78c. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Backbone":"WizMath","Zero-shot":"true"}. This record does not assert independent reproduction. ### MetaMath Pass@1 Accuracy: 68.8. Result ID: 69f0810b47f34ce924d6d790. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Zero-shot":"true"}. This record does not assert independent reproduction. ### Baseline Pass@1 Accuracy: 68.8. Result ID: 6a100628fbc373592c089b70. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### Baseline Pass@1 Accuracy: 67.9. Result ID: 6a100628fbc373592c089b79. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-7B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### SFT Pass@1 Accuracy: 64.1. Result ID: 6a100628fbc373592c089b71. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Base Model":"Qwen2.5-Math-1.5B","Evaluation Protocol":"Zero-shot"}. This record does not assert independent reproduction. ### WizMath Base Pass@1 Accuracy: 57.3. Result ID: 69f0810b47f34ce924d6d78f. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Zero-shot":"true"}. This record does not assert independent reproduction. ### Coconut + dropout-GRPO Pass@1 Accuracy: 44. Result ID: 6a2a5f28fbc373592c26fa66. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"seeds":"3"}. This record does not assert independent reproduction. ### Qwen2.5-1.5B Pass@1 Accuracy: 43. Result ID: 6a2a5f28fbc373592c26fa62. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"mode":"zero-shot"}. This record does not assert independent reproduction. ### Coconut SFT Pass@1 Accuracy: 41.67. Result ID: 6a2a5f28fbc373592c26fa63. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"baseline":"true","seeds":"1"}. This record does not assert independent reproduction. ### Coconut + GRPO Pass@1 Accuracy: 41.67. Result ID: 6a2a5f28fbc373592c26fa64. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"rollout":"deterministic","seeds":"3"}. This record does not assert independent reproduction. ### Coconut + REINFORCE + dropout Pass@1 Accuracy: 38.78. Result ID: 6a2a5f28fbc373592c26fa65. [Reporting paper]() Recorded evaluation configuration: {"baseline\_type":"EMA","seeds":"3"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.