# Mathematical Reasoning on MMLU-Pro Math Source: [Mathematical Reasoning on MMLU-Pro Math]() Collection index: [llms.txt]() Data updated: 2026-05-13T07:49:37.954Z Evaluation of model performance on the MMLU-Pro Math dataset focusing on mathematical reasoning capabilities, measured by Accuracy. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Mathematical Reasoning | | Dataset (including any stated split) | MMLU-Pro Math | | Metric | Accuracy | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 26 of 26 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Accuracy](): Higher is better; unit: PERCENT. - [F1 Score](): Higher is better; unit: PERCENT. - [Precision](): Higher is better; unit: PERCENT. - [Specificity](): Higher is better; unit: PERCENT. - [MCC (Matthews Correlation Coefficient)](): Higher is better; unit: PERCENT. ## Recorded results | Method | Accuracy | Result date | Report paper | | --- | --- | --- | --- | | MAC | 89.79 | Not recorded | 68fa7a4e1e209131cea8259c | | Debate | 86.75 | Not recorded | 68fa7a4e1e209131cea8259c | | QWQ | 85.12 | Not recorded | 68fa7a4e1e209131cea8259c | | Deepseek-R1 | 84.9 | Not recorded | 68fa7a4e1e209131cea8259c | | Openthinker | 84.83 | Not recorded | 68fa7a4e1e209131cea8259c | | Phi4 | 79.42 | Not recorded | 68fa7a4e1e209131cea8259c | | Qwen2.5-14B (Teacher) | 77.6 | Not recorded | 69b8e0050636e5bf4afbe978 | | Qwen2.5-14B | 77.6 | Not recorded | 69b8e0050636e5bf4afbe978 | | Qwen2.5 | 76.68 | Not recorded | 68fa7a4e1e209131cea8259c | | Qwen2.5 | 73.95 | Not recorded | 68fa7a4e1e209131cea8259c | | TSD-KD | 38.5 | Not recorded | 69b8e0050636e5bf4afbe978 | | TSD-KD | 36.9 | Not recorded | 69b8e0050636e5bf4afbe978 | | MiniLLM | 36.2 | Not recorded | 69b8e0050636e5bf4afbe978 | | MiniLLM | 36.2 | Not recorded | 69b8e0050636e5bf4afbe978 | | GKD | 35.7 | Not recorded | 69b8e0050636e5bf4afbe978 | | GKD | 35.7 | Not recorded | 69b8e0050636e5bf4afbe978 | | Supervised-KD | 35.3 | Not recorded | 69b8e0050636e5bf4afbe978 | | Supervised-KD | 35.3 | Not recorded | 69b8e0050636e5bf4afbe978 | | DistiLLM | 35.2 | Not recorded | 69b8e0050636e5bf4afbe978 | | DistiLLM | 35.2 | Not recorded | 69b8e0050636e5bf4afbe978 | | Speculative KD | 34.8 | Not recorded | 69b8e0050636e5bf4afbe978 | | Speculative KD | 34.8 | Not recorded | 69b8e0050636e5bf4afbe978 | | Qwen2.5-1.5B (Student) | 34.4 | Not recorded | 69b8e0050636e5bf4afbe978 | | Qwen2.5-1.5B | 34.4 | Not recorded | 69b8e0050636e5bf4afbe978 | | Sequence-Level KD | 32.6 | Not recorded | 69b8e0050636e5bf4afbe978 | | Sequence-Level KD | 32.6 | Not recorded | 69b8e0050636e5bf4afbe978 | ### MAC Accuracy: 89.79. Result ID: 6a042d0947f34ce924e9d35b. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Debate Accuracy: 86.75. Result ID: 6a042d0947f34ce924e9d35a. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### QWQ Accuracy: 85.12. Result ID: 6a042d0947f34ce924e9d357. [Reporting paper]() Recorded evaluation configuration: {"Parameters":"32B"}. This record does not assert independent reproduction. ### Deepseek-R1 Accuracy: 84.9. Result ID: 6a042d0947f34ce924e9d359. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"32B"}. This record does not assert independent reproduction. ### Openthinker Accuracy: 84.83. Result ID: 6a042d0947f34ce924e9d358. [Reporting paper]() Recorded evaluation configuration: {"Parameters":"32B"}. This record does not assert independent reproduction. ### Phi4 Accuracy: 79.42. Result ID: 6a042d0947f34ce924e9d354. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"14B"}. This record does not assert independent reproduction. ### Qwen2.5-14B (Teacher) Accuracy: 77.6. Result ID: 69b9018c47f34ce924a1157e. [Reporting paper]() Recorded evaluation configuration: {"Model Scale":"14B"}. This record does not assert independent reproduction. ### Qwen2.5-14B Accuracy: 77.6. Result ID: 69b9018c47f34ce924a115a2. [Reporting paper]() Recorded evaluation configuration: {"Role":"Teacher"}. This record does not assert independent reproduction. ### Qwen2.5 Accuracy: 76.68. Result ID: 6a042d0947f34ce924e9d356. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"32B"}. This record does not assert independent reproduction. ### Qwen2.5 Accuracy: 73.95. Result ID: 6a042d0947f34ce924e9d355. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Parameters":"14B"}. This record does not assert independent reproduction. ### TSD-KD Accuracy: 38.5. Result ID: 69b9018c47f34ce924a115aa. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"selection":"Adaptive c%"}. This record does not assert independent reproduction. ### TSD-KD Accuracy: 36.9. Result ID: 69b9018c47f34ce924a11586. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### MiniLLM Accuracy: 36.2. Result ID: 69b9018c47f34ce924a11584. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### MiniLLM Accuracy: 36.2. Result ID: 69b9018c47f34ce924a115a8. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### GKD Accuracy: 35.7. Result ID: 69b9018c47f34ce924a11585. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"beta":"0.9","Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### GKD Accuracy: 35.7. Result ID: 69b9018c47f34ce924a115a9. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"beta":"0.9"}. This record does not assert independent reproduction. ### Supervised-KD Accuracy: 35.3. Result ID: 69b9018c47f34ce924a11581. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### Supervised-KD Accuracy: 35.3. Result ID: 69b9018c47f34ce924a115a5. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### DistiLLM Accuracy: 35.2. Result ID: 69b9018c47f34ce924a11582. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### DistiLLM Accuracy: 35.2. Result ID: 69b9018c47f34ce924a115a6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Speculative KD Accuracy: 34.8. Result ID: 69b9018c47f34ce924a11583. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### Speculative KD Accuracy: 34.8. Result ID: 69b9018c47f34ce924a115a7. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. ### Qwen2.5-1.5B (Student) Accuracy: 34.4. Result ID: 69b9018c47f34ce924a1157f. [Reporting paper]() Recorded evaluation configuration: {"Model Scale":"1.5B"}. This record does not assert independent reproduction. ### Qwen2.5-1.5B Accuracy: 34.4. Result ID: 69b9018c47f34ce924a115a3. [Reporting paper]() Recorded evaluation configuration: {"Role":"Student"}. This record does not assert independent reproduction. ### Sequence-Level KD Accuracy: 32.6. Result ID: 69b9018c47f34ce924a11580. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Distillation Setup":"Qwen2.5 (14B -\> 1.5B)"}. This record does not assert independent reproduction. ### Sequence-Level KD Accuracy: 32.6. Result ID: 69b9018c47f34ce924a115a4. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: Not recorded. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.