Multilingual Mathematical Reasoning on MGSM
94.9AccuracyDeepSeek V3.2
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DeepSeek V3.2Evaluation Mode=Chat2025.12 | 94.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LongCat-Flash Exp-ChatEvaluation Mode=Chat2025.12 | 94.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GLM 4.6Evaluation Mode=Chat2025.12 | 91.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LongCat-Flash ChatEvaluation Mode=Chat2025.12 | 87.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 InstructModel Scale=70B2025.04 | 87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3 InstructModel Scale=70B2025.04 | 84.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ParamΔModel Scale=70B2025.04 | 84.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2-72BProtocol=8-shot CoT2024.07 | 76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Claude 3.5 Haiku (target)Model=Claude 3.5 Haiku2026.04 | 75.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flan-PaLM 540Bprompting=CoT + self-consistency2022.10 | 72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 InstructModel Scale=8B2025.04 | 68.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-3-70BProtocol=8-shot CoT2024.07 | 67.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mixtral-8x22BProtocol=8-shot CoT2024.07 | 62.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SASFTModel=Qwen3-8B-Base2025.07 | 61.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen1.5-72BProtocol=8-shot CoT2024.07 | 61.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen1.5-110BProtocol=8-shot CoT2024.07 | 61.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3 InstructModel Scale=8B2025.04 | 60.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+PenaltyModel=Qwen3-8B-Base2025.07 | 60.72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ParamΔModel Scale=8B2025.04 | 60.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTModel=Qwen3-8B-Base2025.07 | 59.47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+GRPOModel=Qwen3-8B-Base2025.07 | 55.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM + Google translate APIprompting=CoT prompting, self-consistency=false2022.10 | 55 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+GRPOModel=Gemma-2-9B2025.07 | 50.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTModel=Gemma-2-9B2025.07 | 48.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SASFTModel=Gemma-2-9B2025.07 | 47.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+PenaltyModel=Gemma-2-9B2025.07 | 46.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSA-PTTraining Strategy=from-scratch sparse pretraining2026.06 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM 540Bprompting=CoT prompting2022.10 | 45.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 8B (baseline)Model=Llama 3.1 8B2026.04 | 45.67 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MSA-CPTTraining Strategy=sparse continued pretraining2026.06 | 44.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FullTraining Strategy=Full-Attention baseline2026.06 | 44.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GARFA (Exp A)Optimization Strategy=L*+L*2026.04 | 43.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Exp B: L*+LrOptimization Strategy=L*+Lr2026.04 | 43.45 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Exp C: Lr+L*Optimization Strategy=Lr+L*2026.04 | 39.78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM 540Bprompting=CoT + self-consistency2022.10 | 37.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flan-PaLM 540Bprompting=CoT prompting2022.10 | 37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MERGEvolveSetting=single-task, Number of runs=52026.06 | 35.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LoraHubSetting=single-task, Number of runs=52026.06 | 34.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Exp D: Lr+Lr (control)Optimization Strategy=Lr+Lr2026.04 | 34.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Model SwarmsSetting=single-task, Number of runs=52026.06 | 33.9 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+PenaltyModel=Qwen3-1.7B-Base2025.07 | 33.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TIESSetting=single-task, Number of runs=52026.06 | 33 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SASFTModel=Qwen3-1.7B-Base2025.07 | 32.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTModel=Qwen3-1.7B-Base2025.07 | 32.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+GRPOModel=Qwen3-1.7B-Base2025.07 | 32.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Expert FusionSetting=single-task, Number of runs=52026.06 | 31.82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Pack of LLMsSetting=single-task, Number of runs=52026.06 | 30.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Best Single ExpertSetting=single-task, Number of runs=52026.06 | 30.22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMMSetting=single-task, Number of runs=52026.06 | 30.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Data MergeSetting=single-task, Number of runs=52026.06 | 25.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Flan-PaLM 540Bprompting=direct prompting2022.10 | 21.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+PenaltyModel=Llama-3.1-8B2025.07 | 18.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PaLM 540Bprompting=direct prompting2022.10 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 BaseModel Scale=70B2025.04 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTModel=Llama-3.1-8B2025.07 | 15.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SASFTModel=Llama-3.1-8B2025.07 | 13.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+PenaltyModel=Gemma-2-2B2025.07 | 12.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+GRPOModel=Gemma-2-2B2025.07 | 12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTModel=Gemma-2-2B2025.07 | 11.89 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SASFTModel=Gemma-2-2B2025.07 | 11.31 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFT+GRPOModel=Llama-3.1-8B2025.07 | 9.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3 BaseModel Scale=70B2025.04 | 9.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3.1 BaseModel Scale=8B2025.04 | 4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama 3 BaseModel Scale=8B2025.04 | 2.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| baseBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 49.6 | — | 57.6 | — | — | — | — | — | — | — | — | — | 51.5 | 62.4 | 36.4 | — | |
| baseBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 57.2 | — | 42.8 | — | — | — | — | — | — | — | — | — | 52.8 | 81.6 | 29.6 | — | |
| CoTBackbone Model=Llama2-7B, Reasoning Strategy=Chain-of-Thought2025.02 | — | 5.6 | 4.8 | 2 | 15.2 | 25.2 | 32.4 | 30.4 | 26.4 | 30.4 | 43.6 | — | — | 21.6 | — | — | — | |
| CoTBackbone Model=CodeLlama 7B, Reasoning Strategy=Chain-of-Thought2025.02 | — | 9.2 | 16.4 | 3.2 | 18.8 | 27.2 | 33.2 | 32.8 | 26.8 | 39.6 | 43.2 | — | — | 25 | — | — | — | |
| CoTBackbone Model=Llama2-13B, Reasoning Strategy=Chain-of-Thought2025.02 | — | 6.8 | 7.2 | 7.4 | 18.8 | 35.2 | 39.2 | 37.6 | 38 | 41.2 | 47.4 | — | — | 27.9 | — | — | — | |
| CoTBackbone Model=Llama3-8B, Reasoning Strategy=Chain-of-Thought2025.02 | — | 39.6 | 39.6 | 28 | 33.6 | 40 | 51.2 | 52.8 | 45.2 | 54.8 | 62.8 | — | — | 44.8 | — | — | — | |
| DAREBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 52.4 | — | 56 | — | — | — | — | — | — | — | — | — | 49.2 | 53.6 | 34.8 | — | |
| DAREBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 62.4 | — | 40.8 | — | — | — | — | — | — | — | — | — | 55 | 83.6 | 33.2 | — | |
| Gemma3-12B-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84 | |
| Granite4-Small-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 75.7 | |
| Granite4-Tiny-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 55.3 | |
| Lang-OnlyBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 49.2 | — | 58.8 | — | — | — | — | — | — | — | — | — | 48.8 | 53.2 | 34 | — | |
| Lang-OnlyBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 62 | — | 41.2 | — | — | — | — | — | — | — | — | — | 53.8 | 79.6 | 32.4 | — | |
| LangBridge2024.05 | — | 42.8 | 50.4 | 43.2 | 40 | 45.2 | 56.4 | 50.8 | 56.4 | 52.4 | 58 | 45.5 | 52.3 | 50.2 | — | — | — | |
| Layer SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 50.4 | — | 59.6 | — | — | — | — | — | — | — | — | — | 49.6 | 51.6 | 36.8 | — | |
| Layer SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 64.8 | — | 38 | — | — | — | — | — | — | — | — | — | 55.9 | 86 | 34.8 | — | |
| Layer-wise SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 50.4 | — | 57.6 | — | — | — | — | — | — | — | — | — | 50.2 | 57.6 | 35.2 | — | |
| Layer-wise SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 60 | — | 39.6 | — | — | — | — | — | — | — | — | — | 55.4 | 85.6 | 36.4 | — | |
| LFM2-24B-A2B-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 67.8 | |
| LFM2-8B-A1B-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 56.5 | |
| Marco-Mini-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87.4 | |
| Marco-Nano-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.5 | |
| MindMerger-Hardstrategy=Hard2024.05 | — | 46 | 36 | 48.4 | 52.4 | 54.4 | 60.4 | 57.6 | 60.8 | 56 | 60.4 | 43.5 | 59.5 | 54.7 | — | — | — | |
| MindMerger-Softstrategy=Soft2024.05 | — | 50.4 | 52.8 | 57.2 | 54.4 | 53.6 | 61.2 | 58.4 | 62 | 71.2 | 66.8 | 53.5 | 59 | 57.3 | — | — | — | |
| Ministral-3-3B-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 47 | |
| Ministral-3-8B-InstructEvaluation Protocol=Avg@82026.04 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 68.7 | |
| Mixed SFTBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 48 | — | 53.6 | — | — | — | — | — | — | — | — | — | 46.6 | 51.6 | 33.2 | — | |
| Mixed SFTBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 59.2 | — | 40 | — | — | — | — | — | — | — | — | — | 54.9 | 84.4 | 36 | — | |
| Module-wise SwappingBackbone=LLaMA 3.1 8B it, Evaluation Protocol=0-shot2026.01 | — | 47.6 | — | 59.2 | — | — | — | — | — | — | — | — | — | 50.7 | 57.6 | 38.4 | — | |
| Module-wise SwappingBackbone=Qwen 2 7B, Evaluation Protocol=0-shot2026.01 | — | 63.2 | — | 41.2 | — | — | — | — | — | — | — | — | — | 56.4 | 86 | 35.2 | — | |
| MonoReason2024.05 | — | 6.8 | 7.2 | 6.8 | 36.4 | 38.4 | 55.2 | 54.4 | 52 | 57.2 | 68.8 | 6.9 | 51.8 | 38.3 | — | — | — | |
| MultiReason-Loratuning=LoRA2024.05 | — | 29.6 | 35.2 | 28 | 52 | 54.8 | 59.6 | 58.4 | 62.4 | 59.6 | 64.8 | 30.9 | 58.8 | 50.4 | — | — | — | |
| MultiReason-SFTtuning=SFT2024.05 | — | 33.2 | 40 | 42 | 42 | 42 | 38.4 | 45.6 | 43.4 | 37.5 | 46.8 | 38.4 | 42 | 41.2 | — | — | — | |
| PoTBackbone Model=Llama2-7B, Reasoning Strategy=Program-of-Thought2025.02 | — | 12 | 7.6 | 5.6 | 32.7 | 38.4 | 40.4 | 40.4 | 37.1 | 43.6 | 58 | — | — | 31.6 | — | — | — | |
| PoTBackbone Model=CodeLlama 7B, Reasoning Strategy=Program-of-Thought2025.02 | — | 11.2 | 26.8 | 4.4 | 39.6 | 41.6 | 48.4 | 51.6 | 49.8 | 53.6 | 58.8 | — | — | 38.6 | — | — | — | |
| PoTBackbone Model=Llama2-13B, Reasoning Strategy=Program-of-Thought2025.02 | — | 13.6 | 13.9 | 7.2 | 40 | 44 | 52.4 | 54.4 | 51.2 | 55.6 | 64 | — | — | 39.6 | — | — | — |