Massive Multitask Language Understanding on MMLU-Pro (Accuracy)
88.3Accuracy (MMLU-Pro)Qwen-3.5
Evaluation results
| Method | Links | |
|---|---|---|
| Qwen-3.5Parameter Count=397B-17B2026.06 | 88.3 | |
| Kimi-K2.6Parameter Count=1T-A32B2026.06 | 88.1 | |
| DS-v4-ProParameter Count=1.6T-A49B2026.06 | 87.5 | |
| Nemotron 3 UltraParameter Count=550B-A55B2026.06 | 86.8 | |
| DS-v4-FlashParameter Count=284B-A13B2026.06 | 86.4 | |
| GLM-5.1Parameter Count=744B-A40B2026.06 | 85.9 | |
| Hybrid-LoRABackbone=Qwen-2.5 7B Instruct, Post-training Approach=GSPO, #Tunable Params=449M2026.05 | 83.8 | |
| Hybrid-LoRABackbone=Qwen-2.5 7B Instruct, Post-training Approach=GRPO, #Tunable Params=449M2026.05 | 83.1 | |
| AutoLoRABackbone=Qwen-2.5 7B Instruct, Post-training Approach=GSPO, #Tunable Params=453M2026.05 | 81.9 | |
| MiniMax-2.7Parameter Count=230B-A10B2026.06 | 81.9 | |
| MIPROv2Model=Claude 3.7 Sonnet2025.11 | 80.6 | |
| AutoLoRABackbone=Qwen-2.5 7B Instruct, Post-training Approach=GRPO, #Tunable Params=453M2026.05 | 80.3 | |
| BFRSModel=Claude 3.7 Sonnet2025.11 | 80.1 | |
| Zero-Shot CoTModel=Claude 3.7 Sonnet2025.11 | 79.7 | |
| Qwen3-30B-A3BGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 78.6 | |
| Zero-Shot PredictModel=o3 Mini2025.11 | 78.4 | |
| Zero-Shot PredictModel=Claude 3.7 Sonnet2025.11 | 77.7 | |
| Qwen3-14BGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 77.3 | |
| HELM BaselineModel=o3 Mini2025.11 | 77.1 | |
| BFRSModel=o3 Mini2025.11 | 76.5 | |
| HELM BaselineModel=Claude 3.7 Sonnet2025.11 | 76.3 | |
| Zero-Shot CoTModel=o3 Mini2025.11 | 76.2 | |
| MIPROv2Model=o3 Mini2025.11 | 76.1 | |
| BFRSModel=Gemini 2.0 Flash2025.11 | 75.4 | |
| Zero-Shot CoTModel=Gemini 2.0 Flash2025.11 | 75.3 | |
| MIPROv2Model=Gemini 2.0 Flash2025.11 | 75.3 | |
| Full-FT (ref.)Backbone=Qwen-2.5 3B Instruct, Post-training Approach=GSPO, #Tunable Params=3086M2026.05 | 73.4 | |
| Hybrid-LoRABackbone=Qwen-2.5 3B Instruct, Post-training Approach=GSPO, #Tunable Params=192M2026.05 | 72.9 | |
| Hybrid-LoRABackbone=Qwen-2.5 3B Instruct, Post-training Approach=GRPO, #Tunable Params=192M2026.05 | 71.9 | |
| Full-FT (ref.)Backbone=Qwen-2.5 3B Instruct, Post-training Approach=GRPO, #Tunable Params=3086M2026.05 | 71.8 | |
| BFRSModel=GPT 4o2025.11 | 71.1 | |
| AutoLoRABackbone=Qwen-2.5 3B Instruct, Post-training Approach=GSPO, #Tunable Params=194M2026.05 | 71.1 | |
| Zero-Shot PredictModel=Gemini 2.0 Flash2025.11 | 70.3 | |
| AutoLoRABackbone=Qwen-2.5 3B Instruct, Post-training Approach=GRPO, #Tunable Params=201M2026.05 | 69.3 | |
| MIPROv2Model=GPT 4o2025.11 | 68.7 | |
| BFRSModel=Qwen3 4B2025.11 | 68.6 | |
| MIPROv2Model=Qwen3 4B2025.11 | 68.6 | |
| Zero-Shot CoTModel=Llama 3.3 70B2025.11 | 68.5 | |
| BFRSModel=Llama 3.3 70B2025.11 | 68.5 | |
| Zero-Shot CoTModel=GPT 4o2025.11 | 67.6 | |
| OWPOModel=Qwen-3-8B-Base2026.05 | 67.41 | |
| Zero-Shot CoTModel=Qwen3 4B2025.11 | 66.2 | |
| HELM BaselineModel=Gemini 2.0 Flash2025.11 | 66.1 | |
| MOPDModel=Qwen-3-8B-Base2026.05 | 65.99 | |
| OPDModel=Qwen-3-8B-Base2026.05 | 65.91 | |
| Sym-DAPOModel=Qwen-3-8B-Base2026.05 | 65.58 | |
| DAPOModel=Qwen-3-8B-Base2026.05 | 65.29 | |
| GRPOModel=Qwen-3-8B-Base2026.05 | 64.71 | |
| HELM BaselineModel=Llama 3.3 70B2025.11 | 64.7 | |
| Gpt-oss-20b-highGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 63.1 | |
| Zero-Shot PredictModel=Llama 3.3 70B2025.11 | 62.7 | |
| HELM BaselineModel=GPT 4o2025.11 | 62.2 | |
| Zero-Shot PredictModel=GPT 4o2025.11 | 60.7 | |
| EngGPT2-16B-A3BGroup=Comparable, Configuration=Optimal serving and decoding2026.03 | 57.3 | |
| Hybrid-LoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=105M2026.05 | 56.9 | |
| Full-FT (ref.)Backbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=1544M2026.05 | 56.8 | |
| MIPROv2Model=Llama 3.3 70B2025.11 | 56.5 | |
| gemma-3-12b-itGroup=Larger, Configuration=Optimal serving and decoding2026.03 | 56 | |
| Full-FT (ref.)Backbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GRPO, #Tunable Params=1544M2026.05 | 55.6 | |
| Hybrid-LoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GRPO, #Tunable Params=105M2026.05 | 54.9 | |
| AutoLoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=107M2026.05 | 54.9 | |
| LoRA-DropBackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=105M2026.05 | 54.7 | |
| AdaLoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=106M2026.05 | 53.5 | |
| LoRA-DropBackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GRPO, #Tunable Params=105M2026.05 | 53.2 | |
| LoRA-FFNBackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GSPO, #Tunable Params=122M2026.05 | 53.1 | |
| AutoLoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GRPO, #Tunable Params=106M2026.05 | 52.1 | |
| AdaLoRABackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=GRPO, #Tunable Params=105M2026.05 | 50.5 | |
| Llama-3.1-8B-InstructGroup=Comparable, Configuration=Optimal serving and decoding2026.03 | 49.2 | |
| gemma-2-9b-itGroup=Comparable, Configuration=Optimal serving and decoding2026.03 | 48.2 | |
| Moonlight-16B-A3B-InstructGroup=Comparable, Configuration=Optimal serving and decoding2026.03 | 47.1 | |
| Full-FTBackbone=Qwen-2.5 1.5B Instruct, Post-training Approach=SFT, #Tunable Params=1544M2026.05 | 46.9 | |
| OWPOModel=Qwen-2.5-Math-7B-Base2026.05 | 46.85 | |
| Ours (+GDPO)Alignment=GDPO2026.05 | 45.2 | |
| Ours (+SFT)Alignment=SFT2026.05 | 45.14 | |
| STAIR2026.05 | 44.92 | |
| HELM BaselineModel=Qwen3 4B2025.11 | 44.9 | |
| DPO2026.05 | 44.52 | |
| Original2026.05 | 44.25 | |
| DAPOModel=Qwen-2.5-Math-7B-Base2026.05 | 44.09 | |
| Shallow-Align2026.05 | 43.9 | |
| float16Model=Qwen2.5-7B, Precision=float16, Evaluation Protocol (Shots)=5-shot2024.12 | 43.86 | |
| Self-Critique2026.05 | 43.85 | |
| MOPDModel=Qwen-2.5-Math-7B-Base2026.05 | 43.81 | |
| MixLLMModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=5-shot2024.12 | 43.8 | |
| OPDModel=Qwen-2.5-Math-7B-Base2026.05 | 43.34 | |
| MixLLMModel=Qwen2.5-7B, Precision=W4.4A8, Evaluation Protocol (Shots)=5-shot2024.12 | 43.26 | |
| SmoothQuantModel=Qwen2.5-7B, Precision=W8A8, Evaluation Protocol (Shots)=5-shot2024.12 | 42.98 | |
| Sym-DAPOModel=Qwen-2.5-Math-7B-Base2026.05 | 42.93 | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=5-shot2024.12 | 42.45 | |
| QServeModel=Qwen2.5-7B, Precision=W4A8, Evaluation Protocol (Shots)=5-shot2024.12 | 41.72 | |
| Zero-Shot PredictModel=Qwen3 4B2025.11 | 41.7 | |
| SFT2026.05 | 41.59 | |
| MixLLMModel=Mistral-7B-v0.3, Precision=W4A8, Evaluation Protocol (Shots)=5-shot2024.12 | 41.49 | |
| CoTBackbone=Qwen-2.5 3B Instruct, Post-training Approach=-, #Tunable Params=0.02026.05 | 40.4 | |
| GRPOModel=Qwen-2.5-Math-7B-Base2026.05 | 36.94 | |
| MixLLMModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=5-shot2024.12 | 35.54 | |
| float16Model=Average, Precision=float16, Evaluation Protocol (Shots)=5-shot2024.12 | 35.52 | |
| SmoothQuantModel=Average, Precision=W8A8, Evaluation Protocol (Shots)=5-shot2024.12 | 35.04 | |
| MixLLMModel=Average, Precision=W4.4A8, Evaluation Protocol (Shots)=5-shot2024.12 | 34.53 | |
| QuaRotModel=Qwen2.5-7B, Precision=W4A4, Evaluation Protocol (Shots)=5-shot2024.12 | 34.4 |