Massive Multitask Language Understanding on MMLU (Overall Accuracy)
84.2MMLU AccuracyMARI
Evaluation Results
| Method | Links | |
|---|---|---|
| MARIModel=Qwen2.5-32B2026.05 | 84.2 | |
| ITIModel=Qwen2.5-32B2026.05 | 83.7 | |
| NL-ITIModel=Qwen2.5-32B2026.05 | 83.6 | |
| VanillaModel=Qwen2.5-32B2026.05 | 83.5 | |
| ReFTModel=Qwen2.5-32B2026.05 | 83.4 | |
| CAAModel=Qwen2.5-32B2026.05 | 83.2 | |
| MARIModel=Qwen2.5-14B2026.05 | 81.6 | |
| ITIModel=Qwen2.5-14B2026.05 | 81.2 | |
| NL-ITIModel=Qwen2.5-14B2026.05 | 81 | |
| VanillaModel=Qwen2.5-14B2026.05 | 80.9 | |
| ReFTModel=Qwen2.5-14B2026.05 | 80.8 | |
| CAAModel=Qwen2.5-14B2026.05 | 80.6 | |
| Qwen-2.5-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 76.6 | |
| Alpaca 50kSFT training data category=Trained on general SFT data only2026.05 | 76.58 | |
| + Safety 4k + Feature synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05 | 76.58 | |
| + Safety 120kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | 76.34 | |
| + Safety 8kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | 76.25 | |
| + Safety 40kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | 76.08 | |
| + Safety 4k + Random synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.05 | 76.08 | |
| + Safety 200kSFT training data category=Trained on general SFT data + safety SFT data2026.05 | 76 | |
| MARIModel=Qwen2-7B2026.05 | 73.1 | |
| ITIModel=Qwen2-7B2026.05 | 72.6 | |
| NL-ITIModel=Qwen2-7B2026.05 | 72.5 | |
| VanillaModel=Qwen2-7B2026.05 | 72.4 | |
| ReFTModel=Qwen2-7B2026.05 | 72.3 | |
| CAAModel=Qwen2-7B2026.05 | 72.1 | |
| Llama-3.1-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 71.3 | |
| Gemma-2-9B-itDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 69.1 | |
| Ministral-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 68.5 | |
| Tulu 3 8BDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 68.2 | |
| MARIModel=Llama-3-8B2026.05 | 66.6 | |
| ITIModel=Llama-3-8B2026.05 | 66.1 | |
| ReFTModel=Llama-3-8B2026.05 | 66 | |
| VanillaModel=Llama-3-8B2026.05 | 65.9 | |
| NL-ITIModel=Llama-3-8B2026.05 | 65.8 | |
| CAAModel=Llama-3-8B2026.05 | 65.5 | |
| OLMo-2-7B-1124-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 63.9 | |
| OLMo-2-7B-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 62 | |
| MAP-Neo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 56.5 | |
| OLMoE-1B-7B-0924-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 51.6 | |
| EvaByte-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 49.5 | |
| OLMo-v1.7-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 48.9 | |
| OLMo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.11 | 46.3 | |
| MARIModel=Llama-2-13B2026.05 | 26.7 | |
| ITIModel=Llama-2-13B2026.05 | 26.3 | |
| NL-ITIModel=Llama-2-13B2026.05 | 26.1 | |
| VanillaModel=Llama-2-13B2026.05 | 26 | |
| ReFTModel=Llama-2-13B2026.05 | 25.9 | |
| CAAModel=Llama-2-13B2026.05 | 25.7 | |
| ITIModel=Llama-2-7B2026.05 | 23.6 | |
| NL-ITIModel=Llama-2-7B2026.05 | 23.4 | |
| VanillaModel=Llama-2-7B2026.05 | 23.3 | |
| ReFTModel=Llama-2-7B2026.05 | 23.2 | |
| MARIModel=Llama-2-7B2026.05 | 23.2 | |
| CAAModel=Llama-2-7B2026.05 | 23.1 |