List your company
ResearchFormerly WizWand

Massive Multitask Language Understanding on MMLU (Overall Accuracy)

84.2MMLU AccuracyMARI
20.65637.15353.6570.147Nov 14, 2025Dec 16, 2025Jan 17, 2026Feb 19, 2026Mar 23, 2026Apr 24, 2026May 27, 2026
Updated 5mo ago

Evaluation results

MethodLinks
MARIModel=Qwen2.5-32B2026.0584.2
ITIModel=Qwen2.5-32B2026.0583.7
NL-ITIModel=Qwen2.5-32B2026.0583.6
VanillaModel=Qwen2.5-32B2026.0583.5
ReFTModel=Qwen2.5-32B2026.0583.4
CAAModel=Qwen2.5-32B2026.0583.2
MARIModel=Qwen2.5-14B2026.0581.6
ITIModel=Qwen2.5-14B2026.0581.2
NL-ITIModel=Qwen2.5-14B2026.0581
VanillaModel=Qwen2.5-14B2026.0580.9
ReFTModel=Qwen2.5-14B2026.0580.8
CAAModel=Qwen2.5-14B2026.0580.6
Qwen-2.5-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1176.6
Alpaca 50kSFT training data category=Trained on general SFT data only2026.0576.58
+ Safety 4k + Feature synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.0576.58
+ Safety 120kSFT training data category=Trained on general SFT data + safety SFT data2026.0576.34
+ Safety 8kSFT training data category=Trained on general SFT data + safety SFT data2026.0576.25
+ Safety 40kSFT training data category=Trained on general SFT data + safety SFT data2026.0576.08
+ Safety 4k + Random synth 4kSFT training data category=Trained on general SFT data + safety SFT data + SAE synthetic data2026.0576.08
+ Safety 200kSFT training data category=Trained on general SFT data + safety SFT data2026.0576
MARIModel=Qwen2-7B2026.0573.1
ITIModel=Qwen2-7B2026.0572.6
NL-ITIModel=Qwen2-7B2026.0572.5
VanillaModel=Qwen2-7B2026.0572.4
ReFTModel=Qwen2-7B2026.0572.3
CAAModel=Qwen2-7B2026.0572.1
Llama-3.1-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1171.3
Gemma-2-9B-itDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1169.1
Ministral-8B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1168.5
Tulu 3 8BDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1168.2
MARIModel=Llama-3-8B2026.0566.6
ITIModel=Llama-3-8B2026.0566.1
ReFTModel=Llama-3-8B2026.0566
VanillaModel=Llama-3-8B2026.0565.9
NL-ITIModel=Llama-3-8B2026.0565.8
CAAModel=Llama-3-8B2026.0565.5
OLMo-2-7B-1124-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1163.9
OLMo-2-7B-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1162
MAP-Neo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1156.5
OLMoE-1B-7B-0924-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1151.6
EvaByte-SFTDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1149.5
OLMo-v1.7-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1148.9
OLMo-7B-InstructDecoding Method=Medusa-style tree-based greedy decoding, Decoding Head=multi-token head2025.1146.3
MARIModel=Llama-2-13B2026.0526.7
ITIModel=Llama-2-13B2026.0526.3
NL-ITIModel=Llama-2-13B2026.0526.1
VanillaModel=Llama-2-13B2026.0526
ReFTModel=Llama-2-13B2026.0525.9
CAAModel=Llama-2-13B2026.0525.7
ITIModel=Llama-2-7B2026.0523.6
NL-ITIModel=Llama-2-7B2026.0523.4
VanillaModel=Llama-2-7B2026.0523.3
ReFTModel=Llama-2-7B2026.0523.2
MARIModel=Llama-2-7B2026.0523.2
CAAModel=Llama-2-7B2026.0523.1