List your company
ResearchFormerly WizWand

Multi-task Language Understanding on MMLU (MMLU, Avg.5)

80.2MMLU ScoreQWEN-3-30B MOE
2.210422.457742.70562.9523Nov 14, 2025Dec 15, 2025Jan 15, 2026Feb 15, 2026Mar 18, 2026Apr 18, 2026May 20, 2026
Updated 5mo ago

Evaluation results

MethodLinks
QWEN-3-30B MOEPARAMS=30B, Variant=ORIGINAL2025.1180.2—
LLAMA-4-SCOUTPARAMS=109B, Variant=ORIGINAL2025.1180—
LLAMA-4-SCOUTPARAMS=109B, Variant=FCSD2025.1175.9—
QWEN-3-30B MOEPARAMS=30B, Variant=FCSD2025.1174—
Qwen-4B (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.0572.4366.42
LLAMA-4-SCOUTPARAMS=109B, Variant=SFT2025.1169.7—
Phi-mini (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.0568.3263.72
QWEN-3-30B MOEPARAMS=30B, Variant=SFT2025.1163.1—
BaselineBase Model=Qwen1.5-MoE, Weight Bits=162026.0460.8746.1
BaseModel=Mistral-7B2026.0559.8—
LCDDModel=Mistral-7B, Sparsity=59.34%2026.0557—
DEEPSEEK-V2-LITEPARAMS=16B, Variant=ORIGINAL2025.1156.8—
Llama-3B (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.0555.9746.93
SFTModel=DeepSeek-8B2026.0555.3—
LCDDModel=DeepSeek-8B, Sparsity=81.38%2026.0554.8—
PRISMBase model=Llama-2-13B2026.0554.38—
LESSBase model=Llama-2-13B2026.0554.25—
IF-GUIDEBase model=Llama-2-13B2026.0554.19—
BaseModel=DeepSeek-8B2026.0554.1—
TrigModel=Mistral-7B, Sparsity=59.34%2026.0554.1—
BaselineBase Model=OLMoE, Weight Bits=162026.0453.433.45
CONCEPTINFBase model=Llama-2-13B2026.0552.98—
PROJDIFFBase model=Llama-2-13B2026.0552.88—
RandomBase model=Llama-2-13B2026.0552.79—
VECFILTERBase model=Llama-2-13B2026.0552.33—
DEEPSEEK-V2-LITEPARAMS=16B, Variant=FCSD2025.1150.5—
SFTModel=Mistral-7B2026.0550.1—
TrigModel=DeepSeek-8B, Sparsity=81.38%2026.0549.2—
GPTQBase Model=Qwen1.5-MoE, Weight Bits=32026.0449.1137.48
MoBiEBase Model=Qwen1.5-MoE, Weight Bits=1.352026.0448.7237.54
LCDDModel=Vicuna-7B, Sparsity=84.40%2026.0548.6—
BaseModel=Vicuna-7B2026.0548.5—
DEEPSEEK-V2-LITEPARAMS=16B, Variant=SFT2025.1148—
TrigModel=Vicuna-7B, Sparsity=84.40%2026.0547.9—
PRISMBase model=Llama-2-7B2026.0547.43—
IF-GUIDEBase model=Llama-2-7B2026.0547.23—
SFTModel=Vicuna-7B2026.0547.1—
LESSBase model=Llama-2-7B2026.0547.07—
X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Llama-3B, Few-shot protocol=3-shot2026.0546.3240.48
SFTModel=Qwen3-0.6B2026.0546—
X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Qwen-4B + Llama-3B, Few-shot protocol=3-shot2026.0545.8640.15
RandomBase model=Llama-2-7B2026.0545.74—
TrigModel=Qwen3-0.6B, Sparsity=81.05%2026.0545.1—
VECFILTERBase model=Llama-2-7B2026.0545.1—
PROJDIFFBase model=Llama-2-7B2026.0544.7—
X-Token (P-KL)Setting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.0544.6738.85
CONCEPTINFBase model=Llama-2-7B2026.0544.15—
X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Qwen-4B, Few-shot protocol=3-shot2026.0543.9838.49
X-Token (H-KL)Setting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.0543.9339.18
Llama-3B → 1BSetting=Same tokenizer, Teacher=Llama-3B, Few-shot protocol=3-shot2026.0543.8338.4
GOLDSetting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.0543.538.66
GPTQBase Model=OLMoE, Weight Bits=32026.0443.0824.85
GOLDSetting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.0542.5635.03
MoBiEBase Model=OLMoE, Weight Bits=1.462026.0442.3324.54
LCDDModel=Qwen3-0.6B, Sparsity=81.05%2026.0542.2—
ULDSetting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.0541.4338.31
Continued pre-trainingSetting=No distillation, Few-shot protocol=3-shot2026.0540.536.63
ULDSetting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.0540.3436.77
BaseModel=Qwen3-0.6B2026.0540.3—
ARB-LLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.0440.1930.6
BiLLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.0439.0128.8
NoWagBase Model=Qwen1.5-MoE, Weight Bits=2.042026.0438.4426.94
MoEQuantBase Model=Qwen1.5-MoE, Weight Bits=22026.0434.7522.56
Llama-1B (base)Setting=No distillation, Few-shot protocol=3-shot2026.0532.0533.96
MoHGE-14BTotal Parameters=14.122B, Activated Parameters of Experts=0.843B2026.0431.62—
MoE-14BTotal Parameters=16.760B, Activated Parameters of Experts=1.191B2026.0431.18—
DenseTotal Parameters=1.672B2026.0430.78—
ARB-LLMBase Model=OLMoE, Weight Bits=1.112026.0430.1716.97
AWQBase Model=Qwen1.5-MoE, Weight Bits=32026.0428.315.1
BiLLMBase Model=OLMoE, Weight Bits=1.112026.0428.0114.85
GPTQBase Model=Qwen1.5-MoE, Weight Bits=22026.0426.9416.96
MoHGE-3BTotal Parameters=2.821B, Activated Parameters of Experts=0.295B2026.0426.41—
DenseTotal Parameters=0.807B2026.0426.36—
MoE-3BTotal Parameters=3.3614B, Activated Parameters of Experts=0.376B2026.0426.22—
Wanda (channel)Model=Qwen2-7B, Sparsity=50%2026.042639.4
SCAR-ConnModel=Qwen2-7B, Sparsity=50%2026.042642
NoWagBase Model=OLMoE, Weight Bits=2.042026.0424.4611.77
Wanda (channel)Model=Mistral-7B, Sparsity=50%2026.042442.9
SCAR-ConnModel=Llama-2-7B, Sparsity=50%2026.042447.4
Wanda (channel)Model=Llama-2-7B, Sparsity=50%2026.042344.3
SCAR-ConnModel=Mistral-7B, Sparsity=50%2026.042244.1
MoEQuantBase Model=OLMoE, Weight Bits=22026.0420.159.69
AWQBase Model=Qwen1.5-MoE, Weight Bits=22026.0415.2212.1
GPTQBase Model=OLMoE, Weight Bits=22026.0414.786.07
AWQBase Model=OLMoE, Weight Bits=32026.048.33.88
AWQBase Model=OLMoE, Weight Bits=22026.045.211.9