Multi-task Language Understanding on MMLU (MMLU, Avg.5)
80.2MMLU ScoreQWEN-3-30B MOE
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| QWEN-3-30B MOEPARAMS=30B, Variant=ORIGINAL2025.11 | 80.2 | — | |
| LLAMA-4-SCOUTPARAMS=109B, Variant=ORIGINAL2025.11 | 80 | — | |
| LLAMA-4-SCOUTPARAMS=109B, Variant=FCSD2025.11 | 75.9 | — | |
| QWEN-3-30B MOEPARAMS=30B, Variant=FCSD2025.11 | 74 | — | |
| Qwen-4B (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.05 | 72.43 | 66.42 | |
| LLAMA-4-SCOUTPARAMS=109B, Variant=SFT2025.11 | 69.7 | — | |
| Phi-mini (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.05 | 68.32 | 63.72 | |
| QWEN-3-30B MOEPARAMS=30B, Variant=SFT2025.11 | 63.1 | — | |
| BaselineBase Model=Qwen1.5-MoE, Weight Bits=162026.04 | 60.87 | 46.1 | |
| BaseModel=Mistral-7B2026.05 | 59.8 | — | |
| LCDDModel=Mistral-7B, Sparsity=59.34%2026.05 | 57 | — | |
| DEEPSEEK-V2-LITEPARAMS=16B, Variant=ORIGINAL2025.11 | 56.8 | — | |
| Llama-3B (teacher)Setting=Teachers (reference), Few-shot protocol=3-shot2026.05 | 55.97 | 46.93 | |
| SFTModel=DeepSeek-8B2026.05 | 55.3 | — | |
| LCDDModel=DeepSeek-8B, Sparsity=81.38%2026.05 | 54.8 | — | |
| PRISMBase model=Llama-2-13B2026.05 | 54.38 | — | |
| LESSBase model=Llama-2-13B2026.05 | 54.25 | — | |
| IF-GUIDEBase model=Llama-2-13B2026.05 | 54.19 | — | |
| BaseModel=DeepSeek-8B2026.05 | 54.1 | — | |
| TrigModel=Mistral-7B, Sparsity=59.34%2026.05 | 54.1 | — | |
| BaselineBase Model=OLMoE, Weight Bits=162026.04 | 53.4 | 33.45 | |
| CONCEPTINFBase model=Llama-2-13B2026.05 | 52.98 | — | |
| PROJDIFFBase model=Llama-2-13B2026.05 | 52.88 | — | |
| RandomBase model=Llama-2-13B2026.05 | 52.79 | — | |
| VECFILTERBase model=Llama-2-13B2026.05 | 52.33 | — | |
| DEEPSEEK-V2-LITEPARAMS=16B, Variant=FCSD2025.11 | 50.5 | — | |
| SFTModel=Mistral-7B2026.05 | 50.1 | — | |
| TrigModel=DeepSeek-8B, Sparsity=81.38%2026.05 | 49.2 | — | |
| GPTQBase Model=Qwen1.5-MoE, Weight Bits=32026.04 | 49.11 | 37.48 | |
| MoBiEBase Model=Qwen1.5-MoE, Weight Bits=1.352026.04 | 48.72 | 37.54 | |
| LCDDModel=Vicuna-7B, Sparsity=84.40%2026.05 | 48.6 | — | |
| BaseModel=Vicuna-7B2026.05 | 48.5 | — | |
| DEEPSEEK-V2-LITEPARAMS=16B, Variant=SFT2025.11 | 48 | — | |
| TrigModel=Vicuna-7B, Sparsity=84.40%2026.05 | 47.9 | — | |
| PRISMBase model=Llama-2-7B2026.05 | 47.43 | — | |
| IF-GUIDEBase model=Llama-2-7B2026.05 | 47.23 | — | |
| SFTModel=Vicuna-7B2026.05 | 47.1 | — | |
| LESSBase model=Llama-2-7B2026.05 | 47.07 | — | |
| X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Llama-3B, Few-shot protocol=3-shot2026.05 | 46.32 | 40.48 | |
| SFTModel=Qwen3-0.6B2026.05 | 46 | — | |
| X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Qwen-4B + Llama-3B, Few-shot protocol=3-shot2026.05 | 45.86 | 40.15 | |
| RandomBase model=Llama-2-7B2026.05 | 45.74 | — | |
| TrigModel=Qwen3-0.6B, Sparsity=81.05%2026.05 | 45.1 | — | |
| VECFILTERBase model=Llama-2-7B2026.05 | 45.1 | — | |
| PROJDIFFBase model=Llama-2-7B2026.05 | 44.7 | — | |
| X-Token (P-KL)Setting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.05 | 44.67 | 38.85 | |
| CONCEPTINFBase model=Llama-2-7B2026.05 | 44.15 | — | |
| X-TokenSetting=Multi-teacher, Teachers=Phi-mini + Qwen-4B, Few-shot protocol=3-shot2026.05 | 43.98 | 38.49 | |
| X-Token (H-KL)Setting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.05 | 43.93 | 39.18 | |
| Llama-3B → 1BSetting=Same tokenizer, Teacher=Llama-3B, Few-shot protocol=3-shot2026.05 | 43.83 | 38.4 | |
| GOLDSetting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.05 | 43.5 | 38.66 | |
| GPTQBase Model=OLMoE, Weight Bits=32026.04 | 43.08 | 24.85 | |
| GOLDSetting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.05 | 42.56 | 35.03 | |
| MoBiEBase Model=OLMoE, Weight Bits=1.462026.04 | 42.33 | 24.54 | |
| LCDDModel=Qwen3-0.6B, Sparsity=81.05%2026.05 | 42.2 | — | |
| ULDSetting=Cross tokenizer (single teacher), Teacher=Phi-mini, Few-shot protocol=3-shot2026.05 | 41.43 | 38.31 | |
| Continued pre-trainingSetting=No distillation, Few-shot protocol=3-shot2026.05 | 40.5 | 36.63 | |
| ULDSetting=Cross tokenizer (single teacher), Teacher=Qwen-4B, Few-shot protocol=3-shot2026.05 | 40.34 | 36.77 | |
| BaseModel=Qwen3-0.6B2026.05 | 40.3 | — | |
| ARB-LLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.04 | 40.19 | 30.6 | |
| BiLLMBase Model=Qwen1.5-MoE, Weight Bits=1.112026.04 | 39.01 | 28.8 | |
| NoWagBase Model=Qwen1.5-MoE, Weight Bits=2.042026.04 | 38.44 | 26.94 | |
| MoEQuantBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 34.75 | 22.56 | |
| Llama-1B (base)Setting=No distillation, Few-shot protocol=3-shot2026.05 | 32.05 | 33.96 | |
| MoHGE-14BTotal Parameters=14.122B, Activated Parameters of Experts=0.843B2026.04 | 31.62 | — | |
| MoE-14BTotal Parameters=16.760B, Activated Parameters of Experts=1.191B2026.04 | 31.18 | — | |
| DenseTotal Parameters=1.672B2026.04 | 30.78 | — | |
| ARB-LLMBase Model=OLMoE, Weight Bits=1.112026.04 | 30.17 | 16.97 | |
| AWQBase Model=Qwen1.5-MoE, Weight Bits=32026.04 | 28.3 | 15.1 | |
| BiLLMBase Model=OLMoE, Weight Bits=1.112026.04 | 28.01 | 14.85 | |
| GPTQBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 26.94 | 16.96 | |
| MoHGE-3BTotal Parameters=2.821B, Activated Parameters of Experts=0.295B2026.04 | 26.41 | — | |
| DenseTotal Parameters=0.807B2026.04 | 26.36 | — | |
| MoE-3BTotal Parameters=3.3614B, Activated Parameters of Experts=0.376B2026.04 | 26.22 | — | |
| Wanda (channel)Model=Qwen2-7B, Sparsity=50%2026.04 | 26 | 39.4 | |
| SCAR-ConnModel=Qwen2-7B, Sparsity=50%2026.04 | 26 | 42 | |
| NoWagBase Model=OLMoE, Weight Bits=2.042026.04 | 24.46 | 11.77 | |
| Wanda (channel)Model=Mistral-7B, Sparsity=50%2026.04 | 24 | 42.9 | |
| SCAR-ConnModel=Llama-2-7B, Sparsity=50%2026.04 | 24 | 47.4 | |
| Wanda (channel)Model=Llama-2-7B, Sparsity=50%2026.04 | 23 | 44.3 | |
| SCAR-ConnModel=Mistral-7B, Sparsity=50%2026.04 | 22 | 44.1 | |
| MoEQuantBase Model=OLMoE, Weight Bits=22026.04 | 20.15 | 9.69 | |
| AWQBase Model=Qwen1.5-MoE, Weight Bits=22026.04 | 15.22 | 12.1 | |
| GPTQBase Model=OLMoE, Weight Bits=22026.04 | 14.78 | 6.07 | |
| AWQBase Model=OLMoE, Weight Bits=32026.04 | 8.3 | 3.88 | |
| AWQBase Model=OLMoE, Weight Bits=22026.04 | 5.21 | 1.9 |