Knowledge on MMLU, NaturalQuestions, and TriviaQA
60.1MMLU (5-shot)MobileMoE-L
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MobileMoE-LNact/Ntotal=922M/5.3B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 60.1 | 22.4 | 54.9 | |
| Qwen3.5 2BNact/Ntotal=1.9B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 57.4 | 8.8 | 28.1 | |
| MobileMoE-MNact/Ntotal=528M/2.8B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 53.9 | 17.5 | 46.6 | |
| OLMoE-1B-7BNact/Ntotal=1.3B/6.9B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 52.7 | 17.2 | 54.1 | |
| Qwen3.5 0.8BNact/Ntotal=749M, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 50.2 | 3.3 | 16.1 | |
| SmolLM2 1.7BNact/Ntotal=1.7B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 49.4 | 14.2 | 46 | |
| Llama 3.2 1BNact/Ntotal=1.2B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 46.1 | 14.4 | 38.4 | |
| OLMo 2 1BNact/Ntotal=1.5B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 42.9 | 12.4 | 37.6 | |
| MobileMoE-SNact/Ntotal=272M/1.3B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 42.6 | 10.8 | 30.2 | |
| Gemma 3 1BNact/Ntotal=1.0B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 40 | 7.6 | 23.4 | |
| MobileLLM-ProNact/Ntotal=1.1B, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 38.7 | 8.9 | 19.9 | |
| Gemma 3 270MNact/Ntotal=270M, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 26.5 | 2.8 | 9.1 | |
| SmolLM2 360MNact/Ntotal=362M, Model Type=Instruct-tuned, Evaluation Framework=lm-eval2026.05 | 25.9 | 6.4 | 20.4 |