List your company
ResearchFormerly WizWand

Multi-task Language Understanding on MMLU (Generic Score)

63.73MMLU ScoreMONA
-2.2902414.8496331.989549.12937May 26, 2026May 29, 2026Jun 2, 2026Jun 6, 2026Jun 10, 2026Jun 14, 2026Jun 18, 2026
Updated 4mo ago

Evaluation results

MethodLinks
MONAModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.0563.73
MuonModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.0562.81
AdamWModel=MOE-68B-A3B, Training Tokens=700B, Evaluation Mode=FewShot2026.0562.18
Llama-8B (unpruned)Model Backbone=Llama-8B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.667
Llama-8B (unpruned)Model Backbone=Llama-8B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.667
Qwen-3B (unpruned)Model Backbone=Qwen-3B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.644
Qwen-3B (unpruned)Model Backbone=Qwen-3B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.644
Qwen-3B (KOFF)Model Backbone=Qwen-3B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.61
Llama-8B (KOFF)Model Backbone=Llama-8B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.603
Llama-3B (unpruned)Model Backbone=Llama-3B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.56
Llama-3B (unpruned)Model Backbone=Llama-3B, Model Variant=unpruned, Global Sparsity Level=0%2026.050.56
Llama-3B (KOFF)Model Backbone=Llama-3B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.52
Llama-3B (KOFF)Model Backbone=Llama-3B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.484
Llama-8B (KOFF)Model Backbone=Llama-8B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.415
Qwen-3B (KOFF)Model Backbone=Qwen-3B, Model Variant=KOFF, Global Sparsity Level=12%2026.050.413
Llama-8B (pruning only)Model Backbone=Llama-8B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.408
QVecInference Precision=NF42026.060.396
AttackedInference Precision=LLM.int8()2026.060.391
OriginalInference Precision=FP322026.060.39
AttackedInference Precision=NF42026.060.388
GaussianInference Precision=LLM.int8()2026.060.387
Llama-8B (pruning only)Model Backbone=Llama-8B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.384
OriginalInference Precision=FP322026.060.364
QVecInference Precision=LLM.int8()2026.060.361
OriginalInference Precision=FP322026.060.36
GaussianInference Precision=NF42026.060.335
AttackedInference Precision=FP42026.060.333
Qwen-3B (pruning only)Model Backbone=Qwen-3B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.329
GaussianInference Precision=FP42026.060.328
Qwen-3B (pruning only)Model Backbone=Qwen-3B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.326
Llama-3B (pruning only)Model Backbone=Llama-3B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.316
QVecInference Precision=FP42026.060.301
Llama-3B (pruning only)Model Backbone=Llama-3B, Model Variant=pruning only, Global Sparsity Level=12%2026.050.249