List your company
ResearchFormerly WizWand

Multi-task Language Understanding on MMLU Pro (Accuracy)

96.8AccuracyPass@100
3.82427.96252.176.238Apr 23, 2025Jun 25, 2025Aug 27, 2025Oct 30, 2025Jan 1, 2026Mar 5, 2026May 8, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Pass@100Generator Model=Llama 3.1 8B Instruct, Supervision Setting=Oracle2026.0496.8
Pass@100Generator Model=Llama 3.3 70B Instruct, Supervision Setting=Oracle2026.0492
LogisticGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Supervised2026.0491.8
FUSEGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Unsupervised2026.0491.4
Naive BayesGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Supervised2026.0491.4
WeaverGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Supervised2026.0490.2
Naive EnsembleGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Unsupervised2026.0487
OBV (Oracle Best Verifier)Generator Model=Llama 3.3 70B Instruct, Supervision Setting=Supervised2026.0485.6
Qwen3-30B-A3BModel Scale=30B, Active Parameters=3.3A, Teacher Model Status=False2026.0581.11
NanoV3-30B*Model Scale=30B, Active Parameters=3.6A, Teacher Model Status=True2026.0578.86
NanoV3 Elastic-30BModel Scale=30B, Active Parameters=3.6A, Teacher Model Status=False2026.0578.63
NanoV3 Elastic-23BModel Scale=23B, Active Parameters=2.8A, Teacher Model Status=False2026.0576.07
Majority VoteGenerator Model=Llama 3.3 70B Instruct, Supervision Setting=Unsupervised2026.0474.4
Pass@1Generator Model=Llama 3.3 70B Instruct, Supervision Setting=Unsupervised2026.0469.9
FUSEGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Unsupervised2026.0469.8
NanoV3 Elastic-12BModel Scale=12B, Active Parameters=2.0A, Teacher Model Status=False2026.0568.28
Naive EnsembleGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Unsupervised2026.0467.2
WeaverGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Supervised2026.0467.2
OBV (Oracle Best Verifier)Generator Model=Llama 3.1 8B Instruct, Supervision Setting=Supervised2026.0466.5
Llama 3.1 InstructModel Scale=70B2025.0466.3
Naive BayesGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Supervised2026.0466
LogisticGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Supervised2026.0465.2
Llama 3 InstructModel Scale=70B2025.0463.2
Qwen3 ModelParameters=8B2025.0963.2
ParamΔModel Scale=70B2025.0462.1
SPELLBase Model=Qwen2.5-32B2025.0960.22
SPELLBase Model=Qwen2.5-14B2025.0958.86
Majority VoteGenerator Model=Llama 3.1 8B Instruct, Supervision Setting=Unsupervised2026.0456.4
AAPAParameters=4B2025.0956.35
Llama 3 BaseModel Scale=70B2025.0454
Llama 3.1 BaseModel Scale=70B2025.0451.3
SPELLBase Model=Qwen2.5-7B2025.0949.78
Qwen2.5-32B2025.0948.89
Llama 3.1 InstructModel Scale=8B2025.0448.6
Qwen2.5-14B2025.0946.67
Pass@1Generator Model=Llama 3.1 8B Instruct, Supervision Setting=Unsupervised2026.0446.6
Llama 3 InstructModel Scale=8B2025.0445.5
ParamΔModel Scale=8B2025.0445.5
Qwen2.5-7B2025.0940.24
Qwen3 ModelParameters=235B-25072025.0939.45
Llama 3.1 BaseModel Scale=8B2025.0436.4
Qwen3 ModelParameters=4B2025.0935.56
Qwen3 ModelParameters=4B-25072025.0933.1
Llama 3 BaseModel Scale=8B2025.0433
AAPAParameters=0.6B2025.0927.08
Qwen3 ModelParameters=0.6B2025.0923.63
Qwen3 ModelParameters=1.7B2025.0920.53
Qwen3 ModelParameters=32B2025.0920.13
DenseBase Model=Llama-3.2-3B, Model architecture type=Dense, Few-shot=true2025.0619.6
Reg. MoE# train tokens=1T, Active parameters=1B2026.0519.3
MoBBase Model=Llama-3.2-3B, Model architecture type=MoB, Few-shot=true2025.0619.1
MiCRoBase Model=Llama-3.2-3B, Model architecture type=MiCRo, Few-shot=true2025.0619
OLMoE# train tokens=5T, Active parameters=1B2026.0518.7
EMO# train tokens=1T, Active parameters=1B2026.0518.5
Reg. MoE# train tokens=130B, Active parameters=1B2026.0515.8
EMO# train tokens=130B, Active parameters=1B2026.0515.5
Dense# train tokens=130B, Active parameters=1B2026.0512.2
DenseBase Model=Llama-3.2-1B, Model architecture type=Dense, Few-shot=true2025.0611.2
MoBBase Model=Llama-3.2-1B, Model architecture type=MoB, Few-shot=true2025.0611
MiCRoBase Model=Llama-3.2-1B, Model architecture type=MiCRo, Few-shot=true2025.0610.7
MiCRoBase Model=SmollM2-360M, Model architecture type=MiCRo, Few-shot=true2025.0610.1
DenseBase Model=SmollM2-360M, Model architecture type=Dense, Few-shot=true2025.069.9
MoBBase Model=SmollM2-360M, Model architecture type=MoB, Few-shot=true2025.069.8
MiCRoBase Model=SmollM2-135M, Model architecture type=MiCRo, Few-shot=true2025.067.9
DenseBase Model=SmollM2-135M, Model architecture type=Dense, Few-shot=true2025.067.8
MoBBase Model=SmollM2-135M, Model architecture type=MoB, Few-shot=true2025.067.4