List your company
ResearchFormerly WizWand

Multitask Language Understanding on MMLU Pro (pass@1)

86.7pass@1Qwen 3.5
-2.7524820.4707643.69466.91724Aug 27, 2024Dec 8, 2024Mar 22, 2025Jul 4, 2025Oct 15, 2025Jan 27, 2026May 11, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Qwen 3.5Zero-shot=true, Parameters=122B-A10B, Organization=Alibaba Cloud2026.0586.7
Nemotron 3 SuperZero-shot=true, Parameters=120B-A12B, Organization=Nvidia2026.0583.73
Llama 4 MaverickZero-shot=true, Parameters=400B-A17B, Organization=MetaAI2026.0580.62
Phoenix-VL 1.5 MediumZero-shot=true, Parameters=123B, Organization=HTX2026.0576.81
GLM-4.5VZero-shot=true, Parameters=106B-A12B, Organization=Z.ai2026.0572.17
UNA-score (MSE)Base Model=Qwen 8B, Alignment Strategy=UNA-score (MSE)2024.0848.94
DPOBase Model=Qwen 8B, Alignment Strategy=DPO2024.0847.48
UNA-pairwiseBase Model=Qwen 8B, Alignment Strategy=UNA-pairwise2024.0847.48
Qwen 8BBase Model=Qwen 8B, Alignment Strategy=Baseline2024.0847.21
KTOBase Model=Qwen 8B, Alignment Strategy=KTO2024.0847.18
UNA-binary (BCE)Base Model=Qwen 8B, Alignment Strategy=UNA-binary (BCE)2024.0846.89
UNA-score & binaryBase Model=Qwen 8B, Alignment Strategy=UNA-score & binary2024.0844.83
UNA-score (MSE)Base Model=Llama 8B, Alignment Strategy=UNA-score (MSE)2024.0834.42
UNA-score & binaryBase Model=Llama 8B, Alignment Strategy=UNA-score & binary2024.0834.25
DPOBase Model=Llama 8B, Alignment Strategy=DPO2024.0833.05
UNA-pairwiseBase Model=Llama 8B, Alignment Strategy=UNA-pairwise2024.0833.05
UNA-binary (BCE)Base Model=Llama 8B, Alignment Strategy=UNA-binary (BCE)2024.0833.01
KTOBase Model=Llama 8B, Alignment Strategy=KTO2024.0832.86
Llama 8BBase Model=Llama 8B, Alignment Strategy=Baseline2024.0832.73
UNA-binary (BCE)Base Model=Mistral 7B, Alignment Strategy=UNA-binary (BCE)2024.0830.73
KTOBase Model=Mistral 7B, Alignment Strategy=KTO2024.0830.43
DPOBase Model=Mistral 7B, Alignment Strategy=DPO2024.0830.41
UNA-pairwiseBase Model=Mistral 7B, Alignment Strategy=UNA-pairwise2024.0830.41
Mistral 7BBase Model=Mistral 7B, Alignment Strategy=Baseline2024.0830.11
UNA-score & binaryBase Model=Mistral 7B, Alignment Strategy=UNA-score & binary2024.0830.09
UNA-score (MSE)Base Model=Mistral 7B, Alignment Strategy=UNA-score (MSE)2024.0829.72
UNA-score (MSE)Base Model=Gemma 4B, Alignment Strategy=UNA-score (MSE)2024.0828.72
UNA-score & binaryBase Model=Gemma 4B, Alignment Strategy=UNA-score & binary2024.0828.49
DPOBase Model=Gemma 4B, Alignment Strategy=DPO2024.0828.03
UNA-pairwiseBase Model=Gemma 4B, Alignment Strategy=UNA-pairwise2024.0828.03
KTOBase Model=Gemma 4B, Alignment Strategy=KTO2024.0827.98
UNA-binary (BCE)Base Model=Gemma 4B, Alignment Strategy=UNA-binary (BCE)2024.0827.95
Gemma 4BBase Model=Gemma 4B, Alignment Strategy=Baseline2024.0827.92
InfLLM-v2Model Size=14B, Temperature=0.6, Max output length=32k, Training context length=32k, Training tokens=1.2T, Sampling strategy=pass@12026.010.793
SPLAModel Size=14B, Temperature=0.6, Max output length=32k, Training context length=32k, Training tokens=1.2T, Sampling strategy=pass@12026.010.793
SPAModel Size=14B, Temperature=0.6, Max output length=32k, Training context length=32k, Training tokens=1.2T, Sampling strategy=pass@12026.010.791
Dense AttentionModel Size=14B, Temperature=0.6, Max output length=32k, Training context length=32k, Training tokens=1.2T, Sampling strategy=pass@12026.010.789
NSAModel Size=14B, Temperature=0.6, Max output length=32k, Training context length=32k, Training tokens=1.2T, Sampling strategy=pass@12026.010.688