List your company
ResearchFormerly WizWand

Multi-task Language Understanding on MMLU (MMLU Score)

86.4MMLU ScoreGPT-4
23.937640.153856.3772.5862May 16, 2025Jul 17, 2025Sep 18, 2025Nov 19, 2025Jan 21, 2026Mar 24, 2026May 26, 2026
Updated 5mo ago

Evaluation results

MethodLinks
GPT-42026.0186.4
GRPOBackbone=LLaMA-3.1-8B2025.0575
TI-DPOBackbone=LLaMA-3.1-8B2025.0574
TPOBackbone=LLaMA-3.1-8B2025.0573
CPOBackbone=LLaMA-3.1-8B2025.0572
Dream-Instruct 7B + G-StarRefinement Strategy=Loop-based refinement2025.1071.2
KTOBackbone=LLaMA-3.1-8B2025.0571
GPT-3.52026.0170
DPOBackbone=LLaMA-3.1-8B2025.0570
Dream-Instruct 7BSource=Reproduced2025.1069.9
TDPOBackbone=LLaMA-3.1-8B2025.0569.5
SFTBackbone=LLaMA-3.1-8B2025.0569
SIMPOBackbone=LLaMA-3.1-8B2025.0568.5
IPOBackbone=LLaMA-3.1-8B2025.0568
Dream-Instruct 7BSource=Originally Published2025.1067
LLama-3-8BModel Size=8B, Tokens=15T2026.0266.2
baselineBackbone=LLaMA-2 70B, #Bits=FP16, #Eff. (w)=162026.0365.44
DenseAvg. bits=16, 5-shot=true2025.0565.29
baselineBackbone=Qwen-2.5-3B, #Bits=FP16, #Eff. (w)=162026.0365.12
SERQ (GPTQ)Backbone=LLaMA-2 70B, #Bits=W4A8, #Eff. (w)=3.432026.0364.62
L2QERBackbone=LLaMA-2 70B, #Bits=W4A8, #Eff. (w)=3.552026.0364.45
SERQ (RTN)Backbone=LLaMA-2 70B, #Bits=W4A8, #Eff. (w)=3.442026.0364.45
DCLM-7B-8kModel Size=7B, Tokens=2.8T2026.0263.7
SERQ (GPTQ)Backbone=Qwen-2.5-3B, #Bits=W4A8, #Eff. (w)=4.242026.0363.25
SERQ (RTN)Backbone=LLaMA-2 70B, #Bits=W4A4, #Eff. (w)=3.652026.0363.15
SERQ (GPTQ)Backbone=LLaMA-2 70B, #Bits=W4A4, #Eff. (w)=3.642026.0363.09
DCLM-Bas. + CC-TablesModel Size=7B, Tokens=2.7T, alpha=0.82026.0263
SERQ (RTN)Backbone=Qwen-2.5-3B, #Bits=W4A8, #Eff. (w)=4.242026.0362.79
DCLM-Bas. + CC-TablesModel Size=7B, Tokens=2.7T, alpha=0.852026.0262.7
DCLM-Bas. + CC-TablesModel Size=7B, Tokens=2.7T, alpha=0.92026.0262.6
DCLM-Bas. + CC-TablesModel Size=7B, Tokens=2.7T, alpha=12026.0262.3
baselineBackbone=LLaMA-3 8B, #Bits=FP16, #Eff. (w)=162026.0362.13
SERQ-MXFP4Backbone=LLaMA-2 70B, #Bits=W4A4, #Eff. (w)=3.792026.0361.64
L2QER-MXFP4Backbone=LLaMA-2 70B, #Bits=W4A4, #Eff. (w)=3.812026.0361.48
Qwen2.5-DenseLLM=Qwen2.5, Model=Dense, Activated Parameters=1.5 B2026.0561.1
SERQ (GPTQ)Backbone=LLaMA-3 8B, #Bits=W4A8, #Eff. (w)=6.522026.0360.25
SERQ (GPTQ)Backbone=Qwen-2.5-3B, #Bits=W4A4, #Eff. (w)=4.242026.0359.55
SERQ (RTN)Backbone=Qwen-2.5-3B, #Bits=W4A4, #Eff. (w)=4.242026.0359.52
SERQ (RTN)Backbone=LLaMA-3 8B, #Bits=W4A8, #Eff. (w)=6.712026.0358.49
L2QERBackbone=LLaMA-3 8B, #Bits=W4A8, #Eff. (w)=7.162026.0357.81
L2QERBackbone=LLaMA-2 70B, #Bits=W4A4, #Eff. (w)=4.552026.0356.44
DBF + PVAvg. bits=2.3, 5-shot=true2025.0556.37
AQLM + PVAvg. bits=2.3, 5-shot=true2025.0556.2
QTIPAvg. bits=2, 5-shot=true2025.0556.2
SERQ-MXFP4Backbone=Qwen-2.5-3B, #Bits=W4A4, #Eff. (w)=4.372026.0356.13
L2QER-MXFP4Backbone=Qwen-2.5-3B, #Bits=W4A4, #Eff. (w)=4.372026.0355.87
baselineBackbone=LLaMA-3.2 3B, #Bits=FP16, #Eff. (w)=162026.0354.06
L2QER-MXFP4Backbone=LLaMA-3 8B, #Bits=W4A4, #Eff. (w)=7.832026.0353.82
SERQ (GPTQ)Backbone=LLaMA-3 8B, #Bits=W4A4, #Eff. (w)=7.752026.0353.8
SERQ-MXFP4Backbone=LLaMA-3 8B, #Bits=W4A4, #Eff. (w)=7.632026.0353.48
resiliparseTraining Scale=7B-2x (276B Tokens), fastText Thresholds=0.112026.0252.9
DBF + PVAvg. bits=2, 5-shot=true2025.0552.67
Union (Random)Training Scale=7B-2x (276B Tokens), fastText Thresholds=(0.11, 0.11, 0.11)2026.0252.3
Vicuna-13B (all)#Token=370M2026.0152.1
SERQ (RTN)Backbone=LLaMA-3 8B, #Bits=W4A4, #Eff. (w)=8.072026.0351.84
resiliparseTraining Scale=7B-2x (276B Tokens), fastText Thresholds=0.152026.0251.7
Qwen2.5-Dense2MoELLM=Qwen2.5, Model=Ours, Activated Parameters=1.2 B2026.0551.67
SERQ (RTN)Backbone=LLaMA-3.2 3B, #Bits=W4A8, #Eff. (w)=4.242026.0351.45
Union (Manual)Training Scale=7B-2x (276B Tokens), fastText Thresholds=(0.11, 0.11, 0.11)2026.0251.4
L2QERBackbone=LLaMA-3.2 3B, #Bits=W4A8, #Eff. (w)=4.352026.0351.24
Union (Random)Training Scale=7B-2x (276B Tokens), fastText Thresholds=(0.11, 0.15, 0.15)2026.0251
Union (Manual)Training Scale=7B-2x (276B Tokens), fastText Thresholds=(0.11, 0.15, 0.15)2026.0250.8
Llama2-Dense2MoELLM=Llama2, Model=Ours, Activated Parameters=5.7 B2026.0550.8
SERQ (GPTQ)Backbone=LLaMA-3.2 3B, #Bits=W4A8, #Eff. (w)=4.242026.0350.24
Alpaca-13B#Token=4.4M2026.0148.1
resiliparseTraining Scale=7B-2x (276B Tokens), fastText Thresholds=0.182026.0247.4
LLaMA-13B#Token=1T2026.0147
L2QER-MXFP4Backbone=LLaMA-3.2 3B, #Bits=W4A4, #Eff. (w)=4.372026.0346.85
LLM.int4()Backbone=LLaMA-3 8B, #Bits=W4A8, #Eff. (w)=-†2026.0346.72
SERQ (RTN)Backbone=LLaMA-3.2 3B, #Bits=W4A4, #Eff. (w)=4.242026.0346.36
Llama2-DenseLLM=Llama2, Model=Dense, Activated Parameters=7.0 B2026.0545.8
SERQ (GPTQ)Backbone=LLaMA-3.2 3B, #Bits=W4A4, #Eff. (w)=4.242026.0345.7
LLM.int4()Backbone=LLaMA-2 70B, #Bits=W4A8, #Eff. (w)=-†2026.0343.56
Union (Manual)Training Scale=7B-1x (138B Tokens), fastText Thresholds=(0.11, 0.11, 0.11)2026.0243.5
baselineBackbone=LLaMA-2 7B, #Bits=FP16, #Eff. (w)=162026.0341.83
LLM.int4()Backbone=LLaMA-3.2 3B, #Bits=W4A8, #Eff. (w)=-†2026.0341.48
SERQ-MXFP4Backbone=LLaMA-3.2 3B, #Bits=W4A4, #Eff. (w)=4.372026.0341.33
SERQ (GPTQ)Backbone=LLaMA-2 7B, #Bits=W4A8, #Eff. (w)=4.242026.0340.29
SERQ (RTN)Backbone=LLaMA-2 7B, #Bits=W4A8, #Eff. (w)=4.242026.0340.21
resiliparseTraining Scale=7B-1x (138B Tokens), fastText Thresholds=0.112026.0239.5
L2QERBackbone=LLaMA-2 7B, #Bits=W4A8, #Eff. (w)=4.352026.0339.4
L2QERBackbone=LLaMA-3 8B, #Bits=W4A4, #Eff. (w)=11.442026.0338.33
SERQ (RTN)Backbone=LLaMA-2 7B, #Bits=W4A4, #Eff. (w)=4.242026.0338.03
trafilaturaTraining Scale=7B-1x (138B Tokens), fastText Thresholds=0.112026.0237.7
SERQ (GPTQ)Backbone=LLaMA-2 7B, #Bits=W4A4, #Eff. (w)=4.242026.0337.03
baselineBackbone=LLaMA-3.2 1B, #Bits=FP16, #Eff. (w)=162026.0336.76
SERQ-MXFP4Backbone=LLaMA-2 7B, #Bits=W4A4, #Eff. (w)=4.372026.0335.25
L2QER-MXFP4Backbone=LLaMA-2 7B, #Bits=W4A4, #Eff. (w)=4.372026.0335.22
jusTextTraining Scale=7B-1x (138B Tokens), fastText Thresholds=0.112026.0234.5
Union (Random)Training Scale=7B-1x (138B Tokens), fastText Thresholds=(0.11, 0.11, 0.11)2026.0233.6
SERQ (RTN)Backbone=LLaMA-3.2 1B, #Bits=W4A8, #Eff. (w)=4.242026.0332.77
L2QERBackbone=LLaMA-3.2 3B, #Bits=W4A4, #Eff. (w)=4.242026.0332.25
L2QERBackbone=LLaMA-3.2 1B, #Bits=W4A8, #Eff. (w)=4.352026.0331.23
L2QERBackbone=LLaMA-2 7B, #Bits=W4A4, #Eff. (w)=4.242026.0329.63
SERQ (RTN)Backbone=LLaMA-3.2 1B, #Bits=W4A4, #Eff. (w)=4.242026.0329.11
LLM.int4()Backbone=LLaMA-2 7B, #Bits=W4A8, #Eff. (w)=-†2026.0329.01
SERQ-MXFP4Backbone=LLaMA-3.2 1B, #Bits=W4A4, #Eff. (w)=4.372026.0327.23
SERQ (GPTQ)Backbone=LLaMA-3.2 1B, #Bits=W4A8, #Eff. (w)=4.242026.0327.1
L2QER-MXFP4Backbone=LLaMA-3.2 1B, #Bits=W4A4, #Eff. (w)=4.372026.0327.1
SERQ (GPTQ)Backbone=LLaMA-3.2 1B, #Bits=W4A4, #Eff. (w)=4.242026.0326.34