List your company
ResearchFormerly WizWand

Multi-task Language Understanding on MMLU (Top-1 agreement)

74.7Top-1 AccuracyHARP-C
-2.67617.41237.557.588Apr 17, 2025Jun 30, 2025Sep 13, 2025Nov 26, 2025Feb 9, 2026Apr 24, 2026Jul 8, 2026
Updated 3mo ago

Evaluation results

MethodLinks
HARP-CModel=QWEN3-8B, Train Dataset=Tulu2026.0674.7
Qwen2.5-7BModel Backbone=Qwen2.5-7B2026.0674.2
Uni-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Unified energy-based decoding2026.0666.5
LLaMA 3.1 8BCache Size=∞, 5-shot=true2026.0765.25
OURS (GDN)Tok. (B)=0.01, Cache Size=128, Param.=10.3M, 5-shot=true2026.0763.22
Uni-EnergyModel Backbone=Dream-7B, Decoding Strategy=Unified energy-based decoding2026.0663.2
Ind-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Independent energy-based decoding2026.0662.7
Inv-EnergyModel Backbone=Dream-7B, Decoding Strategy=Invariant energy-based decoding2026.0662.7
Ind-EnergyModel Backbone=Dream-7B, Decoding Strategy=Independent energy-based decoding2026.0662.4
COREModel Backbone=Dream-7B, Decoding Strategy=Remask decoding2026.0662.2
Dream-7BModel Backbone=Dream-7B, Decoding Strategy=Base2026.0661.8
HARP-EModel=LLAMA-8B, Train Dataset=SI2026.0661.4
HARP-CModel=LLAMA-8B, Train Dataset=SI2026.0661.3
APDModel Backbone=Dream-7B, Decoding Strategy=Speculative decoding2026.0661.3
STILL (concurrent)Tok. (B)=0.04, Cache Size=NA, 5-shot=true2026.0761.3
LizardTok. (B)=0.04, Cache Size=132, 5-shot=true2026.0761.2
APDModel Backbone=LLaDa-8B, Decoding Strategy=Speculative decoding2026.0660.5
LlambaTok. (B)=12, Cache Size=0, Param.=8B, 5-shot=true2026.0760
Inv-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Invariant energy-based decoding2026.0659.4
OURS (GDN)Tok. (B)=0.01, Cache Size=64, Param.=10.3M, 5-shot=true2026.0759.17
DAWNModel Backbone=LLaDa-8B, Decoding Strategy=Dependency decoding2026.0658.6
LoLaTok. (B)=0.04, Cache Size=128, 5-shot=true2026.0757.6
LLaDa-8BModel Backbone=LLaDa-8B, Decoding Strategy=Base2026.0657.3
COREModel Backbone=LLaDa-8B, Decoding Strategy=Remask decoding2026.0656.5
LolCaTTok. (B)=0.04, Cache Size=64, 5-shot=true2026.0754.88
RandModel=QWEN3-4B, Train Dataset=Wiz2026.0654.1
HARP-EModel=QWEN3-4B, Train Dataset=Wiz2026.0654
Qwen2.5-0.5BModel Backbone=Qwen2.5-0.5B2026.0647.5
Liger-GLATok. (B)=0.02, Cache Size=64, 5-shot=true2026.0746.9
SHED-OModel=LLAMA-8B, Train Dataset=SI2026.0645.5
CLIMBSize=950M2025.0436.47
Llama-3.2Size=1.2B2025.0435.4
CLIMBSize=350M2025.0433.28
Qwen2.5Size=490M2025.0433.03
SmolLMSize=360M2025.0432.98
AMD-OLMoSize=1.2B2025.0431.92
TinyLlamaSize=1.1B2025.0431.6
ADAMS (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.0722.94
SCAPE (d = 0.01)Protocol=Zero-shot, Model=GPT-345M, d=0.012026.0722.93
SCAPE (d = 0.1)Protocol=Zero-shot, Model=GPT-345M, d=0.12026.0722.91
ADAMW (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.0722.88
RandModel=LLAMA-8B, Train Dataset=SI2026.0613.8
Full FTModel=LLAMA-8B, Train Dataset=SI2026.0613.6
DQModel=LLAMA-8B, Train Dataset=SI2026.068.3
DSIRModel=LLAMA-8B, Train Dataset=SI2026.062.2
SHED-WModel=LLAMA-8B, Train Dataset=SI2026.060.3