Multi-task Language Understanding on MMLU (Top-1 agreement)
74.7Top-1 AccuracyHARP-C
Evaluation Results
| Method | Links | |
|---|---|---|
| HARP-CModel=QWEN3-8B, Train Dataset=Tulu2026.06 | 74.7 | |
| Qwen2.5-7BModel Backbone=Qwen2.5-7B2026.06 | 74.2 | |
| Uni-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Unified energy-based decoding2026.06 | 66.5 | |
| LLaMA 3.1 8BCache Size=∞, 5-shot=true2026.07 | 65.25 | |
| OURS (GDN)Tok. (B)=0.01, Cache Size=128, Param.=10.3M, 5-shot=true2026.07 | 63.22 | |
| Uni-EnergyModel Backbone=Dream-7B, Decoding Strategy=Unified energy-based decoding2026.06 | 63.2 | |
| Ind-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Independent energy-based decoding2026.06 | 62.7 | |
| Inv-EnergyModel Backbone=Dream-7B, Decoding Strategy=Invariant energy-based decoding2026.06 | 62.7 | |
| Ind-EnergyModel Backbone=Dream-7B, Decoding Strategy=Independent energy-based decoding2026.06 | 62.4 | |
| COREModel Backbone=Dream-7B, Decoding Strategy=Remask decoding2026.06 | 62.2 | |
| Dream-7BModel Backbone=Dream-7B, Decoding Strategy=Base2026.06 | 61.8 | |
| HARP-EModel=LLAMA-8B, Train Dataset=SI2026.06 | 61.4 | |
| HARP-CModel=LLAMA-8B, Train Dataset=SI2026.06 | 61.3 | |
| APDModel Backbone=Dream-7B, Decoding Strategy=Speculative decoding2026.06 | 61.3 | |
| STILL (concurrent)Tok. (B)=0.04, Cache Size=NA, 5-shot=true2026.07 | 61.3 | |
| LizardTok. (B)=0.04, Cache Size=132, 5-shot=true2026.07 | 61.2 | |
| APDModel Backbone=LLaDa-8B, Decoding Strategy=Speculative decoding2026.06 | 60.5 | |
| LlambaTok. (B)=12, Cache Size=0, Param.=8B, 5-shot=true2026.07 | 60 | |
| Inv-EnergyModel Backbone=LLaDa-8B, Decoding Strategy=Invariant energy-based decoding2026.06 | 59.4 | |
| OURS (GDN)Tok. (B)=0.01, Cache Size=64, Param.=10.3M, 5-shot=true2026.07 | 59.17 | |
| DAWNModel Backbone=LLaDa-8B, Decoding Strategy=Dependency decoding2026.06 | 58.6 | |
| LoLaTok. (B)=0.04, Cache Size=128, 5-shot=true2026.07 | 57.6 | |
| LLaDa-8BModel Backbone=LLaDa-8B, Decoding Strategy=Base2026.06 | 57.3 | |
| COREModel Backbone=LLaDa-8B, Decoding Strategy=Remask decoding2026.06 | 56.5 | |
| LolCaTTok. (B)=0.04, Cache Size=64, 5-shot=true2026.07 | 54.88 | |
| RandModel=QWEN3-4B, Train Dataset=Wiz2026.06 | 54.1 | |
| HARP-EModel=QWEN3-4B, Train Dataset=Wiz2026.06 | 54 | |
| Qwen2.5-0.5BModel Backbone=Qwen2.5-0.5B2026.06 | 47.5 | |
| Liger-GLATok. (B)=0.02, Cache Size=64, 5-shot=true2026.07 | 46.9 | |
| SHED-OModel=LLAMA-8B, Train Dataset=SI2026.06 | 45.5 | |
| CLIMBSize=950M2025.04 | 36.47 | |
| Llama-3.2Size=1.2B2025.04 | 35.4 | |
| CLIMBSize=350M2025.04 | 33.28 | |
| Qwen2.5Size=490M2025.04 | 33.03 | |
| SmolLMSize=360M2025.04 | 32.98 | |
| AMD-OLMoSize=1.2B2025.04 | 31.92 | |
| TinyLlamaSize=1.1B2025.04 | 31.6 | |
| ADAMS (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 22.94 | |
| SCAPE (d = 0.01)Protocol=Zero-shot, Model=GPT-345M, d=0.012026.07 | 22.93 | |
| SCAPE (d = 0.1)Protocol=Zero-shot, Model=GPT-345M, d=0.12026.07 | 22.91 | |
| ADAMW (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 22.88 | |
| RandModel=LLAMA-8B, Train Dataset=SI2026.06 | 13.8 | |
| Full FTModel=LLAMA-8B, Train Dataset=SI2026.06 | 13.6 | |
| DQModel=LLAMA-8B, Train Dataset=SI2026.06 | 8.3 | |
| DSIRModel=LLAMA-8B, Train Dataset=SI2026.06 | 2.2 | |
| SHED-WModel=LLAMA-8B, Train Dataset=SI2026.06 | 0.3 |