Language Modeling on LAMBADA (Accuracy, Perplexity)
56.2AccuracyMoE > <former
Evaluation results
| Method | Links | ||
|---|---|---|---|
| MoE > <formerModel Family=MoE2026.06 | 56.2 | 7.45 | |
| 2B > <formerModel Family=Dense2026.06 | 56.1 | 7.43 | |
| MoE baselineModel Family=MoE2026.06 | 56.1 | 7.78 | |
| 2B constant-widthModel Family=Dense2026.06 | 55.4 | 8.18 | |
| CARVE + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 52.37 | 10.29 | |
| GDN-2 + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 50.9 | 10.43 | |
| Mamba-3 MIMO + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 49.82 | 10.92 | |
| Gated DeltaNetModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 49.62 | 11.89 | |
| KDA + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 49.21 | 10.66 | |
| Mamba-3 SISO + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 49.19 | 10.65 | |
| CARVEModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 48.87 | 11.27 | |
| Gated DeltaNet + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 48.71 | 10.82 | |
| TransformerModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Attention Type=full attention2026.06 | 48.32 | 13.72 | |
| KDAModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 48.13 | 11.68 | |
| GDN-2Model Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 48.09 | 11.41 | |
| Mamba-2 + SWAModel Category=Hybrid, Parameter Scale=1.3B, Token Scale=100B, Sliding-Window Attention (SWA)=true2026.06 | 48.05 | 11.29 | |
| Mamba-3 MIMOModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 47.82 | 11.66 | |
| Mamba-2Model Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 45.24 | 12.38 | |
| Mamba-3 SISOModel Category=Recurrent, Parameter Scale=1.3B, Token Scale=100B2026.06 | 45.06 | 12.99 | |
| SCAPE (d = 0.1)Protocol=Zero-shot, Model=GPT-345M, d=0.12026.07 | 39.71 | — | |
| ADAMW (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 39.57 | — | |
| ADAMS (DENSE all-reduce)Protocol=Zero-shot, Model=GPT-345M, Strategy=DENSE all-reduce2026.07 | 39.51 | — | |
| SCAPE (d = 0.01)Protocol=Zero-shot, Model=GPT-345M, d=0.012026.07 | 39.06 | — | |
| ADAMWBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Communication Strategy=DENSE all-reduce2026.07 | 38.6 | — | |
| SCAPEBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Sparsity (d)=0.12026.07 | 38.39 | — | |
| ADAMSBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Communication Strategy=DENSE all-reduce2026.07 | 36.66 | — | |
| SCAPEBackbone=Llama-500M, Evaluation Protocol=Zero-shot, Sparsity (d)=0.012026.07 | 36.64 | — | |
| RegMix-DProxy models=128, Mode=Online2026.06 | 34.45 | — | |
| RegMix-DProxy models=512, Mode=Online2026.06 | 34.43 | — | |
| RegMix-DProxy models=512, Mode=Offline2026.06 | 34.23 | — | |
| CGLSBackbone=Llama-3.2-1B, Token Budget=20B2025.06 | 34.08 | — | |
| RegMix2026.06 | 32.68 | — | |
| BL-CBackbone=Llama-3.2-1B, Token Budget=20B2025.06 | 32.66 | — | |
| RegMix-DProxy models=128, Mode=Offline2026.06 | 32.63 | — | |
| DoReMi2026.06 | 31.58 | — | |
| BL-RBackbone=Llama-3.2-1B, Token Budget=20B2025.06 | 30.37 | — | |
| GELUEpoch=2, Number of layers=12L, Parameter-matched=true2026.06 | 27.5 | 78.1 | |
| NC-FFNEpoch=2, Number of layers=12L, Parameter-matched=true2026.06 | 27.3 | 85.5 | |
| NC-FFN + decay + gateLayers=12L, Parameter matching=true, Seed=Single2026.06 | 27.1 | 94.6 | |
| Human2026.06 | 27.09 | — | |
| NC-FFNEpoch=1, Number of layers=12L, Parameter-matched=true2026.06 | 26.2 | 102.4 | |
| NC-FFN ({∩, \})Layers=12L, Parameter matching=true, Seed=Single2026.06 | 26.2 | 102.4 | |
| GELUEpoch=1, Number of layers=12L, Parameter-matched=true2026.06 | 25.4 | 109.4 | |
| GELU baselineLayers=12L, Parameter matching=true, Seed=Single2026.06 | 25.4 | 109.4 | |
| NC-FFN + quant (no forgetting)Layers=12L, Parameter matching=true, Seed=Single2026.06 | 25.4 | 110.9 | |
| NC-FFN + decay (learned γ, λ)Layers=12L, Parameter matching=true, Seed=Single2026.06 | 25 | 110.3 | |
| CARTd=1024, Best R=6, zero-shot=true2026.05 | 23.02 | 164 | |
| CARTd=768, Best R=6, zero-shot=true2026.05 | 20.42 | 251 | |
| CARTd=512, Best R=6, zero-shot=true2026.05 | 16.15 | 507 | |
| CARTd=256, Best R=8, zero-shot=true2026.05 | 8.66 | 2,079 |