Zero-shot Language Modeling on WikiText-103 and LAMBADA
19.21WikiText-103 PerplexityQ-Delta
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Q-Deltanumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 19.21 | 15.19 | |
| Mamba2number of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 19.47 | 17.4 | |
| HAM Learned router + EDAScale=800M, KV Cache Usage=50%, Routing=Learned Router, Extension=EDA2026.03 | 19.49 | 14.9 | |
| Gated DeltaNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 19.61 | 15.32 | |
| TransformerScale=800M, KV Cache Usage=Full2026.03 | 19.66 | 13.44 | |
| DeltaNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 19.77 | 16.64 | |
| HAM Fixed τScale=800M, KV Cache Usage=50%, Threshold Type=Fixed2026.03 | 19.85 | 14.16 | |
| Mambanumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 19.89 | 16.98 | |
| HAM Learned τScale=800M, KV Cache Usage=50%, Threshold Type=Learned2026.03 | 19.94 | 15.67 | |
| HAM Learned routerScale=800M, KV Cache Usage=50%, Routing=Learned Router2026.03 | 20.15 | 13.96 | |
| GDN-GSAScale=800M2026.03 | 22.04 | 14.39 | |
| RetNetnumber of parameters=1.3B, training tokens=30B, evaluation mode=zero-shot2026.06 | 22.45 | 21.84 | |
| GDNScale=800M2026.03 | 22.91 | 15.41 | |
| Mamba2number of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 26.6 | 30.35 | |
| Q-Deltanumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 26.89 | 32.67 | |
| Mambanumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 27.5 | 31.2 | |
| Gated DeltaNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 27.82 | 36.27 | |
| DeltaNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 28.78 | 63.04 | |
| RetNetnumber of parameters=340M, training tokens=15B, evaluation mode=zero-shot2026.06 | 31.36 | 52.29 |