Character-level language modeling on Enwik8 (val)
1.04BPCAdaptive Transformer
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Adaptive TransformerLayers=12, Heads=96, Params=39M2019.11 | 1.04 | — | — | — | |
| SHA-LSTMLayers=4, Heads=4, Params=54M, Hidden size (h)=1024, Attention=attention head per layer2019.11 | 1.096 | — | — | — | |
| SHA-LSTMLayers=4, Heads=1, Params=52M, Hidden size (h)=1024, Attention=single attention head2019.11 | 1.1 | — | — | — | |
| Mogrifier LSTMHeads=0, Params=48M2019.11 | 1.135 | — | — | — | |
| SSR-S w/ noisenoise=true2025.11 | 1.136 | — | — | — | |
| SSR-L w/ noisenoise=true2025.11 | 1.141 | — | — | — | |
| Sinkhorn-based SMoE2025.11 | 1.143 | — | — | — | |
| SMoE w/ aux. lossauxiliary loss=true2025.11 | 1.145 | — | — | — | |
| Vanilla SMoE2025.11 | 1.147 | — | — | — | |
| SMoE w/ noisenoise=true2025.11 | 1.154 | — | — | — | |
| LSTMHeads=0, Params=48M2019.11 | 1.182 | — | — | — | |
| SRU (with projection)Size=49m, #layers=12, Unroll size=256, Time=41min, projection dimension=5122017.09 | 1.19 | — | — | — | |
| SRU (with projection)Size=47m, #layers=8, Unroll size=100, Time=39min, projection dimension=5122017.09 | 1.21 | — | — | — | |
| SRU (with projection)Size=37m, #layers=6, Unroll size=100, Time=29min, projection dimension=5122017.09 | 1.25 | — | — | — | |
| SRUSize=37m, #layers=10, Unroll size=100, Time=29min2017.09 | 1.26 | — | — | — | |
| LSTMSize=46m, #layers=4, Unroll size=502017.09 | 1.28 | — | — | — | |
| SRUSize=37m, #layers=6, Unroll size=100, Time=28min2017.09 | 1.29 | — | — | — | |
| SHA-LSTMLayers=4, Heads=0, Params=51M, Hidden size (h)=1024, Attention=no attention head2019.11 | 1.312 | — | — | — | |
| LSTMSize=37m, #layers=6, Unroll size=100, Time=48min2017.09 | 1.35 | — | — | — | |
| QRNN (k=1)Size=37m, #layers=6, Unroll size=100, Time=30min2017.09 | 1.36 | — | — | — | |
| LSTMSize=37m, #layers=3, Unroll size=100, Time=42min2017.09 | 1.37 | — | — | — | |
| TSAα=0.833, TLOps saved=13.9%, ms/step=5992026.04 | 1.8429 | 1.2774 | — | — | |
| Baselineα=1.000, ms/step=5852026.04 | 1.8504 | 1.2826 | — | — | |
| HarmonicSeq. len=1,024, Token budget=65.5M tokens2026.05 | — | 6.571 | 1.4 | — | |
| HarmonicSeq. len=2,048, Token budget=65.5M tokens2026.05 | — | 6.426 | 3.5 | 2.1 | |
| HarmonicSeq. len=4,096, Token budget=65.5M tokens2026.05 | — | 6.687 | 5.1 | 1.6 | |
| HarmonicSeq. len=8,192, Token budget=65.5M tokens2026.05 | — | 6.333 | 6.7 | 1.6 | |
| HarmonicSeq. len=16,384, Token budget=65.5M tokens2026.05 | — | 6.196 | 9.9 | 3.2 | |
| HarmonicSeq. len=32,768, Token budget=131M2026.05 | — | 6.433 | 11.4 | 1.5 | |
| HarmonicSeq. len=65,536, Token budget=65.5M tokens2026.05 | — | 6.169 | — | — | |
| MambaSeq. len=1,024, Token budget=65.5M tokens2026.05 | — | 6.616 | — | — | |
| MambaSeq. len=2,048, Token budget=65.5M tokens2026.05 | — | 6.532 | — | — | |
| MambaSeq. len=4,096, Token budget=65.5M tokens2026.05 | — | 6.74 | — | — | |
| MambaSeq. len=8,192, Token budget=65.5M tokens2026.05 | — | 6.422 | — | — | |
| MambaSeq. len=16,384, Token budget=65.5M tokens2026.05 | — | 6.286 | — | — | |
| MambaSeq. len=32,768, Token budget=131M2026.05 | — | 6.549 | — | — | |
| TransformerSeq. len=1,024, Token budget=65.5M tokens2026.05 | — | 6.662 | — | — | |
| TransformerSeq. len=2,048, Token budget=65.5M tokens2026.05 | — | 6.657 | — | — | |
| TransformerSeq. len=4,096, Token budget=65.5M tokens2026.05 | — | 7.045 | — | — | |
| TransformerSeq. len=8,192, Token budget=65.5M tokens2026.05 | — | 6.787 | — | — | |
| TransformerSeq. len=16,384, Token budget=65.5M tokens2026.05 | — | 6.873 | — | — | |
| TransformerSeq. len=32,768, Token budget=131M2026.05 | — | 7.259 | — | — |