List your company
ResearchFormerly WizWand

Character-level language modeling on Enwik8 (val)

1.04BPCAdaptive Transformer
1.0075841.2263921.44521.664008Sep 8, 2017Feb 14, 2019Jul 22, 2020Dec 28, 2021Jun 5, 2023Nov 10, 2024Apr 18, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Adaptive TransformerLayers=12, Heads=96, Params=39M2019.111.04———
SHA-LSTMLayers=4, Heads=4, Params=54M, Hidden size (h)=1024, Attention=attention head per layer2019.111.096———
SHA-LSTMLayers=4, Heads=1, Params=52M, Hidden size (h)=1024, Attention=single attention head2019.111.1———
Mogrifier LSTMHeads=0, Params=48M2019.111.135———
SSR-S w/ noisenoise=true2025.111.136———
SSR-L w/ noisenoise=true2025.111.141———
Sinkhorn-based SMoE2025.111.143———
SMoE w/ aux. lossauxiliary loss=true2025.111.145———
Vanilla SMoE2025.111.147———
SMoE w/ noisenoise=true2025.111.154———
LSTMHeads=0, Params=48M2019.111.182———
SRU (with projection)Size=49m, #layers=12, Unroll size=256, Time=41min, projection dimension=5122017.091.19———
SRU (with projection)Size=47m, #layers=8, Unroll size=100, Time=39min, projection dimension=5122017.091.21———
SRU (with projection)Size=37m, #layers=6, Unroll size=100, Time=29min, projection dimension=5122017.091.25———
SRUSize=37m, #layers=10, Unroll size=100, Time=29min2017.091.26———
LSTMSize=46m, #layers=4, Unroll size=502017.091.28———
SRUSize=37m, #layers=6, Unroll size=100, Time=28min2017.091.29———
SHA-LSTMLayers=4, Heads=0, Params=51M, Hidden size (h)=1024, Attention=no attention head2019.111.312———
LSTMSize=37m, #layers=6, Unroll size=100, Time=48min2017.091.35———
QRNN (k=1)Size=37m, #layers=6, Unroll size=100, Time=30min2017.091.36———
LSTMSize=37m, #layers=3, Unroll size=100, Time=42min2017.091.37———
TSAα=0.833, TLOps saved=13.9%, ms/step=5992026.041.84291.2774——
Baselineα=1.000, ms/step=5852026.041.85041.2826——
HarmonicSeq. len=1,024, Token budget=65.5M tokens2026.05—6.5711.4—
HarmonicSeq. len=2,048, Token budget=65.5M tokens2026.05—6.4263.52.1
HarmonicSeq. len=4,096, Token budget=65.5M tokens2026.05—6.6875.11.6
HarmonicSeq. len=8,192, Token budget=65.5M tokens2026.05—6.3336.71.6
HarmonicSeq. len=16,384, Token budget=65.5M tokens2026.05—6.1969.93.2
HarmonicSeq. len=32,768, Token budget=131M2026.05—6.43311.41.5
HarmonicSeq. len=65,536, Token budget=65.5M tokens2026.05—6.169——
MambaSeq. len=1,024, Token budget=65.5M tokens2026.05—6.616——
MambaSeq. len=2,048, Token budget=65.5M tokens2026.05—6.532——
MambaSeq. len=4,096, Token budget=65.5M tokens2026.05—6.74——
MambaSeq. len=8,192, Token budget=65.5M tokens2026.05—6.422——
MambaSeq. len=16,384, Token budget=65.5M tokens2026.05—6.286——
MambaSeq. len=32,768, Token budget=131M2026.05—6.549——
TransformerSeq. len=1,024, Token budget=65.5M tokens2026.05—6.662——
TransformerSeq. len=2,048, Token budget=65.5M tokens2026.05—6.657——
TransformerSeq. len=4,096, Token budget=65.5M tokens2026.05—7.045——
TransformerSeq. len=8,192, Token budget=65.5M tokens2026.05—6.787——
TransformerSeq. len=16,384, Token budget=65.5M tokens2026.05—6.873——
TransformerSeq. len=32,768, Token budget=131M2026.05—7.259——