Character-level Language Modeling on text8 100M regime (Current stream split)
2.12Current BPCTransformer (ctx=1024)
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M2026.05 | 2.12 | |
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M2026.05 | 2.15 | |
| SHARPT=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active)2026.05 | 2.32 | |
| GRUT=4, Time Complexity=O(T), Effective Context=T, Params=7.3M2026.05 | 2.5 | |
| Clockwork RNNT=4, Time Complexity=O(T), Effective Context=T, Params=4.3M2026.05 | 2.85 | |
| LSTMT=4, Time Complexity=O(T), Effective Context=T, Params=9.7M2026.05 | 3.16 | |
| RNNT=4, Time Complexity=O(T), Effective Context=T, Params=2.4M2026.05 | 4.63 |