Character-level Language Modeling on text8 100M regime (Backward stream)
2.17Backward BPCTransformer (ctx=1024)
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M2026.05 | 2.17 | |
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M2026.05 | 2.2 | |
| SHARPT=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active)2026.05 | 2.33 | |
| GRUT=4, Time Complexity=O(T), Effective Context=T, Params=7.3M2026.05 | 2.56 | |
| Clockwork RNNT=4, Time Complexity=O(T), Effective Context=T, Params=4.3M2026.05 | 2.78 | |
| LSTMT=4, Time Complexity=O(T), Effective Context=T, Params=9.7M2026.05 | 3.14 | |
| RNNT=4, Time Complexity=O(T), Effective Context=T, Params=2.4M2026.05 | 4.63 |