Character-level Language Modeling on text8 100M regime (Forward split)
2.19Forward BPCTransformer (ctx=1024)
Evaluation results
| Method | Links | |
|---|---|---|
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M2026.05 | 2.19 | |
| Transformer (ctx=1024)T=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M2026.05 | 2.22 | |
| SHARPT=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active)2026.05 | 2.36 | |
| GRUT=4, Time Complexity=O(T), Effective Context=T, Params=7.3M2026.05 | 2.57 | |
| RNNT=4, Time Complexity=O(T), Effective Context=T, Params=2.4M2026.05 | 2.7 | |
| Clockwork RNNT=4, Time Complexity=O(T), Effective Context=T, Params=4.3M2026.05 | 2.88 | |
| LSTMT=4, Time Complexity=O(T), Effective Context=T, Params=9.7M2026.05 | 3.17 |