Character-Level Language Modeling on text8 (most recent 1M tokens)
2.23BPCSHARP
Evaluation Results
| Method | Links | |
|---|---|---|
| SHARPT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active), Wall-time / 1k Tokens (sec)=9.46¹2026.05 | 2.23 | |
| Transformer 5MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M, Wall-time / 1k Tokens (sec)=116.402026.05 | 2.33 | |
| GRUT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=7.3M, Wall-time / 1k Tokens (sec)=14.292026.05 | 2.34 | |
| Transformer 10MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=266.532026.05 | 2.37 | |
| RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=2.4M, Wall-time / 1k Tokens (sec)=4.412026.05 | 2.41 | |
| LSTMT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=20.502026.05 | 2.54 | |
| Clockwork RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=4.3M, Wall-time / 1k Tokens (sec)=4.912026.05 | 2.79 |