Character-Level Language Modeling on text8 (held-out 1M tokens)
2.3BPCSHARP
Evaluation Results
| Method | Links | |
|---|---|---|
| SHARPT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active), Wall-time / 1k Tokens (sec)=9.46¹2026.05 | 2.3 | |
| SHARPT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=α^L, Params=9.7M (5M active), Wall-time / 1k Tokens (sec)=9.46¹2026.05 | 2.32 | |
| GRUT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=7.3M, Wall-time / 1k Tokens (sec)=14.292026.05 | 2.38 | |
| Transformer 5MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M, Wall-time / 1k Tokens (sec)=116.402026.05 | 2.38 | |
| GRUT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=7.3M, Wall-time / 1k Tokens (sec)=14.292026.05 | 2.41 | |
| Transformer 5MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=5.0M, Wall-time / 1k Tokens (sec)=116.402026.05 | 2.41 | |
| Transformer 10MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=266.532026.05 | 2.41 | |
| Transformer 10MT (BPTT window or input length)=1024, Time Complexity=O(T^2), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=266.532026.05 | 2.43 | |
| RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=2.4M, Wall-time / 1k Tokens (sec)=4.412026.05 | 2.45 | |
| RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=2.4M, Wall-time / 1k Tokens (sec)=4.412026.05 | 2.47 | |
| LSTMT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=20.502026.05 | 2.5 | |
| LSTMT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=9.7M, Wall-time / 1k Tokens (sec)=20.502026.05 | 2.61 | |
| Clockwork RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=4.3M, Wall-time / 1k Tokens (sec)=4.912026.05 | 2.92 | |
| Clockwork RNNT (BPTT window or input length)=4, Time Complexity=O(T), Effective Context=T, Params=4.3M, Wall-time / 1k Tokens (sec)=4.912026.05 | 2.95 |