Character-level language modeling on text8
0.98BPCGPT2
Evaluation results
| Method | Links | |
|---|---|---|
| GPT2#params=1542M2023.05 | 0.98 | |
| Focus#params=22M2023.05 | 0.98 | |
| 24L Transformer-XLNumber of Parameters=277M2019.01 | 1.08 | |
| Transformer XL#params=277M2023.05 | 1.08 | |
| Focus-H (ablation)#params=21M2023.05 | 1.1 | |
| 64L TransformerNumber of Parameters=235M2019.01 | 1.13 | |
| 12L TransformerNumber of Parameters=44M2019.01 | 1.18 | |
| Transformer-XL 24BArchitecture=(sf)x12, Latency on A100 (ms)=12.52020.09 | 1.18 | |
| Sandwich Transformer 24BArchitecture=(s)x3 (sf)x9 (f)x3, Latency on A100 (ms)=12.52020.09 | 1.18 | |
| PAR Transformer 24BArchitecture=(sff)x5 (f)x9, Latency on A100 (ms)=8.42020.09 | 1.18 | |
| RHNNumber of Parameters=45M2019.01 | 1.27 | |
| Large mLSTMNumber of Parameters=45M2019.01 | 1.27 | |
| LN HM-LSTMNumber of Parameters=35M2019.01 | 1.29 | |
| HM-LSTMSize=>12M2018.03 | 1.29 | |
| BN-LSTM2019.01 | 1.36 | |
| TCNSize=4.6M2018.03 | 1.45 |