Language Modeling on Enwiki8
0.94BPCGPT2
Evaluation Results
| Method | Links | |
|---|---|---|
| GPT2#params=1542M2023.05 | 0.94 | |
| Focus#params=22M2023.05 | 0.94 | |
| Transformer XL#params=277M2023.05 | 0.99 | |
| GPT-2 1.5Bzero-shot=true, parameters=1.5B2023.05 | 1 | |
| Mega#params=39M2023.05 | 1.02 | |
| w/o Norm2019.11 | 1.04 | |
| GPT-2 762Mzero-shot=true, parameters=762M2023.05 | 1.04 | |
| Focus-H (ablation)#params=21M2023.05 | 1.06 | |
| LayerNorm2019.11 | 1.07 | |
| LayerNorm-simple2019.11 | 1.07 | |
| AdaNormk=1/102019.11 | 1.07 | |
| GPT-2 345Mzero-shot=true, parameters=345M2023.05 | 1.09 | |
| Transformer-XL 24BArchitecture=(sf)x12, Latency on A100 (ms)=12.52020.09 | 1.1 | |
| Sandwich Transformer 24BArchitecture=(s)x5 (sf)x7 (f)x5, Latency on A100 (ms)=12.52020.09 | 1.1 | |
| PAR Transformer 24BArchitecture=(sff)x5 (f)x9, Latency on A100 (ms)=8.42020.09 | 1.11 | |
| Plaid 1Bzero-shot=true, parameters=1.3B2023.05 | 1.18 | |
| Cluster-Formernumber of clusters=5122020.09 | 1.22 | |
| Cluster-Formernumber of clusters=2562020.09 | 1.24 | |
| GPT-2 124Mzero-shot=true, parameters=124M2023.05 | 1.24 | |
| Cluster-Formernumber of clusters=642020.09 | 1.28 | |
| Sparse Attention2020.09 | 1.29 | |
| Locality-Sensitive Hashing2020.09 | 1.33 | |
| Sliding window2020.09 | 1.34 |