List your company
ResearchFormerly WizWand

Language Modeling on Enwiki8

0.94BPCGPT2
0.9241.0321.141.248Nov 16, 2019Jun 18, 2020Jan 19, 2021Aug 22, 2021Mar 25, 2022Oct 26, 2022May 30, 2023
Updated 8mo ago

Evaluation results

MethodLinks
GPT2#params=1542M2023.050.94
Focus#params=22M2023.050.94
Transformer XL#params=277M2023.050.99
GPT-2 1.5Bzero-shot=true, parameters=1.5B2023.051
Mega#params=39M2023.051.02
w/o Norm2019.111.04
GPT-2 762Mzero-shot=true, parameters=762M2023.051.04
Focus-H (ablation)#params=21M2023.051.06
LayerNorm2019.111.07
LayerNorm-simple2019.111.07
AdaNormk=1/102019.111.07
GPT-2 345Mzero-shot=true, parameters=345M2023.051.09
Transformer-XL 24BArchitecture=(sf)x12, Latency on A100 (ms)=12.52020.091.1
Sandwich Transformer 24BArchitecture=(s)x5 (sf)x7 (f)x5, Latency on A100 (ms)=12.52020.091.1
PAR Transformer 24BArchitecture=(sff)x5 (f)x9, Latency on A100 (ms)=8.42020.091.11
Plaid 1Bzero-shot=true, parameters=1.3B2023.051.18
Cluster-Formernumber of clusters=5122020.091.22
Cluster-Formernumber of clusters=2562020.091.24
GPT-2 124Mzero-shot=true, parameters=124M2023.051.24
Cluster-Formernumber of clusters=642020.091.28
Sparse Attention2020.091.29
Locality-Sensitive Hashing2020.091.33
Sliding window2020.091.34