Word-level Language Modeling on WikiText-103 (test)
15.79PerplexityKNN-LM
Evaluation results
| Method | Links | |
|---|---|---|
| KNN-LM2019.11 | 15.79 | |
| Transformer-XL + RMS dynamic eval# of params=257M, Dynamic Evaluation=RMS2019.04 | 16.4 | |
| Transformer-XL + SGD dynamic eval# of params=257M, Dynamic Evaluation=SGD2019.04 | 17 | |
| Sandwich TransformerSandwich Coefficient (k)=62019.11 | 17.96 | |
| Transformer-XL*# of params=257M, batch size=12019.04 | 18.1 | |
| Transformer-XL Large#Param=257M2019.01 | 18.3 | |
| Transformer-XL Large#Params=257M2019.07 | 18.3 | |
| Transformer XL2019.11 | 18.3 | |
| BaselineNumber of runs=52019.11 | 18.63 | |
| Transformer-XL# Params=257M, Non-Embedding Model Size=224M, Memory=12.0GB, configuration=X-large, adaptive embed.=true, device=TPU2019.09 | 18.7 | |
| Baseline (Baevski and Auli, 2019)2019.11 | 18.7 | |
| Transformer + adaptive input# of params=247M2019.04 | 20.5 | |
| Adaptive Input#Param=247M, contemporary_work=true2019.01 | 20.5 | |
| All-attention network + adaptive span#Params=133M2019.07 | 20.6 | |
| DEQ-Transformer# Params=110M, Non-Embedding Model Size=70M, Memory=3.7GB, size=medium, adaptive embed.=true2019.09 | 23.2 | |
| Transformer-XL# Params=110M, Non-Embedding Model Size=72M, Memory=9.0GB, size=medium, layers=18, adaptive embed.=true2019.09 | 23.6 | |
| Transformer-XL Standard#Param=151M2019.01 | 24 | |
| Transformer-XL Standard#Params=151M2019.07 | 24 | |
| DEQ-Transformer# Params=172M, Non-Embedding Model Size=43M, Memory=2.7GB, size=medium2019.09 | 24.2 | |
| Transformer-XL# Params=165M, Non-Embedding Model Size=44M, Memory=8.5GB, size=medium, layers=162019.09 | 24.3 | |
| DEQ-TrellisNet# Params=180M, Non-Embedding Model Size=45M, Memory=3.3GB2019.09 | 29 | |
| TrellisNetSize=180M2018.10 | 29.19 | |
| 70-layer TrellisNet# Params=180M, Non-Embedding Model Size=45M, Memory=24.7GB, auxiliary loss=true2019.09 | 29.2 | |
| 70-layer TrellisNet# Params=180M, Non-Embedding Model Size=45M, Memory=5.2GB, gradient checkpointing=true2019.09 | 29.2 | |
| LSTM + hebbian + cache2019.04 | 29.9 | |
| Hebbian + Cache2019.01 | 29.9 | |
| LSTM + Hebbian + Cache2019.07 | 29.9 | |
| Relational Memory CoreSize=195M2018.10 | 31.6 | |
| Relational Memory Core# Params=195M, Non-Embedding Model Size=60M2019.09 | 31.6 | |
| DEQ-Transformer# Params=138M, Non-Embedding Model Size=4.5M, Memory=1.1GB, size=small2019.09 | 32.4 | |
| 4 layer QRNNNumber of layers=4, h=2500, Number of GPUs=12018.03 | 33 | |
| AWD-QRNNSize=159M2018.10 | 33 | |
| QRNN# of params=151M2019.04 | 33 | |
| QRNN#Param=151M2019.01 | 33 | |
| 4-layer QRNN#Params=151M2019.07 | 33 | |
| AWD-QRNN# Params=159M, Non-Embedding Model Size=51M, Memory=7.1GB2019.09 | 33 | |
| Uniform Refine LSTM (UR)gate=refine, initialization=uniform2019.10 | 34.6 | |
| Ordered Master LSTM (OM)gate=master, initialization=ordered, activation=cumax2019.10 | 34.7 | |
| Transformer-XL# Params=138M, Non-Embedding Model Size=4.5M, Memory=6.8GB, size=small, weight-tied layers=162019.09 | 34.9 | |
| Uniform LSTM (U-)initialization=uniform2019.10 | 34.9 | |
| Transformer-XL# Params=139M, Non-Embedding Model Size=4.9M, Memory=4.8GB, size=small, layers=42019.09 | 35.8 | |
| LSTM baseline2019.10 | 35.8 | |
| Chrono LSTMinitialization=chrono, T_max=112019.10 | 35.8 | |
| Chrono LSTMinitialization=chrono, T_max=82019.10 | 36.1 | |
| Chrono LSTMinitialization=chrono2019.10 | 36.4 | |
| Dauphin et al.Number of GPUs=42018.03 | 37.2 | |
| Gated Linear ConvNetSize=230M2018.10 | 37.2 | |
| GCNN-142019.04 | 37.2 | |
| GCNN-142019.01 | 37.2 | |
| Gated Linear ConvNet# Params=230M2019.09 | 37.2 | |
| LSTM + continuous cache2018.10 | 40.8 | |
| LSTM+ neural cache2019.04 | 40.8 | |
| LSTM + Neural cache2019.01 | 40.8 | |
| LSTM + Neural cache2019.07 | 40.8 | |
| Dauphin et al.Number of GPUs=12018.03 | 44.9 | |
| GCNN-82019.01 | 44.9 | |
| GCNN-82019.07 | 44.9 | |
| Generic TCNSize=150M2018.10 | 45.2 | |
| TCN2019.01 | 45.2 | |
| TCN2019.07 | 45.2 | |
| Generic TCN# Params=150M, Non-Embedding Model Size=34M2019.09 | 45.2 | |
| Grave et al.2018.03 | 48.7 | |
| LSTM2018.10 | 48.7 | |
| LSTM2019.01 | 48.7 | |
| LSTM2019.07 | 48.7 |