Language Modeling on Penn Treebank (test)
46.34PerplexityGL-LWGC-AWD-MOS-LSTM + dynamic evaluation
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| GL-LWGC-AWD-MOS-LSTM + dynamic evaluationSize=26M, dynamic evaluation=true2017.08 | 46.34 | — | — | — | — | |
| AWD-LSTM-DOC (fin) x 5#Param=114M, fine-tuning=true, ensemble=true, ensemble_count=52018.08 | 47.17 | — | — | — | — | |
| AWD-LSTM-MoS+PDR + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 47.3 | — | — | — | — | |
| Ensemble of AllWeight=12023.11 | 47.31 | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluationSize=22M, dynamic evaluation=true2017.08 | 47.69 | — | — | — | — | |
| AWD-LSTM-MoS#Param=22M, dynamic evaluation=true2017.11 | 47.69 | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluationParameters=22M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 47.69 | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 47.7 | — | — | — | — | |
| AWD-LSTM-DOC x 5#Param=114M, ensemble=true, ensemble_count=52018.08 | 48.44 | — | — | — | — | |
| AWD-LSTM+PDR + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 49.3 | — | — | — | — | |
| AWD-LSTM-DRILL + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 49.4 | — | — | — | — | |
| AWD-LSTM + dynamic evaluationSize=24M, dynamic evaluation=true2017.08 | 51.1 | — | — | — | — | |
| AWD-LSTM + dynamic evalparameters=24M2017.09 | 51.1 | — | — | — | — | |
| AWD-LSTM#Param=24M, dynamic evaluation=true2017.11 | 51.1 | — | — | — | — | |
| AWD-LSTM + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 51.1 | — | — | — | — | |
| AWD-LSTM + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 51.1 | — | — | — | — | |
| AWD-LSTM+PDR + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 51.6 | — | — | — | — | |
| DOC + Finetune* + Partial ShufflePartial Shuffle=true, Finetune=repeated 3 times2019.03 | 52 | — | — | — | — | |
| AWD-LSTM-DOC (fin)#Param=23M, fine-tuning=true2018.08 | 52.38 | — | — | — | — | |
| DOC + Finetune*Partial Shuffle=false, Finetune=repeated 3 times2019.03 | 52.38 | — | — | — | — | |
| AWD-LSTM-DOCWeight=0.342023.11 | 52.38 | — | — | — | — | |
| GL-LWGC-AWD-MOS-LSTM + finetuneSize=26M, finetune=true2017.08 | 52.57 | — | — | — | — | |
| DOC + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.03 | 52.58 | — | — | — | — | |
| AWD-LSTM - 3-layer LSTM (tied) + continuous cache pointerParameters=24M2017.08 | 52.8 | — | — | — | — | |
| AWD-LSTM + neural cacheparameters=24M2017.09 | 52.8 | — | — | — | — | |
| AWD-LSTM#Param=24M, continuous cache pointer=true, dynamic evaluation=true2017.11 | 52.8 | — | — | — | — | |
| AWD-LSTM + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 52.8 | — | — | — | — | |
| AWD-LSTM + continuous cache pointerParameters=24M, Softmax Type=Multiple2019.05 | 52.8 | — | — | — | — | |
| AWD-LSTM-DOC#Param=23M2018.08 | 52.87 | — | — | — | — | |
| DOC + FinetunePartial Shuffle=false, Finetune=true2019.03 | 52.87 | — | — | — | — | |
| GL-LWGC-AWD-MOS-LSTMSize=26M2017.08 | 52.95 | — | — | — | — | |
| DOC + Partial ShufflePartial Shuffle=true, Finetune=false2019.03 | 53.28 | — | — | — | — | |
| DOCPartial Shuffle=false, Finetune=false2019.03 | 53.44 | — | — | — | — | |
| 2-layers GL-LWGC-AWD-MOS-LSTM + finetuneSize=19M, layers=2, finetune=true2017.08 | 53.54 | — | — | — | — | |
| AWD-LSTM-MoS+PDR#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 53.8 | — | — | — | — | |
| MoS + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.03 | 53.92 | — | — | — | — | |
| AWD-LSTM-MOSWeight=0.192023.11 | 54 | — | — | — | — | |
| TrellisNet2021.03 | 54.19 | — | — | — | — | |
| TrellisNet-MoS (1.4x larger)Size=34M2018.10 | 54.19 | — | — | — | — | |
| AWD-LSTM-MoS#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 54.4 | — | — | — | — | |
| AWD-LSTM-MoS + finetuning#Params=22M, finetuning=true2019.06 | 54.4 | — | — | — | — | |
| AWD-LSTM-MoS + finetuneSize=22M, finetune=true2017.08 | 54.44 | — | — | — | — | |
| AWD-LSTM-MoS#Param=22M, finetuned=true2017.11 | 54.44 | — | — | — | — | |
| AWD-LSTM-MoS#Param=22M2018.08 | 54.44 | — | — | — | — | |
| AWD-LSTM-MoSParameters=22M, Softmax Type=Multiple2019.05 | 54.44 | — | — | — | — | |
| Transformer-XL#Params=24M2019.06 | 54.5 | — | — | — | — | |
| MoS + FinetunePartial Shuffle=false, Finetune=true2019.03 | 54.64 | — | — | — | — | |
| TrellisNet-MoSSize=25M2018.10 | 54.67 | — | — | — | — | |
| Li et al. (2018)parameters=52 M, effective MAC=51.6 M2022.06 | 55.2 | — | — | — | — | |
| MoS + Partial ShufflePartial Shuffle=true, Finetune=false2019.03 | 55.35 | — | — | — | — | |
| Transformer-XLWeight=0.202023.11 | 55.41 | — | — | — | — | |
| Reference ModelSize=26M2017.08 | 55.58 | — | — | — | — | |
| AWD-LSTM+PDR#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 55.6 | — | — | — | — | |
| AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.05 | 55.7 | — | — | — | — | |
| AWD-LSTM + Phrase Induction + finetuning#Params=24M, Phrase Induction=true, finetuning=true2019.06 | 55.7 | — | — | — | — | |
| ENASAdditional Techniques=VD, WT, l2, Params (million)=242018.02 | 55.8 | — | — | — | — | |
| ENASSize=24M2018.10 | 55.8 | — | — | — | — | |
| AWD-LSTM-MoSSize=22M2017.08 | 55.97 | — | — | — | — | |
| AWD-LSTM-MoS#Param=22M, finetuned=false2017.11 | 55.97 | — | — | — | — | |
| AWD-LSTM-MoSSize=24M2018.10 | 55.97 | — | — | — | — | |
| LSTMAdditional Techniques=VD, WT, l2, AWD, MoS, Params (million)=222018.02 | 56 | — | — | — | — | |
| DARTSSize=23M2018.10 | 56.1 | — | — | — | — | |
| Differentiable NAS#Params=23M2019.06 | 56.1 | — | — | — | — | |
| MoSPartial Shuffle=false, Finetune=false2019.03 | 56.18 | — | — | — | — | |
| ON-LSTM#Params=25M2019.06 | 56.2 | — | — | — | — | |
| Fast WeightsWeight=0.112023.11 | 56.45 | — | — | — | — | |
| Fraternal dropout + AWD-LSTM 3-layerParameters=24M2017.10 | 56.8 | — | — | — | — | |
| AWD-LSTM + Fraternal Dropout#Param=24M2018.08 | 56.8 | — | — | — | — | |
| TrellisNet (1.4x larger)Size=33M2018.10 | 56.8 | — | — | — | — | |
| TrellisNetSize=24M2018.10 | 56.97 | — | — | — | — | |
| AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Excluded2019.05 | 57 | — | — | — | — | |
| EGRUhidden dim=2000, parameters=55 M, effective MAC=12.7 M2022.06 | 57 | — | — | 84.8 | 42.9 | |
| EGRUWeight=0.152023.11 | 57.18 | — | — | — | — | |
| EGRUhidden dim=1350, parameters=33 M, effective MAC=9.9 M2022.06 | 57.2 | — | — | 79.7 | 46.3 | |
| AWD-LSTMSize=24M2017.08 | 57.3 | — | — | — | — | |
| AWD-LSTM - 3-layer LSTM (tied)Parameters=24M2017.08 | 57.3 | — | — | — | — | |
| AWD-LSTMparameters=24M2017.09 | 57.3 | — | — | — | — | |
| LSTMAdditional Techniques=VD, WT, l2, AWD, Params (million)=242018.02 | 57.3 | — | — | — | — | |
| AWD-LSTM#Param=24M, finetuned=true2017.11 | 57.3 | — | — | — | — | |
| Merity et al. (2017a) - AWD-LSTM 3-layer (baseline)Parameters=24M2017.10 | 57.3 | — | — | — | — | |
| AWD-LSTM#Param=24M2018.08 | 57.3 | — | — | — | — | |
| AWD-LSTM#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 57.3 | — | — | — | — | |
| AWD-LSTMParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.05 | 57.3 | — | — | — | — | |
| AWD-LSTM + finetuning#Params=24M, finetuning=true2019.06 | 57.3 | — | — | — | — | |
| Merity et al. (2018)hidden dim=1150, parameters=24 M, effective MAC=24.0 M2022.06 | 57.3 | — | — | — | — | |
| AWD-LSTM + Phrase Induction#Params=24M, Phrase Induction=true2019.06 | 57.5 | — | — | — | — | |
| AWD-LSTM MoCSize=22M2018.10 | 57.55 | — | — | — | — | |
| LSTMAdditional Techniques=VD, WT, l2, AWD, MoC, Params (million)=222018.02 | 57.6 | — | — | — | — | |
| AWD-LSTMparameters=24M, implementation=ours2017.09 | 57.7 | — | — | — | — | |
| Lookahead (ADAM)Model=LSTM2019.07 | 57.72 | — | — | — | — | |
| AWD-LSTM + Phrase Induction (No Attention)#Params=24M, Phrase Induction=true, attention=false2019.06 | 58 | — | — | — | — | |
| 2-layer skip connection LSTMSize=24M2017.08 | 58.3 | — | — | — | — | |
| 4-layer skip connection LSTM (tied)Parameters=24M, Author=Melis et al. (2017)2017.08 | 58.3 | — | — | — | — | |
| LSTM + BB tuningparameters=24M2017.09 | 58.3 | — | — | — | — | |
| 4-layer skip connection LSTM (tied)weight-tying=true2017.11 | 58.3 | — | — | — | — | |
| 2-layer skip connection LSTM#Param=24M2017.11 | 58.3 | — | — | — | — | |
| Melis et al. (2017) - 4-layer skip connection LSTMParameters=24M2017.10 | 58.3 | — | — | — | — | |
| LSTM with skip connections#Param=24M2018.08 | 58.3 | — | — | — | — | |
| LSTM + skip conn. (Black-box tuned)Size=24M2018.10 | 58.3 | — | — | — | — | |
| 2-layer skip connection LSTMParameters=24M, Softmax Type=Single2019.05 | 58.3 | — | — | — | — |