Language Modeling on Penn Treebank (val)
46.64PerplexityGL-LWGC-AWD-MOS-LSTM + dynamic evaluation
Evaluation Results
| Method | Links | |
|---|---|---|
| GL-LWGC-AWD-MOS-LSTM + dynamic evaluationSize=26M, dynamic evaluation=true2017.08 | 46.64 | |
| AWD-LSTM-MoS+PDR + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 48 | |
| AWD-LSTM-MoS + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 48.3 | |
| AWD-LSTM-MoS + dynamic evaluationSize=22M, dynamic evaluation=true2017.08 | 48.33 | |
| AWD-LSTM-MoS#Param=22M, dynamic evaluation=true2017.11 | 48.33 | |
| AWD-LSTM-MoS + dynamic evaluationParameters=22M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 48.33 | |
| Ensemble of AllWeight=12023.11 | 48.92 | |
| AWD-LSTM-DRILL + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 49.5 | |
| AWD-LSTM+PDR + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 50.1 | |
| AWD-LSTM + dynamic evaluationSize=24M, dynamic evaluation=true2017.08 | 51.6 | |
| AWD-LSTM + dynamic evalparameters=24M2017.09 | 51.6 | |
| AWD-LSTM#Param=24M, dynamic evaluation=true2017.11 | 51.6 | |
| AWD-LSTM + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 51.6 | |
| AWD-LSTM + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.05 | 51.6 | |
| AWD-LSTM+PDR + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 52.4 | |
| DOC + Finetune* + Partial ShufflePartial Shuffle=true, Finetune=repeated 3 times2019.03 | 53.79 | |
| AWD-LSTM - 3-layer LSTM (tied) + continuous cache pointerParameters=24M2017.08 | 53.9 | |
| AWD-LSTM + neural cacheparameters=24M2017.09 | 53.9 | |
| AWD-LSTM#Param=24M, continuous cache pointer=true, dynamic evaluation=true2017.11 | 53.9 | |
| AWD-LSTM + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 53.9 | |
| AWD-LSTM + continuous cache pointerParameters=24M, Softmax Type=Multiple2019.05 | 53.9 | |
| AWD-LSTM-DOCWeight=0.342023.11 | 54.12 | |
| DOC + Finetune*Partial Shuffle=false, Finetune=repeated 3 times2019.03 | 54.18 | |
| GL-LWGC-AWD-MOS-LSTM + finetuneSize=26M, finetune=true2017.08 | 54.24 | |
| DOC + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.03 | 54.3 | |
| GL-LWGC-AWD-MOS-LSTMSize=26M2017.08 | 54.57 | |
| DOC + FinetunePartial Shuffle=false, Finetune=true2019.03 | 54.62 | |
| DOC + Partial ShufflePartial Shuffle=true, Finetune=false2019.03 | 54.9 | |
| 2-layers GL-LWGC-AWD-MOS-LSTM + finetuneSize=19M, layers=2, finetune=true2017.08 | 55.18 | |
| DOCPartial Shuffle=false, Finetune=false2019.03 | 55.39 | |
| MoS + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.03 | 55.89 | |
| AWD-LSTM-MOSWeight=0.192023.11 | 56.04 | |
| AWD-LSTM-MoS+PDR#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 56.2 | |
| AWD-LSTM-MoS#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.08 | 56.5 | |
| AWD-LSTM-MoS + finetuneSize=22M, finetune=true2017.08 | 56.54 | |
| AWD-LSTM-MoS#Param=22M, finetuned=true2017.11 | 56.54 | |
| AWD-LSTM-MoSParameters=22M, Softmax Type=Multiple2019.05 | 56.54 | |
| MoS + FinetunePartial Shuffle=false, Finetune=true2019.03 | 56.76 | |
| Reference ModelSize=26M2017.08 | 57.41 | |
| MoS + Partial ShufflePartial Shuffle=true, Finetune=false2019.03 | 57.43 | |
| AWD-LSTM+PDR#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 57.9 | |
| Transformer-XLWeight=0.202023.11 | 57.93 | |
| AWD-LSTM-MoSSize=22M2017.08 | 58.08 | |
| AWD-LSTM-MoS#Param=22M, finetuned=false2017.11 | 58.08 | |
| AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.05 | 58.2 | |
| MoSPartial Shuffle=false, Finetune=false2019.03 | 58.27 | |
| Fast WeightsWeight=0.112023.11 | 58.49 | |
| Fraternal dropout + AWD-LSTM 3-layerParameters=24M2017.10 | 58.9 | |
| AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Excluded2019.05 | 59.6 | |
| AWD-LSTMparameters=24M, implementation=ours2017.09 | 59.8 | |
| AWD-LSTMSize=24M2017.08 | 60 | |
| AWD-LSTM - 3-layer LSTM (tied)Parameters=24M2017.08 | 60 | |
| AWD-LSTMparameters=24M2017.09 | 60 | |
| AWD-LSTM#Param=24M, finetuned=true2017.11 | 60 | |
| Merity et al. (2017a) - AWD-LSTM 3-layer (baseline)Parameters=24M2017.10 | 60 | |
| AWD-LSTM#Params=24.2M, Softmax Architecture=Single Softmax2018.08 | 60 | |
| AWD-LSTMParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.05 | 60 | |
| Lookahead (ADAM)Model=LSTM2019.07 | 60.28 | |
| QRNNWeight=0.002023.11 | 60.38 | |
| AWD-LSTM#Param=24M, finetuned=false2017.11 | 60.7 | |
| AWD-LSTMParameters=24M, Softmax Type=Single, Fine-tuning=Excluded2019.05 | 60.7 | |
| 2-layer skip connection LSTMSize=24M2017.08 | 60.9 | |
| 4-layer skip connection LSTM (tied)Parameters=24M, Author=Melis et al. (2017)2017.08 | 60.9 | |
| LSTM + BB tuningparameters=24M2017.09 | 60.9 | |
| 2-layer skip connection LSTM#Param=24M2017.11 | 60.9 | |
| Melis et al. (2017) - 4-layer skip connection LSTMParameters=24M2017.10 | 60.9 | |
| 2-layer skip connection LSTMParameters=24M, Softmax Type=Single2019.05 | 60.9 | |
| PolyakModel=LSTM2019.07 | 61.18 | |
| EGRUWeight=0.152023.11 | 61.21 | |
| ADAMModel=LSTM2019.07 | 61.64 | |
| Melis et al. (2017) - 5-layer RHNParameters=24M2017.10 | 64.8 | |
| Lookahead (SGD)Model=LSTM2019.07 | 65.1 | |
| SGDModel=LSTM2019.07 | 66 | |
| Variational RHN + WTSize=23M2017.08 | 67.9 | |
| Variational RHN + WTSize=23 M2016.07 | 67.9 | |
| Variational RHN (tied)Parameters=23M, Author=Zilly et al. (2016)2017.08 | 67.9 | |
| Variational RHNparameters=23M2017.09 | 67.9 | |
| Variational RHN#Param=23M2017.11 | 67.9 | |
| Zilly et al. (2016) - Variational RHNParameters=23M2017.10 | 67.9 | |
| Variational RHNParameters=23M, Softmax Type=Single2019.05 | 67.9 | |
| VD-RHN +REParameters=24M2016.11 | 68.1 | |
| Inan et al. (2016) - Variational RHNParameters=24M2017.10 | 68.1 | |
| VD-LSTM +REAL (large)Parameters=51M2016.11 | 71.1 | |
| Variational LSTM + WT + augmented lossSize=51 M2016.07 | 71.1 | |
| Variational LSTM (tied) + augmented lossParameters=51M, Author=Inan et al. (2016)2017.08 | 71.1 | |
| Variational LSTM + augmented lossparameters=51M2017.09 | 71.1 | |
| Inan et al. (2016) - Variational LSTMParameters=51M2017.10 | 71.1 | |
| VD-RHNParameters=32M2016.11 | 71.2 | |
| Variational RHNSize=32 M2016.07 | 71.2 | |
| Large regularized LSTMModel Type=Model averaging, Ensemble Size=38 models2014.09 | 71.9 | |
| 38 Large LSTMsParameters=2.51B2016.11 | 71.9 | |
| Pointer Sentinel-LSTM(medium)Parameters=21M2016.11 | 72.4 | |
| Pointer Sentinel-LSTMSize=21 M2016.07 | 72.4 | |
| Pointer Sentinel-LSTMParameters=21M, Author=Merity et al. (2016)2017.08 | 72.4 | |
| Pointer sentinel-LSTMparameters=21M2017.09 | 72.4 | |
| Pointer Sentinel-LSTM#Param=21M2017.11 | 72.4 | |
| Pointer Sentinel-LSTMParameters=21M, Softmax Type=Single2019.05 | 72.4 | |
| Large regularized LSTMModel Type=Model averaging, Ensemble Size=10 models2014.09 | 72.8 | |
| LSTM + dynamic evalparameters=20M, optimizer=RMS, bptt=5, RMS global prior=true2017.09 | 73.5 | |
| LSTM + dynamic evalparameters=20M, optimizer=RMS, bptt=5, global prior=true2017.09 | 74.3 |