List your company
ResearchFormerly WizWand

Language Modeling on Penn Treebank (val)

46.64PerplexityGL-LWGC-AWD-MOS-LSTM + dynamic evaluation
45.533653.001860.4767.9382Sep 8, 2014Mar 22, 2016Oct 4, 2017Apr 19, 2019Oct 31, 2020May 15, 2022Nov 28, 2023
Updated 8mo ago

Evaluation results

MethodLinks
GL-LWGC-AWD-MOS-LSTM + dynamic evaluationSize=26M, dynamic evaluation=true2017.0846.64
AWD-LSTM-MoS+PDR + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.0848
AWD-LSTM-MoS + dynamic evaluation#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.0848.3
AWD-LSTM-MoS + dynamic evaluationSize=22M, dynamic evaluation=true2017.0848.33
AWD-LSTM-MoS#Param=22M, dynamic evaluation=true2017.1148.33
AWD-LSTM-MoS + dynamic evaluationParameters=22M, Softmax Type=Multiple, Dynamic Evaluation=true2019.0548.33
Ensemble of AllWeight=12023.1148.92
AWD-LSTM-DRILL + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.0549.5
AWD-LSTM+PDR + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.0850.1
AWD-LSTM + dynamic evaluationSize=24M, dynamic evaluation=true2017.0851.6
AWD-LSTM + dynamic evalparameters=24M2017.0951.6
AWD-LSTM#Param=24M, dynamic evaluation=true2017.1151.6
AWD-LSTM + dynamic evaluation#Params=24.2M, Softmax Architecture=Single Softmax2018.0851.6
AWD-LSTM + dynamic evaluationParameters=24M, Softmax Type=Multiple, Dynamic Evaluation=true2019.0551.6
AWD-LSTM+PDR + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.0852.4
DOC + Finetune* + Partial ShufflePartial Shuffle=true, Finetune=repeated 3 times2019.0353.79
AWD-LSTM - 3-layer LSTM (tied) + continuous cache pointerParameters=24M2017.0853.9
AWD-LSTM + neural cacheparameters=24M2017.0953.9
AWD-LSTM#Param=24M, continuous cache pointer=true, dynamic evaluation=true2017.1153.9
AWD-LSTM + continuous cache pointer#Params=24.2M, Softmax Architecture=Single Softmax2018.0853.9
AWD-LSTM + continuous cache pointerParameters=24M, Softmax Type=Multiple2019.0553.9
AWD-LSTM-DOCWeight=0.342023.1154.12
DOC + Finetune*Partial Shuffle=false, Finetune=repeated 3 times2019.0354.18
GL-LWGC-AWD-MOS-LSTM + finetuneSize=26M, finetune=true2017.0854.24
DOC + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.0354.3
GL-LWGC-AWD-MOS-LSTMSize=26M2017.0854.57
DOC + FinetunePartial Shuffle=false, Finetune=true2019.0354.62
DOC + Partial ShufflePartial Shuffle=true, Finetune=false2019.0354.9
2-layers GL-LWGC-AWD-MOS-LSTM + finetuneSize=19M, layers=2, finetune=true2017.0855.18
DOCPartial Shuffle=false, Finetune=false2019.0355.39
MoS + Finetune + Partial ShufflePartial Shuffle=true, Finetune=true2019.0355.89
AWD-LSTM-MOSWeight=0.192023.1156.04
AWD-LSTM-MoS+PDR#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.0856.2
AWD-LSTM-MoS#Params=22M, Softmax Architecture=Mixture-of-Softmax2018.0856.5
AWD-LSTM-MoS + finetuneSize=22M, finetune=true2017.0856.54
AWD-LSTM-MoS#Param=22M, finetuned=true2017.1156.54
AWD-LSTM-MoSParameters=22M, Softmax Type=Multiple2019.0556.54
MoS + FinetunePartial Shuffle=false, Finetune=true2019.0356.76
Reference ModelSize=26M2017.0857.41
MoS + Partial ShufflePartial Shuffle=true, Finetune=false2019.0357.43
AWD-LSTM+PDR#Params=24.2M, Softmax Architecture=Single Softmax2018.0857.9
Transformer-XLWeight=0.202023.1157.93
AWD-LSTM-MoSSize=22M2017.0858.08
AWD-LSTM-MoS#Param=22M, finetuned=false2017.1158.08
AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.0558.2
MoSPartial Shuffle=false, Finetune=false2019.0358.27
Fast WeightsWeight=0.112023.1158.49
Fraternal dropout + AWD-LSTM 3-layerParameters=24M2017.1058.9
AWD-LSTM-DRILLParameters=24M, Softmax Type=Single, Fine-tuning=Excluded2019.0559.6
AWD-LSTMparameters=24M, implementation=ours2017.0959.8
AWD-LSTMSize=24M2017.0860
AWD-LSTM - 3-layer LSTM (tied)Parameters=24M2017.0860
AWD-LSTMparameters=24M2017.0960
AWD-LSTM#Param=24M, finetuned=true2017.1160
Merity et al. (2017a) - AWD-LSTM 3-layer (baseline)Parameters=24M2017.1060
AWD-LSTM#Params=24.2M, Softmax Architecture=Single Softmax2018.0860
AWD-LSTMParameters=24M, Softmax Type=Single, Fine-tuning=Included2019.0560
Lookahead (ADAM)Model=LSTM2019.0760.28
QRNNWeight=0.002023.1160.38
AWD-LSTM#Param=24M, finetuned=false2017.1160.7
AWD-LSTMParameters=24M, Softmax Type=Single, Fine-tuning=Excluded2019.0560.7
2-layer skip connection LSTMSize=24M2017.0860.9
4-layer skip connection LSTM (tied)Parameters=24M, Author=Melis et al. (2017)2017.0860.9
LSTM + BB tuningparameters=24M2017.0960.9
2-layer skip connection LSTM#Param=24M2017.1160.9
Melis et al. (2017) - 4-layer skip connection LSTMParameters=24M2017.1060.9
2-layer skip connection LSTMParameters=24M, Softmax Type=Single2019.0560.9
PolyakModel=LSTM2019.0761.18
EGRUWeight=0.152023.1161.21
ADAMModel=LSTM2019.0761.64
Melis et al. (2017) - 5-layer RHNParameters=24M2017.1064.8
Lookahead (SGD)Model=LSTM2019.0765.1
SGDModel=LSTM2019.0766
Variational RHN + WTSize=23M2017.0867.9
Variational RHN + WTSize=23 M2016.0767.9
Variational RHN (tied)Parameters=23M, Author=Zilly et al. (2016)2017.0867.9
Variational RHNparameters=23M2017.0967.9
Variational RHN#Param=23M2017.1167.9
Zilly et al. (2016) - Variational RHNParameters=23M2017.1067.9
Variational RHNParameters=23M, Softmax Type=Single2019.0567.9
VD-RHN +REParameters=24M2016.1168.1
Inan et al. (2016) - Variational RHNParameters=24M2017.1068.1
VD-LSTM +REAL (large)Parameters=51M2016.1171.1
Variational LSTM + WT + augmented lossSize=51 M2016.0771.1
Variational LSTM (tied) + augmented lossParameters=51M, Author=Inan et al. (2016)2017.0871.1
Variational LSTM + augmented lossparameters=51M2017.0971.1
Inan et al. (2016) - Variational LSTMParameters=51M2017.1071.1
VD-RHNParameters=32M2016.1171.2
Variational RHNSize=32 M2016.0771.2
Large regularized LSTMModel Type=Model averaging, Ensemble Size=38 models2014.0971.9
38 Large LSTMsParameters=2.51B2016.1171.9
Pointer Sentinel-LSTM(medium)Parameters=21M2016.1172.4
Pointer Sentinel-LSTMSize=21 M2016.0772.4
Pointer Sentinel-LSTMParameters=21M, Author=Merity et al. (2016)2017.0872.4
Pointer sentinel-LSTMparameters=21M2017.0972.4
Pointer Sentinel-LSTM#Param=21M2017.1172.4
Pointer Sentinel-LSTMParameters=21M, Softmax Type=Single2019.0572.4
Large regularized LSTMModel Type=Model averaging, Ensemble Size=10 models2014.0972.8
LSTM + dynamic evalparameters=20M, optimizer=RMS, bptt=5, RMS global prior=true2017.0973.5
LSTM + dynamic evalparameters=20M, optimizer=RMS, bptt=5, global prior=true2017.0974.3