Language Modeling on Penn Treebank (PTB) (val)
46.63PerplexityAWD-LSTM + MoS + Partial Shuffled + Adversarial Training
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| AWD-LSTM + MoS + Partial Shuffled + Adversarial TrainingParams=22M, dynamic evaluation=true2019.06 | 46.63 | — | — | — | — | |
| AWD-LSTM + MoS + Adversarial TrainingParams=22M, dynamic evaluation=true2019.06 | 47.15 | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluation with FRAGEParas=24M2018.09 | 47.38 | — | — | — | — | |
| AWD-LSTM + MoS + Partial ShuffledParams=22M, dynamic evaluation=true2019.06 | 47.93 | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, dynamic evaluation=true2019.06 | 48.33 | — | — | — | — | |
| AWD-LSTM-MoS + dynamic evaluationParas=24M2018.09 | 48.33 | — | — | — | — | |
| AWD-LSTM + Adversarial TrainingParams=24M, dynamic evaluation=true2019.06 | 49.31 | — | — | — | — | |
| ADV-AWD-LSTM + MaxUpParams=24M, Dynamic evaluation=true2020.02 | 50.83 | — | — | — | — | |
| AWD-LSTMParams=24M, dynamic evaluation=true2019.06 | 51.6 | — | — | — | — | |
| ADV-AWD-LSTMParams=24M, Dynamic evaluation=true2020.02 | 51.6 | — | — | — | — | |
| AWD-LSTM + continuous cache pointer with FRAGEParas=24M2018.09 | 52.3 | — | — | — | — | |
| AWD-LSTM + continuous cache pointerParas=24M2018.09 | 53.9 | — | — | — | — | |
| AWD-LSTM + MoS + Partial Shuffled + Adversarial TrainingParams=22M, dynamic evaluation=false2019.06 | 54.1 | — | — | — | — | |
| AWD-LSTM-DOC#Params=23M2022.03 | 54.1 | — | — | — | — | |
| AWD-LSTM + MoS + Adversarial TrainingParams=22M, dynamic evaluation=false2019.06 | 54.98 | — | — | — | — | |
| AWD-LSTM-MoS with FRAGEParas=24M2018.09 | 55.52 | — | — | — | — | |
| AWD-LSTM + MoS + Partial ShuffledParams=22M, dynamic evaluation=false2019.06 | 55.89 | — | — | — | — | |
| ADV-AWD-LSTM + MaxUpParams=24M, Dynamic evaluation=false2020.02 | 56.25 | — | — | — | — | |
| LSTM + MoSSource=[49], Search Method=manual2019.02 | 56.5 | 22 | — | — | — | |
| AWD-LSTM-MoS#Params=22M2022.03 | 56.5 | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, dynamic evaluation=false2019.06 | 56.54 | — | — | — | — | |
| AWD-LSTM + MoSParams=22M, Dynamic evaluation=false2020.02 | 56.54 | — | — | — | — | |
| AWD-LSTM-MoSParas=24M2018.09 | 56.54 | — | — | — | — | |
| AWD-FWM (ours)Number of parameters=24M, Seeds=32020.11 | 56.76 | — | — | — | — | |
| AWD-LSTM + Adversarial TrainingParams=24M, dynamic evaluation=false2019.06 | 57.15 | — | — | — | — | |
| ADV-AWD-LSTMParams=24M, Dynamic evaluation=false2020.02 | 57.15 | — | — | — | — | |
| AWD-LSTM-MoS w/o finetune with FRAGEParas=24M2018.09 | 57.55 | — | — | — | — | |
| Random search WSSource=Ours, Comparable Search Space=Y, Search Method=random2019.02 | 57.8 | 23 | 0.25 | 1 | 1.25 | |
| AWD-LSTM-MoS w/o finetuneParas=24M2018.09 | 58.08 | — | — | — | — | |
| ASHA + LSTM + DropConnectSource=[30], Comparable Search Space=N, Search Method=HP-tuned, Search Cost Unit=CPU-days2019.02 | 58.1 | 24 | — | — | 13 | |
| DARTS (second order)Source=[34], Comparable Search Space=Y, Search Method=gradient-based2019.02 | 58.1 | 23 | 1 | 1 | 2 | |
| LSTM + SEParams (M)=222019.10 | 58.1 | — | — | — | — | |
| DARTS (2nd)Params (M)=23, Search Cost (GPU days)=0.252019.10 | 58.1 | — | — | — | — | |
| AWD-LSTM + FRAGEParams=24M, Dynamic evaluation=false2020.02 | 58.1 | — | — | — | — | |
| AWD-LSTM with FRAGEParas=24M2018.09 | 58.1 | — | — | — | — | |
| DARTS (second order)Source=Ours (reproduced), Comparable Search Space=Y, Search Method=gradient-based2019.02 | 58.2 | 23 | 1 | 1 | 2 | |
| DARTSParams=23M2019.06 | 58.3 | — | — | — | — | |
| ON-LSTM#Params=25M2022.03 | 58.3 | — | — | — | — | |
| AWD-LSTMParams=24M, Dynamic evaluation=false2020.02 | 58.5 | — | — | — | — | |
| ASHA baselineSource=Ours, Comparable Search Space=Y, Search Method=random2019.02 | 58.6 | 23 | — | — | 2 | |
| DM-LSTM#Params=24M2022.03 | 58.6 | — | — | — | — | |
| Fraternal dropout#Params=24M2022.03 | 58.9 | — | — | — | — | |
| AWD-TXL (ours)Number of parameters=24M, Seeds=3, AWD-style regularisation=true, Model averaging=true, Softmax temperature tuning=true2020.11 | 59.39 | — | — | — | — | |
| GDASParams (M)=23, Search Cost (GPU days)=0.42019.10 | 59.8 | — | — | — | — | |
| LSTM + DropConnectSource=[38], Search Method=manual2019.02 | 60 | 24 | — | — | — | |
| AWD-LSTMParams=24M, dynamic evaluation=false2019.06 | 60 | — | — | — | — | |
| AWD-LSTMParas=24M2018.09 | 60 | — | — | — | — | |
| AWD-LSTM#Params=24M2022.03 | 60 | — | — | — | — | |
| AWD-LSTM2022.05 | 60 | — | — | — | — | |
| AWD-LSTMNumber of parameters=24M, Seeds=32020.11 | 60 | — | — | — | — | |
| DARTS (first order)Source=[34], Comparable Search Space=Y, Search Method=gradient-based2019.02 | 60.2 | 23 | 0.5 | 1 | 1.5 | |
| DARTS (1st)Params (M)=23, Search Cost (GPU days)=0.132019.10 | 60.2 | — | — | — | — | |
| AWD-LSTM w/o finetune with FRAGEParas=24M2018.09 | 60.2 | — | — | — | — | |
| LSTMParams (M)=242019.10 | 60.7 | — | — | — | — | |
| AWD-LSTM w/o finetuneParas=24M2018.09 | 60.7 | — | — | — | — | |
| ENASSource=[34], Comparable Search Space=Y, Search Method=random2019.02 | 60.8 | 24 | 0.5 | — | — | |
| ENASParams (M)=24, Search Cost (GPU days)=0.52019.10 | 60.8 | — | — | — | — | |
| LSTM + SCParams (M)=242019.10 | 60.9 | — | — | — | — | |
| Random search baselineSource=[34], Comparable Search Space=Y, Search Method=random2019.02 | 61.8 | 23 | — | — | 2 | |
| V-RHNParams (M)=232019.10 | 67.9 | — | — | — | — | |
| Variational RHN#Params=23M2022.03 | 67.9 | — | — | — | — | |
| Variational LSTM + weight tyingParams=51M2019.06 | 71.1 | — | — | — | — | |
| Pointer Sentinel-LSTM#Params=21M2022.03 | 72.4 | — | — | — | — | |
| VL-HMMNumber of states=2^15, Clustering method=Brown2022.05 | 125 | — | — | — | — | |
| Rank-Space ModelsNumber of states=2^142022.05 | 135.6 | — | — | — | — | |
| VL-HMMNumber of states=2^14, Clustering method=Brown2022.05 | 136 | — | — | — | — | |
| Rank-Space ModelsNumber of states=2^152022.05 | 137 | — | — | — | — | |
| LHMMNumber of states=2^142022.05 | 141.4 | — | — | — | — | |
| HMM+RNN2022.05 | 142.3 | — | — | — | — | |
| VL-HMMNumber of states=2^14, Clustering method=Uniform2022.05 | 146 | — | — | — | — |