Language Modeling on Penn Treebank word-level (test)
49.95Perplexitydense-IndRNN+dynamic eval
Evaluation results
| Method | Links | ||
|---|---|---|---|
| dense-IndRNN+dynamic evaldynamic evaluation=true2019.10 | 49.95 | — | |
| AWD-LSTM+Finetue+dynamic evalmode=fine-tuned, dynamic evaluation=true2019.10 | 51.1 | — | |
| AWD-LSTM-MoSParameters=22M2018.10 | 54.4 | — | |
| MoS#Param=22M, Two-step finetuning=true2019.01 | 54.44 | — | |
| Transformer-XL#Param=24M, Two-step finetuning=false2019.01 | 54.52 | — | |
| dense-IndRNN2019.10 | 55.24 | — | |
| Dropout tuning#Param=24M, Two-step finetuning=false2019.01 | 55.3 | — | |
| DARTS architecture search (second order)# Params=23M, Non-embedding model size=20M2019.09 | 55.7 | — | |
| AWD-LSTM-MoS#Param=22M, Two-step finetuning=false2019.01 | 55.97 | — | |
| Differentiable NAS#Param=23M, Two-step finetuning=false2019.01 | 56.1 | — | |
| ON-LSTM - 3-layer (tied)Parameters=25M, tied=true2018.10 | 56.17 | — | |
| AWD-PRU#Params=19 M, dropout=weight dropout2018.11 | 56.56 | — | |
| 60-layer TrellisNet (w/ auxiliary loss, w/o MoS)# Params=24M, Non-embedding model size=20M2019.09 | 57 | — | |
| DEQ-TrellisNet# Params=24M, Non-embedding model size=20M2019.09 | 57.1 | — | |
| AWD-LSTM#Params=24 M2018.11 | 57.3 | — | |
| AWD-LSTM - 3-layer LSTM (tied)Parameters=24M, tied=true2018.10 | 57.3 | — | |
| AWD-LSTM#Param=24M, Two-step finetuning=true2019.01 | 57.3 | — | |
| AWD-LSTM+Finetuemode=fine-tuned2019.10 | 57.3 | — | |
| Skip-connection LSTM#Params=24 M2018.11 | 58.3 | — | |
| 4-layer skip connection LSTM (tied)Parameters=24M, tied=true2018.10 | 58.3 | — | |
| Efficient NAS#Param=24M, Two-step finetuning=false2019.01 | 58.6 | — | |
| AWD-LSTM#Param=24M, Two-step finetuning=false2019.01 | 58.8 | — | |
| AWD-LSTM# Params=24M, Non-embedding model size=20M2019.09 | 58.8 | — | |
| AWD-LSTM2019.10 | 58.8 | — | |
| res-IndRNNdepth=12 layers2019.10 | 58.99 | — | |
| NAS (w/ black-box hyperparameter tuner)# Params=24M, Non-embedding model size=20M2019.09 | 59.7 | — | |
| SRU#Params=24 M2018.11 | 60.3 | — | |
| SRU2019.10 | 60.3 | — | |
| PRPN-LM2018.10 | 62 | — | |
| NAS Cell (tied)Parameters=54M, tied=true2018.10 | 62.4 | — | |
| Neural Architecture Search2018.03 | 62.4 | — | |
| NAS Cell# Params=54M2019.09 | 62.4 | — | |
| Neural Architecture Search2019.10 | 62.4 | — | |
| PRU#Params=19 M, dropout=standard dropout2018.11 | 62.42 | — | |
| ERU#Params=15 M, dropout=standard dropout2018.11 | 63.47 | — | |
| NAS Cell#Param=25M, Two-step finetuning=false2019.01 | 64 | — | |
| res-IndRNNNumber of Layers=112018.03 | 65.3 | — | |
| Variational RHN (tied)Parameters=23M, tied=true2018.10 | 65.4 | — | |
| Variational RHN#Param=23M, Two-step finetuning=false2019.01 | 65.4 | — | |
| RHN2018.03 | 65.4 | — | |
| RHN2019.10 | 65.4 | — | |
| Variational LSTM (tied) + augmented lossParameters=51M, tied=true2018.10 | 68.5 | — | |
| Pointer Sentinel-LSTMParameters=21M2018.10 | 70.9 | — | |
| Pointer Sentinel LSTM2018.03 | 70.9 | — | |
| Pointer Sentinel LSTM2019.10 | 70.9 | — | |
| LSTM + continuous cache pointer2018.10 | 72.1 | — | |
| Tied Variational LSTM#Param=24M, Two-step finetuning=false2019.01 | 73.2 | — | |
| Variational LSTM (large, MC)Parameters=66M2018.10 | 73.4 | — | |
| LSTM+Variational Dropout2018.03 | 73.4 | — | |
| Variational LSTM# Params=66M2019.09 | 73.4 | — | |
| LSTM+Variational Dropout2019.10 | 73.4 | — | |
| ERU#Params=7 M, dropout=standard dropout2018.11 | 73.63 | — | |
| LSTM+Zoneout2018.03 | 77.4 | — | |
| LSTM+Zoneout2019.10 | 77.4 | — | |
| QRNN#Params=18 M2018.11 | 78.3 | — | |
| LSTM (large)Parameters=66M2018.10 | 78.4 | — | |
| Variational LSTM#Params=20 M2018.11 | 78.6 | — | |
| CharCNNParameters=19M2018.10 | 78.9 | — | |
| CharCNN2018.03 | 78.9 | — | |
| CharCNN2019.10 | 78.9 | — | |
| LSTM2018.10 | 82.3 | — | |
| R-Transformer2019.10 | 84.38 | — | |
| LSTM+Recurrent dropout2018.03 | 87 | — | |
| LSTM+Recurrent dropout2019.10 | 87 | — | |
| Quantized LSTM2018.11 | 89.8 | — | |
| RNN-LDA + KN-5 + cache2018.03 | 92 | — | |
| RNN-LDA + KN-5 + cache2019.10 | 92 | — | |
| Deep RNN2018.03 | 107.5 | — | |
| Deep RNN2019.10 | 107.5 | — | |
| LSTM2018.03 | 114.5 | — | |
| LSTM2019.10 | 114.5 | — | |
| Transformer2019.10 | 122.37 | — |