Character-level Language Modeling on Penn Treebank (test)
1.175BPC3 layer LSTM
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| 3 layer LSTMh=1000, Params=13.8M2018.03 | 1.175 | — | — | |
| 3-layer LSTMGeneration time=3.8 min2019.05 | 1.18 | — | — | |
| 6 layer QRNNh=1024, Params=13.8M2018.03 | 1.187 | — | — | |
| FS-RUM-2# Parameters=11.2M2017.10 | 1.189 | — | — | |
| FS-LSTM-2Param Count=7.2M2017.05 | 1.19 | — | — | |
| FS-LSTM-2# Parameters=7.2M2017.10 | 1.19 | — | — | |
| FS-LSTM-2Params=7.2M2018.03 | 1.19 | — | — | |
| Fast-slow LSTM2018.03 | 1.19 | — | — | |
| res-IndRNNNumber of layers=11, Time steps=150, Residual connection=true2018.03 | 1.19 | — | — | |
| FS-LSTM-4Param Count=6.5M2017.05 | 1.193 | — | — | |
| FS-LSTM-4# Parameters=6.5M2017.10 | 1.193 | — | — | |
| FS-LSTM-4Params=6.5M2018.03 | 1.193 | — | — | |
| PRPN2017.11 | 1.202 | — | — | |
| Neural Architecture Search2018.03 | 1.21 | — | — | |
| res-IndRNNNumber of layers=21, Time steps=50, Residual connection=true2018.03 | 1.21 | — | — | |
| Two Layer with New CellParameters=16.28M2016.11 | 1.214 | — | — | |
| NASCellParam Count=16.3M2017.05 | 1.214 | — | — | |
| NASCell# Parameters=16.3M2017.10 | 1.214 | — | — | |
| NASCellParams=16.3M2018.03 | 1.214 | — | — | |
| 2-Layer Norm HyperLSTMParameters=14.41M2016.11 | 1.219 | — | — | |
| HyperLSTMParam Count=14.4M2017.05 | 1.219 | — | — | |
| HyperLSTM# Parameters=14.4M2017.10 | 1.219 | — | — | |
| HyperLSTMParams=14.4M2018.03 | 1.219 | — | — | |
| HyperNetworks2016.03 | 1.22 | — | — | |
| HyperNetworks# layers=1 / 2, hidden / layer=1k, # parameters (≈, M)=4.91 / 14.41, Max dilations=1, Note=2nd result 1.22 includes layer normalization [3]2017.10 | 1.22 | — | — | |
| Two Layer with New CellParameters=6.57M2016.11 | 1.228 | — | — | |
| NASCell - smallParam Count=6.6M2017.05 | 1.228 | — | — | |
| NASCell (small)Params=6.6M2018.03 | 1.228 | — | — | |
| LayerNorm HyperNetworksNormalization=LayerNorm2016.09 | 1.23 | — | — | |
| LayerNorm HyperNetworksNormalization=Layer Normalization2017.11 | 1.23 | — | — | |
| IndRNNNumber of layers=6, Time steps=1502018.03 | 1.23 | — | — | |
| Layer Norm HyperLSTM Large EmbeddingsParameters=5.06M2016.11 | 1.233 | — | — | |
| HyperLSTM - smallParam Count=5.1M2017.05 | 1.233 | — | — | |
| HyperLSTM - smallParams=5.1M2018.03 | 1.233 | — | — | |
| LSTMRegularisation=WEIGHT NOISE, Dynamic=true, Epochs=252013.08 | 1.24 | 124 | 26.9 | |
| LSTMRegularisation=ADAPT. WT. NOISE, Dynamic=true, Epochs=262013.08 | 1.24 | 122 | 26.9 | |
| HM-LSTM2016.03 | 1.24 | — | — | |
| LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Step Fn. & Slope Annealing2016.09 | 1.24 | — | — | |
| HM-LSTM2017.05 | 1.24 | — | — | |
| LayerNorm HM-LSTM# layers=3, hidden / layer=512, Max dilations=12017.10 | 1.24 | — | — | |
| HM-LSTM2017.10 | 1.24 | — | — | |
| LayerNorm HM-LSTMNormalization=Layer Normalization2017.11 | 1.24 | — | — | |
| HM-LSTM2018.03 | 1.24 | — | — | |
| R-Transformer# of layers=3, hidden size=5122019.07 | 1.24 | — | — | |
| Hierarchical Multiscale LSTM + LN2018.03 | 1.24 | — | — | |
| Two layer LSTMParameters=6.57M2016.11 | 1.243 | — | — | |
| 2-Layers LSTMParam Count=6.6M2017.05 | 1.243 | — | — | |
| 2-Layers LSTMParams=6.6M2018.03 | 1.243 | — | — | |
| Layer Norm HyperLSTMParameters=4.92M2016.11 | 1.25 | — | — | |
| LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Step Fn.2016.09 | 1.25 | — | — | |
| HyperLSTM + LN2018.03 | 1.25 | — | — | |
| LSTMRegularisation=ADAPT. WT. NOISE, Dynamic=false, Epochs=262013.08 | 1.26 | 133 | 27.4 | |
| LSTM2016.03 | 1.26 | — | — | |
| 2 x RUM 1500# Parameters=16.4M2017.10 | 1.26 | — | — | |
| IndRNNNumber of layers=6, Time steps=502018.03 | 1.26 | — | — | |
| LSTMRegularisation=WEIGHT NOISE, Dynamic=false, Epochs=252013.08 | 1.27 | 140 | 27.4 | |
| Zoneout2016.03 | 1.27 | — | — | |
| Zoneout RNN2016.09 | 1.27 | — | — | |
| HyperNetworks2016.09 | 1.27 | — | — | |
| LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Sampling2016.09 | 1.27 | — | — | |
| LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Soft*2016.09 | 1.27 | — | — | |
| Zoneout LSTM2017.05 | 1.27 | — | — | |
| Zoneout# layers=1, hidden / layer=1k, # parameters (≈, M)=4.25, Max dilations=12017.10 | 1.27 | — | — | |
| Dilated GRU# layers=5, hidden / layer=256, # parameters (≈, M)=1.42, Max dilations=642017.10 | 1.27 | — | — | |
| Zoneout LSTM2017.10 | 1.27 | — | — | |
| Zoneout RNNRegularization=Zoneout2017.11 | 1.27 | — | — | |
| HyperNetworks2017.11 | 1.27 | — | — | |
| Zoneout LSTM2018.03 | 1.27 | — | — | |
| LSTM+Zoneout2018.03 | 1.27 | — | — | |
| RUM 2000# Parameters=8.9M2017.10 | 1.28 | — | — | |
| LSTMRegularisation=NONE, Dynamic=true, Epochs=92013.08 | 1.29 | 148 | 28 | |
| LayerNorm LSTMNormalization=LayerNorm2016.09 | 1.29 | — | — | |
| Recurrent dropout LSTM# layers=1, hidden / layer=1k, # parameters (≈, M)=4.25, Max dilations=12017.10 | 1.3 | — | — | |
| Dilated LSTM# layers=5, hidden / layer=256, # parameters (≈, M)=1.9, Max dilations=642017.10 | 1.31 | — | — | |
| TCN# of layers=3, hidden size=4502019.07 | 1.31 | — | — | |
| LSTMRegularisation=NONE, Dynamic=false, Epochs=92013.08 | 1.32 | 167 | 28.5 | |
| BN-LSTMhidden units=10002016.03 | 1.32 | — | — | |
| BatchNorm LSTM2016.09 | 1.32 | — | — | |
| Recurrent BN-LSTM# layers=1, hidden / layer=1k, Max dilations=12017.10 | 1.32 | — | — | |
| BatchNorm LSTMNormalization=Batch Normalization2017.11 | 1.32 | — | — | |
| LSTM+Recurrent dropout2018.03 | 1.32 | — | — | |
| LSTM+Recurrent batchnorm2018.03 | 1.32 | — | — | |
| LSTM# layers=1 / 5, hidden / layer=1k / 256, # parameters (≈, M)=4.25 / 1.9, Max dilations=12017.10 | 1.33 | — | — | |
| GRU# layers=1 / 5, hidden / layer=1k / 256, # parameters (≈, M)=3.19 / 1.42, Max dilations=12017.10 | 1.33 | — | — | |
| LSTM# of layers=2, hidden size=6002019.07 | 1.36 | — | — | |
| LSTM2018.03 | 1.36 | — | — | |
| ME n-gram2016.03 | 1.37 | — | — | |
| ME n-gram2016.09 | 1.37 | — | — | |
| Dilated Vanilla# layers=5, hidden / layer=256, # parameters (≈, M)=0.6, Max dilations=642017.10 | 1.37 | — | — | |
| ME n-gram2017.11 | 1.37 | — | — | |
| GRU2019.07 | 1.37 | — | — | |
| LSTMhidden units=10002016.03 | 1.38 | — | — | |
| Bipartite flowGeneration time=0.17 sec2019.05 | 1.38 | — | — | |
| Norm-stabilized LSTM2016.03 | 1.39 | — | — | |
| MI-RNN2016.09 | 1.39 | — | — | |
| MI-RNN2017.11 | 1.39 | — | — | |
| LayerNorm CW-RNNNormalization=LayerNorm2016.09 | 1.4 | — | — | |
| HF-MRNN2016.03 | 1.41 | — | — | |
| HF-MRNN2016.09 | 1.41 | — | — | |
| HF-MRNN2017.11 | 1.41 | — | — |