List your company
ResearchFormerly WizWand

Character-level Language Modeling on Penn Treebank (test)

1.175BPC3 layer LSTM
1.16561.229051.29251.35595Aug 4, 2013Jul 31, 2014Jul 27, 2015Jul 23, 2016Jul 19, 2017Jul 15, 2018Jul 12, 2019
Updated 8mo ago

Evaluation results

MethodLinks
3 layer LSTMh=1000, Params=13.8M2018.031.175——
3-layer LSTMGeneration time=3.8 min2019.051.18——
6 layer QRNNh=1024, Params=13.8M2018.031.187——
FS-RUM-2# Parameters=11.2M2017.101.189——
FS-LSTM-2Param Count=7.2M2017.051.19——
FS-LSTM-2# Parameters=7.2M2017.101.19——
FS-LSTM-2Params=7.2M2018.031.19——
Fast-slow LSTM2018.031.19——
res-IndRNNNumber of layers=11, Time steps=150, Residual connection=true2018.031.19——
FS-LSTM-4Param Count=6.5M2017.051.193——
FS-LSTM-4# Parameters=6.5M2017.101.193——
FS-LSTM-4Params=6.5M2018.031.193——
PRPN2017.111.202——
Neural Architecture Search2018.031.21——
res-IndRNNNumber of layers=21, Time steps=50, Residual connection=true2018.031.21——
Two Layer with New CellParameters=16.28M2016.111.214——
NASCellParam Count=16.3M2017.051.214——
NASCell# Parameters=16.3M2017.101.214——
NASCellParams=16.3M2018.031.214——
2-Layer Norm HyperLSTMParameters=14.41M2016.111.219——
HyperLSTMParam Count=14.4M2017.051.219——
HyperLSTM# Parameters=14.4M2017.101.219——
HyperLSTMParams=14.4M2018.031.219——
HyperNetworks2016.031.22——
HyperNetworks# layers=1 / 2, hidden / layer=1k, # parameters (≈, M)=4.91 / 14.41, Max dilations=1, Note=2nd result 1.22 includes layer normalization [3]2017.101.22——
Two Layer with New CellParameters=6.57M2016.111.228——
NASCell - smallParam Count=6.6M2017.051.228——
NASCell (small)Params=6.6M2018.031.228——
LayerNorm HyperNetworksNormalization=LayerNorm2016.091.23——
LayerNorm HyperNetworksNormalization=Layer Normalization2017.111.23——
IndRNNNumber of layers=6, Time steps=1502018.031.23——
Layer Norm HyperLSTM Large EmbeddingsParameters=5.06M2016.111.233——
HyperLSTM - smallParam Count=5.1M2017.051.233——
HyperLSTM - smallParams=5.1M2018.031.233——
LSTMRegularisation=WEIGHT NOISE, Dynamic=true, Epochs=252013.081.2412426.9
LSTMRegularisation=ADAPT. WT. NOISE, Dynamic=true, Epochs=262013.081.2412226.9
HM-LSTM2016.031.24——
LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Step Fn. & Slope Annealing2016.091.24——
HM-LSTM2017.051.24——
LayerNorm HM-LSTM# layers=3, hidden / layer=512, Max dilations=12017.101.24——
HM-LSTM2017.101.24——
LayerNorm HM-LSTMNormalization=Layer Normalization2017.111.24——
HM-LSTM2018.031.24——
R-Transformer# of layers=3, hidden size=5122019.071.24——
Hierarchical Multiscale LSTM + LN2018.031.24——
Two layer LSTMParameters=6.57M2016.111.243——
2-Layers LSTMParam Count=6.6M2017.051.243——
2-Layers LSTMParams=6.6M2018.031.243——
Layer Norm HyperLSTMParameters=4.92M2016.111.25——
LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Step Fn.2016.091.25——
HyperLSTM + LN2018.031.25——
LSTMRegularisation=ADAPT. WT. NOISE, Dynamic=false, Epochs=262013.081.2613327.4
LSTM2016.031.26——
2 x RUM 1500# Parameters=16.4M2017.101.26——
IndRNNNumber of layers=6, Time steps=502018.031.26——
LSTMRegularisation=WEIGHT NOISE, Dynamic=false, Epochs=252013.081.2714027.4
Zoneout2016.031.27——
Zoneout RNN2016.091.27——
HyperNetworks2016.091.27——
LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Sampling2016.091.27——
LayerNorm HM-LSTMNormalization=LayerNorm, Configuration=Soft*2016.091.27——
Zoneout LSTM2017.051.27——
Zoneout# layers=1, hidden / layer=1k, # parameters (≈, M)=4.25, Max dilations=12017.101.27——
Dilated GRU# layers=5, hidden / layer=256, # parameters (≈, M)=1.42, Max dilations=642017.101.27——
Zoneout LSTM2017.101.27——
Zoneout RNNRegularization=Zoneout2017.111.27——
HyperNetworks2017.111.27——
Zoneout LSTM2018.031.27——
LSTM+Zoneout2018.031.27——
RUM 2000# Parameters=8.9M2017.101.28——
LSTMRegularisation=NONE, Dynamic=true, Epochs=92013.081.2914828
LayerNorm LSTMNormalization=LayerNorm2016.091.29——
Recurrent dropout LSTM# layers=1, hidden / layer=1k, # parameters (≈, M)=4.25, Max dilations=12017.101.3——
Dilated LSTM# layers=5, hidden / layer=256, # parameters (≈, M)=1.9, Max dilations=642017.101.31——
TCN# of layers=3, hidden size=4502019.071.31——
LSTMRegularisation=NONE, Dynamic=false, Epochs=92013.081.3216728.5
BN-LSTMhidden units=10002016.031.32——
BatchNorm LSTM2016.091.32——
Recurrent BN-LSTM# layers=1, hidden / layer=1k, Max dilations=12017.101.32——
BatchNorm LSTMNormalization=Batch Normalization2017.111.32——
LSTM+Recurrent dropout2018.031.32——
LSTM+Recurrent batchnorm2018.031.32——
LSTM# layers=1 / 5, hidden / layer=1k / 256, # parameters (≈, M)=4.25 / 1.9, Max dilations=12017.101.33——
GRU# layers=1 / 5, hidden / layer=1k / 256, # parameters (≈, M)=3.19 / 1.42, Max dilations=12017.101.33——
LSTM# of layers=2, hidden size=6002019.071.36——
LSTM2018.031.36——
ME n-gram2016.031.37——
ME n-gram2016.091.37——
Dilated Vanilla# layers=5, hidden / layer=256, # parameters (≈, M)=0.6, Max dilations=642017.101.37——
ME n-gram2017.111.37——
GRU2019.071.37——
LSTMhidden units=10002016.031.38——
Bipartite flowGeneration time=0.17 sec2019.051.38——
Norm-stabilized LSTM2016.031.39——
MI-RNN2016.091.39——
MI-RNN2017.111.39——
LayerNorm CW-RNNNormalization=LayerNorm2016.091.4——
HF-MRNN2016.031.41——
HF-MRNN2016.091.41——
HF-MRNN2017.111.41——