Character-level Language Modeling on Hutter Prize Wikipedia (test)
1.08Bits/CharmLSTM
Evaluation results
| Method | Links | ||
|---|---|---|---|
| mLSTMparameters=46M, evaluation_protocol=dynamic eval2017.09 | 1.08 | — | |
| mLSTMparameters=46M, evaluation_protocol=sparse dynamic eval (d = 250k)2017.09 | 1.13 | — | |
| large mLSTM +emb +WN +VD# of parameters=46M, embedding layer=true, weight normalization=true, variational dropout=true2016.09 | 1.24 | — | |
| mLSTMparameters=46M2017.09 | 1.24 | — | |
| Fast-slow LSTM (rec depth 4) + zoneout# of parameters=47M2016.09 | 1.25 | — | |
| Fast-slow LSTMparameters=47M2017.09 | 1.25 | — | |
| RHN (rec depth 7) + VD# of parameters=46M2016.09 | 1.27 | — | |
| Recurrent highway networksparameters=46M2017.09 | 1.27 | — | |
| mLSTM +emb +WN +VD# of parameters=22M, embedding layer=true, weight normalization=true, variational dropout=true2016.09 | 1.28 | — | |
| LSTM (4 layer) + VD + BB tuning# of parameters=46M2016.09 | 1.3 | — | |
| LSTM + BB tuningparameters=46M2017.09 | 1.3 | — | |
| bytenet decoder2016.09 | 1.31 | — | |
| Bytenet decoder2017.09 | 1.31 | — | |
| hierarchical multiscale LSTM2016.09 | 1.32 | — | |
| Hierarchical multiscale LSTM2017.09 | 1.32 | — | |
| stacked LSTM (7-layer) + dynamic eval# of parameters=21M, dynamic evaluation=true2016.09 | 1.33 | — | |
| Stacked LSTMparameters=21M, evaluation_protocol=traditional dynamic eval2017.09 | 1.33 | — | |
| hyperLSTM# of parameters=27M2016.09 | 1.34 | — | |
| HyperLSTMparameters=27M2017.09 | 1.34 | — | |
| feedback LSTM + zoneout2016.09 | 1.37 | — | |
| recurrent memory array structures2016.09 | 1.4 | — | |
| unregularised mLSTM# of parameters=20M2016.09 | 1.4 | — | |
| unregularised mLSTM (RMS prop, 4 epoch)# of parameters=20M, optimizer=RMS prop, epochs=42016.09 | 1.42 | — | |
| MI-LSTM# of parameters=17M2016.09 | 1.44 | — | |
| mLSTM +emb +WN# of parameters=22M, embedding layer=true, weight normalization=true2016.09 | 1.44 | — | |
| Multiplicative integration LSTMparameters=17M2017.09 | 1.44 | — | |
| stacked LSTM (7-layer)# of parameters=21M2016.09 | 1.67 | — | |
| Stacked LSTMparameters=21M2017.09 | 1.67 | — | |
| ByteNet Decoderresidual_blocks=30, hidden_units=512, kernel_size=32016.10 | — | 1.31 | |
| GF-LSTM2016.10 | — | 1.58 | |
| Grid-LSTM2016.10 | — | 1.47 | |
| HM-LSTM2016.10 | — | 1.4 | |
| Large Layer Norm HyperLSTMvariant=Large2016.10 | — | 1.34 | |
| Layer Norm HyperLSTM2016.10 | — | 1.38 | |
| Layer-normalized LSTM2016.10 | — | 1.46 | |
| MI-LSTM2016.10 | — | 1.44 | |
| Recurrent Highway Networks2016.10 | — | 1.32 | |
| Recurrent Memory Array Structures2016.10 | — | 1.4 | |
| Stacked LSTM2016.10 | — | 1.67 |