Character-level Language Modeling on text8 (test)
1.038BPCTransformer-XL + RMS dynamic eval + decay
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| Transformer-XL + RMS dynamic eval + decay# of params=277M2019.04 | 1.038 | — | |
| Transformer-XL + SGD dynamic eval# of params=277M2019.04 | 1.042 | — | |
| Transformer + adaptive span#Params=209M, Model scale=Large2019.07 | 1.07 | — | |
| Adaptive-SpanModel Category=Large, Span limit (S)=8192, #layers=24, Avg. span=245, #Params=209M, #FLOPS=179M2019.05 | 1.07 | — | |
| Adaptive Span2019.11 | 1.07 | — | |
| Adaptive Span TransformerModel Category=Best result*, Sequence length=5122023.08 | 1.07 | — | |
| Sandwichsandwich coefficient (k)=3/42019.11 | 1.076 | — | |
| Transformer-XL#Params=277M, Model scale=Large2019.07 | 1.08 | — | |
| All-attention network + adaptive span#Params=114M, Model scale=Large2019.07 | 1.08 | — | |
| T-XLModel Category=Large, #layers=24, Avg. span=3800, #Params=277M, #FLOPS=438M2019.05 | 1.08 | — | |
| Transformer-XL2019.11 | 1.08 | — | |
| TransformerXLModel type=ARM2021.02 | 1.08 | — | |
| Transformer XLModel steps=256, Layers=24, Context window=7842021.07 | 1.08 | — | |
| Adaptive Span (Baseline)Number of runs=52019.11 | 1.0802 | — | |
| Transformer-XL# of params=277M2019.04 | 1.085 | — | |
| Transformer-LS#Param=44M2021.07 | 1.09 | — | |
| Longformer#Param=41M2020.04 | 1.1 | — | |
| Longformer#Param=41M2021.07 | 1.1 | — | |
| Transformer + adaptive span#Params=38M, Model scale=Small2019.07 | 1.11 | — | |
| All-attention network + adaptive span#Params=38M, Model scale=Small2019.07 | 1.11 | — | |
| Adaptive-SpanModel Category=Small, Span limit (S)=8192, #layers=12, Avg. span=314, #Params=38M, #FLOPS=42M2019.05 | 1.11 | — | |
| Adaptive#Param=38M2020.04 | 1.11 | — | |
| BP-Transformer#Param=39M2020.04 | 1.11 | — | |
| Adaptive#Param=38M2021.07 | 1.11 | — | |
| BP-Transformer#Param=38M2021.07 | 1.11 | — | |
| T64Parameters (train)=235M, Parameters (inference)=219M2018.08 | 1.13 | — | |
| Transformer + aux losses# of params=235M2019.04 | 1.13 | — | |
| T64#Params=235M, Model scale=Large2019.07 | 1.13 | — | |
| T64Model Category=Large, #layers=64, Avg. span=512, #Params=235M, #FLOPS=120G2019.05 | 1.13 | — | |
| 64 Layer TransformerModel type=ARM2021.02 | 1.13 | — | |
| T12Parameters (train)=44M, Parameters (inference)=41M2018.08 | 1.18 | — | |
| T12#Params=44M, Model scale=Small2019.07 | 1.18 | — | |
| T12Model Category=Small, #layers=12, Avg. span=512, #Params=44M, #FLOPS=22G2019.05 | 1.18 | — | |
| T12#Param=44M2020.04 | 1.18 | — | |
| T12#Param=44M2021.07 | 1.18 | — | |
| Transformer decoderModel steps=2562021.07 | 1.18 | — | |
| mLSTM + dynamic evalparameters=45M, dynamic evaluation=true2017.09 | 1.19 | — | |
| mLSTM + dynamic evalParameters (train)=45M2018.08 | 1.19 | — | |
| Multiplicative LSTM + dynamic eval# of params=45M2019.04 | 1.19 | — | |
| Discrete FlowArchitecture=8 x 3 layers2021.07 | 1.23 | 0.16 | |
| Transformer decoderModel steps=2562021.07 | 1.23 | — | |
| Discrete FlowBackbone=8 x 3 layers Transformer2024.06 | 1.23 | — | |
| Transformer ARBackbone=12-layer Transformer2024.06 | 1.23 | — | |
| Discrete FlowModel Category=Flow-based models, Sequence length=2562023.08 | 1.23 | — | |
| TransformerModel Category=Autoregressive baseline, Sequence length=2562023.08 | 1.23 | — | |
| Large RHNSize=45 M, Recurrence depth=102016.07 | 1.27 | — | |
| Recurrent highway networksrecurrent depth=10, regularization=+VD2016.09 | 1.27 | — | |
| large mLSTMhidden dimensionality=1900, regularization=+emb +WN +VD, parameters=~45 million2016.09 | 1.27 | — | |
| Recurrent highway networksparameters=45M2017.09 | 1.27 | — | |
| mLSTMparameters=45M2017.09 | 1.27 | — | |
| Recurrent HighwayParameters (train)=45M, Parameters (inference)=45M2018.08 | 1.27 | — | |
| mLSTMParameters (train)=45M, Parameters (inference)=45M2018.08 | 1.27 | — | |
| Recurrent highway networks# of params=45M2019.04 | 1.27 | — | |
| Multiplicative LSTM# of params=45M2019.04 | 1.27 | — | |
| Recurrent highway networks#Params=45M, Model scale=Small2019.07 | 1.27 | — | |
| Large mLSTM#Params=45M, Model scale=Small2019.07 | 1.27 | — | |
| HM-LSTM2016.03 | 1.29 | — | |
| LayerNorm HM-LSTMLayer Normalization=true2016.09 | 1.29 | — | |
| LN HM-LSTMSize=35 M, Layer Normalization=true2016.07 | 1.29 | — | |
| RHNSize=20 M, Recurrence depth=102016.07 | 1.29 | — | |
| layer-norm hierarchical multiscale LSTM2016.09 | 1.29 | — | |
| Hierarchical multiscale LSTM2017.09 | 1.29 | — | |
| HM-LSTMParameters (train)=35M, Parameters (inference)=35M2018.08 | 1.29 | — | |
| HM-LSTM# of params=35M2019.04 | 1.29 | — | |
| LN HM-LSTM#Params=35M, Model scale=Small2019.07 | 1.29 | — | |
| HM-LSTMLayer Normalization=false2016.09 | 1.32 | — | |
| HM-LSTMSize=35 M2016.07 | 1.32 | — | |
| GenMD4Backbone=12-layer Transformer2024.06 | 1.34 | — | |
| BN-LSTMimplementation=ours2016.03 | 1.36 | — | |
| BatchNorm LSTM2016.09 | 1.36 | — | |
| BN LSTMSize=16 M2016.07 | 1.36 | — | |
| batch normalised LSTM2016.09 | 1.36 | — | |
| Batch normalised LSTM2017.09 | 1.36 | — | |
| BN-LSTM2018.08 | 1.36 | — | |
| MD4Backbone=12-layer Transformer2024.06 | 1.37 | — | |
| Argmax Flow, ARModel type=Generative Flow2021.02 | 1.39 | — | |
| AR Argmax FlowBackbone=12-layer Transformer2024.06 | 1.39 | — | |
| SEDD AbsorbBackbone=12-layer Transformer2024.06 | 1.39 | — | |
| mLSTMSize=10 M2016.07 | 1.4 | — | |
| unregularised mLSTMhidden dimensionality=1900, optimizer=RMSprop2016.09 | 1.4 | — | |
| MACBackbone=12-layer Transformer2024.06 | 1.4 | — | |
| MACModel Category=Order-agnostic Models, Sequence length=2562023.08 | 1.4 | — | |
| BFNBackbone=12-layer Transformer2024.06 | 1.41 | — | |
| BFNModel Category=Bayesian Flow Network, Sequence length=256, Parameters=170M, Batch size=33282023.08 | 1.41 | — | |
| LSTMimplementation=ours2016.03 | 1.43 | — | |
| LSTM2016.09 | 1.43 | — | |
| LSTM2017.09 | 1.43 | — | |
| LSTM2018.08 | 1.43 | — | |
| ARDMBackbone=12-layer Transformer2024.06 | 1.43 | — | |
| MI-LSTM2016.09 | 1.44 | — | |
| MI-LSTMSize=17 M2016.07 | 1.44 | — | |
| MI-LSTM2016.09 | 1.44 | — | |
| Multiplicative integration LSTMparameters=4M2017.09 | 1.44 | — | |
| CategoricalNF (AR)Model type=VAE2021.02 | 1.45 | — | |
| D3PM AbsorbBackbone=12-layer Transformer2024.06 | 1.45 | — | |
| skipping RNN2016.03 | 1.48 | — | |
| Skipping-RNN2016.09 | 1.48 | — | |
| PlaidBackbone=12-layer Transformer2024.06 | 1.48 | — | |
| MI-RNN2016.09 | 1.52 | — | |
| HF-MRNN2016.03 | 1.54 | — |