List your company
ResearchFormerly WizWand

Character-level Language Modeling on text8 (test)

1.038BPCTransformer-XL + RMS dynamic eval + decay
1.017921.153461.2891.42454Mar 30, 2016Aug 10, 2017Dec 21, 2018May 3, 2020Sep 13, 2021Jan 24, 2023Jun 6, 2024
Updated 8mo ago

Evaluation results

MethodLinks
Transformer-XL + RMS dynamic eval + decay# of params=277M2019.041.038—
Transformer-XL + SGD dynamic eval# of params=277M2019.041.042—
Transformer + adaptive span#Params=209M, Model scale=Large2019.071.07—
Adaptive-SpanModel Category=Large, Span limit (S)=8192, #layers=24, Avg. span=245, #Params=209M, #FLOPS=179M2019.051.07—
Adaptive Span2019.111.07—
Adaptive Span TransformerModel Category=Best result*, Sequence length=5122023.081.07—
Sandwichsandwich coefficient (k)=3/42019.111.076—
Transformer-XL#Params=277M, Model scale=Large2019.071.08—
All-attention network + adaptive span#Params=114M, Model scale=Large2019.071.08—
T-XLModel Category=Large, #layers=24, Avg. span=3800, #Params=277M, #FLOPS=438M2019.051.08—
Transformer-XL2019.111.08—
TransformerXLModel type=ARM2021.021.08—
Transformer XLModel steps=256, Layers=24, Context window=7842021.071.08—
Adaptive Span (Baseline)Number of runs=52019.111.0802—
Transformer-XL# of params=277M2019.041.085—
Transformer-LS#Param=44M2021.071.09—
Longformer#Param=41M2020.041.1—
Longformer#Param=41M2021.071.1—
Transformer + adaptive span#Params=38M, Model scale=Small2019.071.11—
All-attention network + adaptive span#Params=38M, Model scale=Small2019.071.11—
Adaptive-SpanModel Category=Small, Span limit (S)=8192, #layers=12, Avg. span=314, #Params=38M, #FLOPS=42M2019.051.11—
Adaptive#Param=38M2020.041.11—
BP-Transformer#Param=39M2020.041.11—
Adaptive#Param=38M2021.071.11—
BP-Transformer#Param=38M2021.071.11—
T64Parameters (train)=235M, Parameters (inference)=219M2018.081.13—
Transformer + aux losses# of params=235M2019.041.13—
T64#Params=235M, Model scale=Large2019.071.13—
T64Model Category=Large, #layers=64, Avg. span=512, #Params=235M, #FLOPS=120G2019.051.13—
64 Layer TransformerModel type=ARM2021.021.13—
T12Parameters (train)=44M, Parameters (inference)=41M2018.081.18—
T12#Params=44M, Model scale=Small2019.071.18—
T12Model Category=Small, #layers=12, Avg. span=512, #Params=44M, #FLOPS=22G2019.051.18—
T12#Param=44M2020.041.18—
T12#Param=44M2021.071.18—
Transformer decoderModel steps=2562021.071.18—
mLSTM + dynamic evalparameters=45M, dynamic evaluation=true2017.091.19—
mLSTM + dynamic evalParameters (train)=45M2018.081.19—
Multiplicative LSTM + dynamic eval# of params=45M2019.041.19—
Discrete FlowArchitecture=8 x 3 layers2021.071.230.16
Transformer decoderModel steps=2562021.071.23—
Discrete FlowBackbone=8 x 3 layers Transformer2024.061.23—
Transformer ARBackbone=12-layer Transformer2024.061.23—
Discrete FlowModel Category=Flow-based models, Sequence length=2562023.081.23—
TransformerModel Category=Autoregressive baseline, Sequence length=2562023.081.23—
Large RHNSize=45 M, Recurrence depth=102016.071.27—
Recurrent highway networksrecurrent depth=10, regularization=+VD2016.091.27—
large mLSTMhidden dimensionality=1900, regularization=+emb +WN +VD, parameters=~45 million2016.091.27—
Recurrent highway networksparameters=45M2017.091.27—
mLSTMparameters=45M2017.091.27—
Recurrent HighwayParameters (train)=45M, Parameters (inference)=45M2018.081.27—
mLSTMParameters (train)=45M, Parameters (inference)=45M2018.081.27—
Recurrent highway networks# of params=45M2019.041.27—
Multiplicative LSTM# of params=45M2019.041.27—
Recurrent highway networks#Params=45M, Model scale=Small2019.071.27—
Large mLSTM#Params=45M, Model scale=Small2019.071.27—
HM-LSTM2016.031.29—
LayerNorm HM-LSTMLayer Normalization=true2016.091.29—
LN HM-LSTMSize=35 M, Layer Normalization=true2016.071.29—
RHNSize=20 M, Recurrence depth=102016.071.29—
layer-norm hierarchical multiscale LSTM2016.091.29—
Hierarchical multiscale LSTM2017.091.29—
HM-LSTMParameters (train)=35M, Parameters (inference)=35M2018.081.29—
HM-LSTM# of params=35M2019.041.29—
LN HM-LSTM#Params=35M, Model scale=Small2019.071.29—
HM-LSTMLayer Normalization=false2016.091.32—
HM-LSTMSize=35 M2016.071.32—
GenMD4Backbone=12-layer Transformer2024.061.34—
BN-LSTMimplementation=ours2016.031.36—
BatchNorm LSTM2016.091.36—
BN LSTMSize=16 M2016.071.36—
batch normalised LSTM2016.091.36—
Batch normalised LSTM2017.091.36—
BN-LSTM2018.081.36—
MD4Backbone=12-layer Transformer2024.061.37—
Argmax Flow, ARModel type=Generative Flow2021.021.39—
AR Argmax FlowBackbone=12-layer Transformer2024.061.39—
SEDD AbsorbBackbone=12-layer Transformer2024.061.39—
mLSTMSize=10 M2016.071.4—
unregularised mLSTMhidden dimensionality=1900, optimizer=RMSprop2016.091.4—
MACBackbone=12-layer Transformer2024.061.4—
MACModel Category=Order-agnostic Models, Sequence length=2562023.081.4—
BFNBackbone=12-layer Transformer2024.061.41—
BFNModel Category=Bayesian Flow Network, Sequence length=256, Parameters=170M, Batch size=33282023.081.41—
LSTMimplementation=ours2016.031.43—
LSTM2016.091.43—
LSTM2017.091.43—
LSTM2018.081.43—
ARDMBackbone=12-layer Transformer2024.061.43—
MI-LSTM2016.091.44—
MI-LSTMSize=17 M2016.071.44—
MI-LSTM2016.091.44—
Multiplicative integration LSTMparameters=4M2017.091.44—
CategoricalNF (AR)Model type=VAE2021.021.45—
D3PM AbsorbBackbone=12-layer Transformer2024.061.45—
skipping RNN2016.031.48—
Skipping-RNN2016.091.48—
PlaidBackbone=12-layer Transformer2024.061.48—
MI-RNN2016.091.52—
HF-MRNN2016.031.54—