List your company
ResearchFormerly WizWand

Character-level Language Modeling on enwik8 (test)

0.94BPCTransformer-XL + RMS dynamic eval + decay
0.929680.999341.0691.13866May 24, 2017Jun 22, 2018Jul 21, 2019Aug 18, 2020Sep 16, 2021Oct 15, 2022Nov 14, 2023
Updated 8mo ago

Evaluation results

MethodLinks
Transformer-XL + RMS dynamic eval + decay# of params=277M, Dynamic evaluation=RMS + decay2019.040.94———
Transformer-XL + SGD dynamic eval# of params=277M, Dynamic evaluation=SGD2019.040.946———
Expire-SpanNumber of parameters=277M, Layers=242021.100.95———
Feedback TransformerNumber of parameters=77M2021.100.96———
Sandwichsandwich coefficient (k)=2/42019.110.968———
24L Compressive TransformerLayers=24, Backbone=Compressive Transformer, Sequence window size=768, Memory size=768, Compressed memory size=11522019.110.97———
Compressive2019.110.97———
Compressive#Param=277M2020.040.97———
Compressive#Param=227M2021.070.97———
Transformer-LS#Param=110M2021.070.97———
Transformer-LSNumber of parameters=110M2021.100.97———
Adaptive Span (Baseline)Number of runs=52019.110.9752———
Transformer + adaptive span#Params=209M, Model size category=Large models2019.070.98———
All-attention network + adaptive span#Params=114M, Model size category=Large models2019.070.98———
Adaptive#layers=24, #Params=209M, #FLOPS=181M, Span limit S=81922019.050.98———
Adaptive TransformerBackbone=Adaptive Transformer2019.110.98———
24L Transformer-XLLayers=24, Backbone=Transformer-XL, Implementation=ours2019.110.98———
Adaptive Span2019.110.98———
Adaptive#Param=209M2020.040.98———
Adaptive#Param=209M2021.070.98———
HourglassNumber of parameters=146M2021.100.98———
Adaptive-SpanNumber of parameters=209M, Layers=242021.100.98———
Transformer-XL#Param=277M, Layers=24L2019.010.99———
Transformer-XL 24l#Params=277M, Model size category=Large models2019.070.99———
Sparse Transformer (fixed)#Params=95M, Model size category=Large models2019.070.99———
T-XL#layers=24, #Params=277M, #FLOPS=438M2019.050.99———
24L Transformer-XLLayers=24, Backbone=Transformer-XL2019.110.99———
Sparse TransformerLayers=30, Heads=240, Params=95M2019.110.99———
Transformer-XL2019.110.99———
Sparse#Param=≈100M2020.040.99———
Transformer-XL#Param=277M, layers=242020.040.99———
Routing#Param=≈223M2020.040.99———
Longformer#Param=102M2020.040.99———
Transformer-LS#Param=44M2021.070.99———
Transformer-XL#Param=277M2021.070.99———
Routing#Param=223M2021.070.99———
Longformer#Param=102M2021.070.99———
Sparse#Param=95M2021.070.99———
Transformer-XLNumber of parameters=277M, Layers=242021.100.99———
Sparse TransformerBackbone=Sparse Transformer2019.110.991———
Transformer-XL# of params=277M, Dynamic evaluation=false2019.040.993———
Longformer#Param=41M2020.041———
Longformer#Param=41M2021.071———
All-attention network + adaptive span#Params=39M, Model size category=Small models2019.071.01———
Transformer + adaptive span#Params=39M, Model size category=Small models2019.071.02———
Adaptive#layers=12, #Params=39M, #FLOPS=41M, Span limit S=81922019.051.02———
Adaptive TransformerLayers=12, Heads=96, Params=39M2019.111.02———
Adaptive#Param=39M2020.041.02———
BP-Transformer#Param=38M2020.041.02———
Adaptive#Param=38M2021.071.02———
BP-Transformer#Param=38M2021.071.02———
MEGA#Param.=39M2022.091.02———
Adaptive Span#Param.=39M2023.061.02———
MEGA-chunk#Param.=39M2023.061.02—11
SeqBoat#Param.=39M2023.061.02—1.161.07
Adaptive Attention SpanContext=8192, #Params=39M2023.111.02———
Transformer-XL#Param=88M, Layers=18L2019.011.03———
Transformer-XL 18l#Params=88M, Model size category=Large models2019.071.03———
T-XL#layers=18, #Params=88M, #FLOPS=329M2019.051.03———
Transformer-XLLayers=18, Heads=160, Params=88M2019.111.03———
Transformer-XL#Param=88M, layers=182020.041.03———
Transformer-XL#Param=88M2021.071.03———
Skip Cross-Head Transformer-XLContext=3800, #Params=41M2023.111.033———
Skip Cross-Head Transformer-XLContext=1600, #Params=41M2023.111.037———
Reformer2020.041.05———
Reformer#Param=not specified2021.071.05———
Transformer + aux losses# of params=235M, Dynamic evaluation=false2019.041.06———
Transformer-XL#Param=41M, Layers=12L2019.011.06———
64L Transformer#Param=235M2019.011.06———
Transformer-XL#Params=41M, Model size category=Small models2019.071.06———
T64#Params=235M, Model size category=Large models2019.071.06———
T-XL#layers=12, #Params=41M, #FLOPS=64M2019.051.06———
T64#layers=64, #Params=235M, #FLOPS=120G2019.051.06———
64L TransformerLayers=64, Backbone=Transformer2019.111.06———
T64Layers=64, Heads=128, Params=235M2019.111.06———
Transformer-XLLayers=12, Heads=160, Params=41M2019.111.06———
Transformer XL#Param=41M2020.041.06———
Transformer-XL#Param=41M2021.071.06———
XFM-XL#Param.=41M2022.091.06———
Transformer-XL#Param.=41M2023.061.06———
64L Transformer#Params=235M2023.111.06———
Transformer-XLContext=3800, #Params=41M2023.111.06———
SHA-LSTMLayers=4, Heads=4, Params=54M, Hidden size (h)=1024, Attention=attention head per layer2019.111.068———
SHA-LSTMLayers=4, Heads=1, Params=52M, Hidden size (h)=1024, Attention=single attention head2019.111.076———
Multiplicative LSTM + dynamic eval# of params=46M, Dynamic evaluation=true2019.041.08———
12L Transformer#Param=44M2019.011.11———
T12#Params=44M, Model size category=Small models2019.071.11———
T12#layers=12, #Params=44M, #FLOPS=22G2019.051.11———
T12Layers=12, Heads=24, Params=44M2019.111.11———
T12#Param=44M2020.041.11———
T12#Param=44M2021.071.11———
12L Transformer#Params=44M2023.111.11———
TransformerL=24, d=256, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.051.13———
TransformerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.051.137———
Mogrifier LSTMHeads=0, Params=48M2019.111.146———
RWKV-RNNL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Optimizer=AdamW, Weight Decay=0.1, Dropout=0.1, Batch Size=16, Initial Learning Rate=6e-42023.051.178———
SRU (with projection)Size=49m, #layers=12, Unroll size=256, Time=41min, projection dimension=5122017.091.19———
LSTMHeads=0, Params=48M2019.111.195———
ReformerL=12, d=512, T=1024, Time Complexity=O(T log T d), Space Complexity=O(T log T + Td)2023.051.195———
2 x Large FS-LSTM-4Param Count=2 x 47M2017.051.198———