Character-level Language Modeling on enwik8 (test)
0.94BPCTransformer-XL + RMS dynamic eval + decay
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| Transformer-XL + RMS dynamic eval + decay# of params=277M, Dynamic evaluation=RMS + decay2019.04 | 0.94 | — | — | — | |
| Transformer-XL + SGD dynamic eval# of params=277M, Dynamic evaluation=SGD2019.04 | 0.946 | — | — | — | |
| Expire-SpanNumber of parameters=277M, Layers=242021.10 | 0.95 | — | — | — | |
| Feedback TransformerNumber of parameters=77M2021.10 | 0.96 | — | — | — | |
| Sandwichsandwich coefficient (k)=2/42019.11 | 0.968 | — | — | — | |
| 24L Compressive TransformerLayers=24, Backbone=Compressive Transformer, Sequence window size=768, Memory size=768, Compressed memory size=11522019.11 | 0.97 | — | — | — | |
| Compressive2019.11 | 0.97 | — | — | — | |
| Compressive#Param=277M2020.04 | 0.97 | — | — | — | |
| Compressive#Param=227M2021.07 | 0.97 | — | — | — | |
| Transformer-LS#Param=110M2021.07 | 0.97 | — | — | — | |
| Transformer-LSNumber of parameters=110M2021.10 | 0.97 | — | — | — | |
| Adaptive Span (Baseline)Number of runs=52019.11 | 0.9752 | — | — | — | |
| Transformer + adaptive span#Params=209M, Model size category=Large models2019.07 | 0.98 | — | — | — | |
| All-attention network + adaptive span#Params=114M, Model size category=Large models2019.07 | 0.98 | — | — | — | |
| Adaptive#layers=24, #Params=209M, #FLOPS=181M, Span limit S=81922019.05 | 0.98 | — | — | — | |
| Adaptive TransformerBackbone=Adaptive Transformer2019.11 | 0.98 | — | — | — | |
| 24L Transformer-XLLayers=24, Backbone=Transformer-XL, Implementation=ours2019.11 | 0.98 | — | — | — | |
| Adaptive Span2019.11 | 0.98 | — | — | — | |
| Adaptive#Param=209M2020.04 | 0.98 | — | — | — | |
| Adaptive#Param=209M2021.07 | 0.98 | — | — | — | |
| HourglassNumber of parameters=146M2021.10 | 0.98 | — | — | — | |
| Adaptive-SpanNumber of parameters=209M, Layers=242021.10 | 0.98 | — | — | — | |
| Transformer-XL#Param=277M, Layers=24L2019.01 | 0.99 | — | — | — | |
| Transformer-XL 24l#Params=277M, Model size category=Large models2019.07 | 0.99 | — | — | — | |
| Sparse Transformer (fixed)#Params=95M, Model size category=Large models2019.07 | 0.99 | — | — | — | |
| T-XL#layers=24, #Params=277M, #FLOPS=438M2019.05 | 0.99 | — | — | — | |
| 24L Transformer-XLLayers=24, Backbone=Transformer-XL2019.11 | 0.99 | — | — | — | |
| Sparse TransformerLayers=30, Heads=240, Params=95M2019.11 | 0.99 | — | — | — | |
| Transformer-XL2019.11 | 0.99 | — | — | — | |
| Sparse#Param=≈100M2020.04 | 0.99 | — | — | — | |
| Transformer-XL#Param=277M, layers=242020.04 | 0.99 | — | — | — | |
| Routing#Param=≈223M2020.04 | 0.99 | — | — | — | |
| Longformer#Param=102M2020.04 | 0.99 | — | — | — | |
| Transformer-LS#Param=44M2021.07 | 0.99 | — | — | — | |
| Transformer-XL#Param=277M2021.07 | 0.99 | — | — | — | |
| Routing#Param=223M2021.07 | 0.99 | — | — | — | |
| Longformer#Param=102M2021.07 | 0.99 | — | — | — | |
| Sparse#Param=95M2021.07 | 0.99 | — | — | — | |
| Transformer-XLNumber of parameters=277M, Layers=242021.10 | 0.99 | — | — | — | |
| Sparse TransformerBackbone=Sparse Transformer2019.11 | 0.991 | — | — | — | |
| Transformer-XL# of params=277M, Dynamic evaluation=false2019.04 | 0.993 | — | — | — | |
| Longformer#Param=41M2020.04 | 1 | — | — | — | |
| Longformer#Param=41M2021.07 | 1 | — | — | — | |
| All-attention network + adaptive span#Params=39M, Model size category=Small models2019.07 | 1.01 | — | — | — | |
| Transformer + adaptive span#Params=39M, Model size category=Small models2019.07 | 1.02 | — | — | — | |
| Adaptive#layers=12, #Params=39M, #FLOPS=41M, Span limit S=81922019.05 | 1.02 | — | — | — | |
| Adaptive TransformerLayers=12, Heads=96, Params=39M2019.11 | 1.02 | — | — | — | |
| Adaptive#Param=39M2020.04 | 1.02 | — | — | — | |
| BP-Transformer#Param=38M2020.04 | 1.02 | — | — | — | |
| Adaptive#Param=38M2021.07 | 1.02 | — | — | — | |
| BP-Transformer#Param=38M2021.07 | 1.02 | — | — | — | |
| MEGA#Param.=39M2022.09 | 1.02 | — | — | — | |
| Adaptive Span#Param.=39M2023.06 | 1.02 | — | — | — | |
| MEGA-chunk#Param.=39M2023.06 | 1.02 | — | 1 | 1 | |
| SeqBoat#Param.=39M2023.06 | 1.02 | — | 1.16 | 1.07 | |
| Adaptive Attention SpanContext=8192, #Params=39M2023.11 | 1.02 | — | — | — | |
| Transformer-XL#Param=88M, Layers=18L2019.01 | 1.03 | — | — | — | |
| Transformer-XL 18l#Params=88M, Model size category=Large models2019.07 | 1.03 | — | — | — | |
| T-XL#layers=18, #Params=88M, #FLOPS=329M2019.05 | 1.03 | — | — | — | |
| Transformer-XLLayers=18, Heads=160, Params=88M2019.11 | 1.03 | — | — | — | |
| Transformer-XL#Param=88M, layers=182020.04 | 1.03 | — | — | — | |
| Transformer-XL#Param=88M2021.07 | 1.03 | — | — | — | |
| Skip Cross-Head Transformer-XLContext=3800, #Params=41M2023.11 | 1.033 | — | — | — | |
| Skip Cross-Head Transformer-XLContext=1600, #Params=41M2023.11 | 1.037 | — | — | — | |
| Reformer2020.04 | 1.05 | — | — | — | |
| Reformer#Param=not specified2021.07 | 1.05 | — | — | — | |
| Transformer + aux losses# of params=235M, Dynamic evaluation=false2019.04 | 1.06 | — | — | — | |
| Transformer-XL#Param=41M, Layers=12L2019.01 | 1.06 | — | — | — | |
| 64L Transformer#Param=235M2019.01 | 1.06 | — | — | — | |
| Transformer-XL#Params=41M, Model size category=Small models2019.07 | 1.06 | — | — | — | |
| T64#Params=235M, Model size category=Large models2019.07 | 1.06 | — | — | — | |
| T-XL#layers=12, #Params=41M, #FLOPS=64M2019.05 | 1.06 | — | — | — | |
| T64#layers=64, #Params=235M, #FLOPS=120G2019.05 | 1.06 | — | — | — | |
| 64L TransformerLayers=64, Backbone=Transformer2019.11 | 1.06 | — | — | — | |
| T64Layers=64, Heads=128, Params=235M2019.11 | 1.06 | — | — | — | |
| Transformer-XLLayers=12, Heads=160, Params=41M2019.11 | 1.06 | — | — | — | |
| Transformer XL#Param=41M2020.04 | 1.06 | — | — | — | |
| Transformer-XL#Param=41M2021.07 | 1.06 | — | — | — | |
| XFM-XL#Param.=41M2022.09 | 1.06 | — | — | — | |
| Transformer-XL#Param.=41M2023.06 | 1.06 | — | — | — | |
| 64L Transformer#Params=235M2023.11 | 1.06 | — | — | — | |
| Transformer-XLContext=3800, #Params=41M2023.11 | 1.06 | — | — | — | |
| SHA-LSTMLayers=4, Heads=4, Params=54M, Hidden size (h)=1024, Attention=attention head per layer2019.11 | 1.068 | — | — | — | |
| SHA-LSTMLayers=4, Heads=1, Params=52M, Hidden size (h)=1024, Attention=single attention head2019.11 | 1.076 | — | — | — | |
| Multiplicative LSTM + dynamic eval# of params=46M, Dynamic evaluation=true2019.04 | 1.08 | — | — | — | |
| 12L Transformer#Param=44M2019.01 | 1.11 | — | — | — | |
| T12#Params=44M, Model size category=Small models2019.07 | 1.11 | — | — | — | |
| T12#layers=12, #Params=44M, #FLOPS=22G2019.05 | 1.11 | — | — | — | |
| T12Layers=12, Heads=24, Params=44M2019.11 | 1.11 | — | — | — | |
| T12#Param=44M2020.04 | 1.11 | — | — | — | |
| T12#Param=44M2021.07 | 1.11 | — | — | — | |
| 12L Transformer#Params=44M2023.11 | 1.11 | — | — | — | |
| TransformerL=24, d=256, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.05 | 1.13 | — | — | — | |
| TransformerL=12, d=512, T=1024, Time Complexity=O(T^2d), Space Complexity=O(T^2 + Td)2023.05 | 1.137 | — | — | — | |
| Mogrifier LSTMHeads=0, Params=48M2019.11 | 1.146 | — | — | — | |
| RWKV-RNNL=12, d=512, T=1024, Time Complexity=O(Td), Space Complexity=O(d), Optimizer=AdamW, Weight Decay=0.1, Dropout=0.1, Batch Size=16, Initial Learning Rate=6e-42023.05 | 1.178 | — | — | — | |
| SRU (with projection)Size=49m, #layers=12, Unroll size=256, Time=41min, projection dimension=5122017.09 | 1.19 | — | — | — | |
| LSTMHeads=0, Params=48M2019.11 | 1.195 | — | — | — | |
| ReformerL=12, d=512, T=1024, Time Complexity=O(T log T d), Space Complexity=O(T log T + Td)2023.05 | 1.195 | — | — | — | |
| 2 x Large FS-LSTM-4Param Count=2 x 47M2017.05 | 1.198 | — | — | — |