Language Modeling on WikiText-103 (val)
1.01PPLBiLSTM
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| BiLSTMNum Params=81,447,505, Size (MB)=310.7, Epoch=5, Maximum sequence length=2562025.12 | 1.01 | — | — | — | 0.01 | 0.02 | 99 | — | — | — | — | — | |
| Ensemble of AllWeight=12023.11 | 13.11 | — | — | — | — | — | — | — | — | — | — | — | |
| kNN LMWeight=0.532023.11 | 14.39 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL + RMS dynamic eval# of params=257M, Dynamic Evaluation=RMS2019.04 | 15.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive Transformerlayers=18L, memory_size=M=10242019.11 | 16 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive Transformer#Param.=257M, Memory length (M)=10242022.03 | 16 | — | — | — | — | — | — | — | — | — | — | — | |
| Compressive Transformer2022.02 | 16 | — | — | — | — | — | — | — | — | — | — | — | |
| Segatron-XL large + HCP#Param.=257M, Memory length (M)=3842022.03 | 16.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL + SGD dynamic eval# of params=257M, Dynamic Evaluation=SGD2019.04 | 16.3 | — | — | — | — | — | — | — | — | — | — | — | |
| MEGAWeight=0.292023.11 | 17.17 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL*# of params=257M, batch size=12019.04 | 17.3 | — | — | — | — | — | — | — | — | — | — | — | |
| ShortFormerWeight=0.092023.11 | 17.47 | — | — | — | — | — | — | — | — | — | — | — | |
| Shortformer2022.02 | 17.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL (ours)reimplementation=true2022.02 | 17.58 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext Length=81922022.02 | 17.58 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext Length=20482022.02 | 17.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext Length=40962022.02 | 17.66 | — | — | — | — | — | — | — | — | — | — | — | |
| SEQUENCE#Params=234M2021.04 | 17.71 | — | — | — | — | — | — | — | — | — | — | — | |
| CODAnumber of parameters=246.96M, context size (train/test)=512/4802021.05 | 17.81 | — | — | — | — | — | — | — | — | — | — | — | |
| Perceiver ARContext Length=10242022.02 | 17.86 | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMER+number of parameters=246.93M, context size (train/test)=3072/25602021.05 | 17.97 | — | — | — | — | — | — | — | — | — | — | — | |
| Adaptive inputs2022.02 | 18 | — | — | — | — | — | — | — | — | — | — | — | |
| Hybrid-GRUParams (M)=48.922026.02 | 18.07 | — | — | — | — | — | — | — | — | — | — | — | |
| UNCOMPRESSEDWeights=UNCOMPRESSED, Activation=None, Model Size (MB)=9422021.04 | 18.1 | — | — | — | — | — | — | — | — | — | — | — | |
| DIFFQlambda=5, g=16, Activation=8 BITS, Model Size (MB)=1302021.04 | 18.1 | — | — | — | — | — | — | — | — | — | — | — | |
| AdaptiveInputsWeight=0.032023.11 | 18.15 | — | — | — | — | — | — | — | — | — | — | — | |
| suGRU(100M)Params (M)=88.982026.02 | 18.29 | — | — | — | — | — | — | — | — | — | — | — | |
| Tensor-Transformercores=1, normalization=PowerNorm2020.03 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL2022.02 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | |
| 8 BITSWeights=8 BITS, Activation=8 BITS, Model Size (MB)=2362021.04 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | |
| UNCOMPRESSEDWeights=UNCOMPRESSED, Activation=None, Layer-drop=0.2, Model Size (MB)=9422021.04 | 18.3 | — | — | — | — | — | — | — | — | — | — | — | |
| TransformerImplementation Source=suGRU implementation performance, Params (M)=44.652026.02 | 18.32 | — | — | — | — | — | — | — | — | — | — | — | |
| TRANSFORMERnumber of parameters=246.93M, context size (train/test)=512/4802021.05 | 18.35 | — | — | — | — | — | — | — | — | — | — | — | |
| B2T connectionDecoder Layers=16L2022.06 | 18.38 | — | — | — | — | — | — | — | — | — | — | — | |
| Pre-LNDecoder Layers=16L2022.06 | 18.53 | — | — | — | — | — | — | — | — | — | — | — | |
| Baevski and Auli (2019)#Params=247M, is_re_run=true2021.04 | 18.53 | — | — | — | — | — | — | — | — | — | — | — | |
| GHTNumber of parameters=201M, Inference Speed=9.9 tok/s, FLOPs=6106M2023.05 | 18.57 | — | — | — | — | — | — | — | — | — | — | — | |
| GHT-PSNumber of parameters=167M, Inference Speed=19.0 tok/s, FLOPs=4573M2023.05 | 18.58 | — | — | — | — | — | — | — | — | — | — | — | |
| DIFFQlambda=10, g=16, Activation=8 BITS, Model Size (MB)=1132021.04 | 18.6 | — | — | — | — | — | — | — | — | — | — | — | |
| QNWeights=8 BITS, Activation=QN 8 BITS, Layer-drop=0.2, Model Size (MB)=2362021.04 | 18.7 | — | — | — | — | — | — | — | — | — | — | — | |
| LSQWeights=4 BITS, Activation=8 BITS, Model Size (MB)=1182021.04 | 18.9 | — | — | — | — | — | — | — | — | — | — | — | |
| Diff AttentionAttention=Diff Attention, Params=∼350M, d=1024, H=16, L=242026.04 | 18.93 | — | — | — | — | — | — | — | — | — | — | — | |
| SEQUENCE#Params=121M2021.04 | 18.97 | — | — | — | — | — | — | — | — | — | — | — | |
| CYCLE#Params=121M2021.04 | 19 | — | — | — | — | — | — | — | — | — | — | — | |
| VT w/ AINumber of parameters=201M, Inference Speed=9.9 tok/s, FLOPs=6106M2023.05 | 19.03 | — | — | — | — | — | — | — | — | — | — | — | |
| suGRUParams (M)=44.862026.02 | 19.28 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled QKAttention=Coupled QK, Params=∼350M, d=1024, H=16, L=242026.04 | 19.35 | — | — | — | — | — | — | — | — | — | — | — | |
| QNWeights=4 BITS, Activation=QN 8 BITS, Layer-drop=0.2, Model Size (MB)=1182021.04 | 19.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard (baseline)Attention=Standard (baseline), Params=∼350M, d=1024, H=16, L=242026.04 | 19.55 | — | — | — | — | — | — | — | — | — | — | — | |
| CYCLE (REV)#Params=121M2021.04 | 19.6 | — | — | — | — | — | — | — | — | — | — | — | |
| BERT-Large-CAS2019.04 | 19.67 | — | — | — | — | — | — | — | — | — | — | — | |
| BERT-L-CASNumber of parameters=395M2023.05 | 19.67 | — | — | — | — | — | — | — | — | — | — | — | |
| S4Number of parameters=249M2023.05 | 19.69 | — | — | — | — | — | — | — | — | — | — | — | |
| Tensor-Transformercores=22020.03 | 19.7 | — | — | — | — | — | — | — | — | — | — | — | |
| QATWeights=8 BITS, Activation=8 BITS, Model Size (MB)=2362021.04 | 19.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer + adaptive input# of params=247M2019.04 | 19.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Adaptive Input2020.03 | 19.8 | — | — | — | — | — | — | — | — | — | — | — | |
| Sub-tokenModel=Qwen2.5-72B, KV Retention=75%2026.04 | 19.97 | — | — | — | — | — | — | — | — | — | — | — | |
| Coupled QKParams=152.4M, d=768, H=12, L=122026.04 | 20.12 | — | — | — | — | — | — | — | — | — | — | — | |
| Post-LNDecoder Layers=6L2022.06 | 20.24 | — | — | — | — | — | — | — | — | — | — | — | |
| Vanilla#Params=121M2021.04 | 20.39 | — | — | — | — | — | — | — | — | — | — | — | |
| S4Weight=0.062023.11 | 20.47 | — | — | — | — | — | — | — | — | — | — | — | |
| B2T connectionDecoder Layers=6L2022.06 | 20.5 | — | — | — | — | — | — | — | — | — | — | — | |
| PQWeights=PQ, Activation=None, Layer-drop=0.2, Model Size (MB)=382021.04 | 20.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Glauber-UL2-LZero-shot=true, Model size=Large, N value=3, Note=upper bound2026.05 | 20.83 | — | — | — | — | — | — | — | — | — | — | — | |
| MoE-LoRAModel=Qwen2.5-72B, KV Retention=100%2026.04 | 20.93 | — | — | — | — | — | — | — | — | — | — | — | |
| Pre-LNDecoder Layers=6L2022.06 | 20.98 | — | — | — | — | — | — | — | — | — | — | — | |
| Diff AttentionParams=152.3M, d=768, H=12, L=122026.04 | 21.11 | — | — | — | — | — | — | — | — | — | — | — | |
| LoRAModel=Qwen2.5-72B, KV Retention=100%2026.04 | 21.28 | — | — | — | — | — | — | — | — | — | — | — | |
| Lite Transformer#Params=37.2M, #MAdds (100)=3.9G, #MAdds (1000)=48.7G, Speed (tokens/s)=10.2K2020.04 | 21.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Standard (baseline)Params=152.3M, d=768, H=12, L=122026.04 | 21.57 | — | — | — | — | — | — | — | — | — | — | — | |
| GQA (baseline)Params=144.4M, d=768, H=12, L=122026.04 | 21.87 | — | — | — | — | — | — | — | — | — | — | — | |
| Segatron-XL base + HCP#Param.=151M, Memory length (M)=1502022.03 | 21.9 | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-2-LZero-shot=true, Model size=Large2026.05 | 22.05 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL Base + URPE-based Attention#Params=151M2022.05 | 22.4 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=QUEST2026.03 | 22.436 | — | — | — | — | — | — | — | — | — | — | — | |
| Segatron-XL base#Param.=151M, Memory length (M)=1502022.03 | 22.5 | — | — | — | — | — | — | — | — | — | — | — | |
| MambaParams (M)=44.392026.02 | 22.58 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLModel size (Parameters)=Base (151M), Attention=Standard2026.03 | 22.65 | — | — | — | — | — | — | — | — | — | — | — | |
| Tensor-Transformercores=1, normalization=LayerNorm, re-implemented=true2020.03 | 22.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Universal#Params=121M2021.04 | 22.75 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XLsize=base2020.03 | 23.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-XL Base#Params=151M2022.05 | 23.1 | — | — | — | — | — | — | — | — | — | — | — | |
| HGRN2Params (M)=44.662026.02 | 23.1 | — | — | — | — | — | — | — | — | — | — | — | |
| Adaptive Inputs#Params=37.8M, #MAdds (100)=3.9G, #MAdds (1000)=50.3G, Speed (tokens/s)=7.6K2020.04 | 23.2 | — | — | — | — | — | — | — | — | — | — | — | |
| RFA-Gate-Gaussian2022.02 | 23.2 | — | — | — | — | — | — | — | — | — | — | — | |
| Sub-tokenModel=Qwen2.5-14B, KV Retention=75%2026.04 | 23.43 | — | — | — | — | — | — | — | — | — | — | — | |
| COSFORMER2022.02 | 23.5 | — | — | — | — | — | — | — | — | — | — | — | |
| Tensor-Transformercores=12020.03 | 23.6 | — | — | — | — | — | — | — | — | — | — | — | |
| Tensor-Transformercores=1, normalization=PowerNorm-V2020.03 | 23.6 | — | — | — | — | — | — | — | — | — | — | — | |
| SPADE + SMRMode=Convolution, SMR=Included2024.05 | 23.68 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer Large + HCP#Param.=257M2022.03 | 23.7 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer-LSSMR=Excluded2024.05 | 23.71 | — | — | — | — | — | — | — | — | — | — | — | |
| Sub-tokenModel=Qwen2.5-32B, KV Retention=75%2026.04 | 23.78 | — | — | — | — | — | — | — | — | — | — | — | |
| S6 + SMRMode=Recurrence, SMR=Included2024.05 | 23.85 | — | — | — | — | — | — | — | — | — | — | — | |
| S6Mode=Recurrence, SMR=Excluded2024.05 | 23.97 | — | — | — | — | — | — | — | — | — | — | — | |
| TNNArchitecture Type=FFT-based, Params (M)=48.68, Setting=TNN[59]2023.11 | 23.98 | — | — | — | — | — | — | — | — | — | — | — | |
| Transformer Large#Param.=257M2022.03 | 24 | — | — | — | — | — | — | — | — | — | — | — | |
| HGRNArchitecture Type=Ours, Params (M)=46.25, Setting=TNN[59]2023.11 | 24.14 | — | — | — | — | — | — | — | — | — | — | — | |
| SPADEMode=Convolution, SMR=Excluded2024.05 | 24.18 | — | — | — | — | — | — | — | — | — | — | — |