Language Modeling on One Billion Word Benchmark (test)
20.25Test PerplexityH-Transformer-1D
Evaluation results
| Method | Links | ||
|---|---|---|---|
| H-Transformer-1DNr=16, parameters=144M, embedding size=1024, feed-forward module size=40962021.07 | 20.25 | — | |
| Transformer-XL Large#Param=0.8B2019.01 | 21.8 | — | |
| Transformer-XLparameters=800M2021.07 | 21.8 | — | |
| Adaptive inputs (very large)Train Time (hours)=145, Parameters=1026M, GPUs=64, N=24, eff=8192, e=15362018.09 | 23.02 | — | |
| Baevski and Auliparameters=1000M2021.07 | 23.02 | — | |
| BIG GLSTM-G4Hardware=8 GPUs2016.12 | 23.3 | — | |
| Transformer-XL Base#Param=0.46B2019.01 | 23.5 | — | |
| Transformer-XLparameters=465M2021.07 | 23.5 | — | |
| 10 LSTMs + SNM10-SKIP2016.02 | 23.7 | — | |
| Józefowicz et al. (2016)Type=Word2018.08 | 23.7 | — | |
| 10 LSTMs + SNM10-SKIPCitation=Shazeer et al. (2016)2018.09 | 23.7 | — | |
| Adaptive Input#Param=1.0B2019.01 | 23.7 | — | |
| Adaptive inputs (large)Train Time (hours)=72, Parameters=465M, GPUs=64, N=20, eff=61442018.09 | 23.91 | — | |
| Baevski and Auliparameters=465M2021.07 | 23.91 | — | |
| H-Transformer-1DNr=16, parameters=53M, embedding size=512, feed-forward module size=20482021.07 | 23.95 | — | |
| Mesh Tensorflow#Param=4.9B2019.01 | 24 | — | |
| Shazeer et al.parameters=4900M2021.07 | 24 | — | |
| Adaptive Input#Param=0.46B2019.01 | 24.1 | — | |
| Transformer baselineparameters=144M, embedding size=1024, feed-forward module size=40962021.07 | 24.8 | — | |
| 10 LSTMs + KN-5Weighting=Optimal weights2016.02 | 25.1 | — | |
| Adaptive inputsTrain Time (hours)=55, Parameters=331M, GPUs=642018.09 | 25.22 | — | |
| 10 LSTMs + KN-5Weighting=Equal weights2016.02 | 25.3 | — | |
| Char-CNNTrain Time (hours)=79, Parameters=366M, GPUs=642018.09 | 25.88 | — | |
| LDDM-MDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=false2025.10 | 25.95 | — | |
| 10 Best LSTM ModelsWeighting=Optimal weights2016.02 | 26.1 | — | |
| 10 Best LSTM ModelsWeighting=Equal weights2016.02 | 26.3 | — | |
| MDLMDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 27.6 | — | |
| Shazeer et al. (2017)Parameters=4,371M, Note=Does not include embedding and softmax layers2018.09 | 28 | — | |
| High-Budget MoE#Param=~5B2019.01 | 28 | — | |
| SEDD AbsorbDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 28.39 | — | |
| LDDM-UDiffusion Type=Uniform Diffusion, Training Steps=1 million, Retrained status=false2025.10 | 29.21 | — | |
| BIG LSTM+CNN INPUTSNumber of Params [Billions]=1.042016.02 | 30 | — | |
| CNN-8192-1024Hid. #=8192, Layer #=2, RNN Parameters (10^7)=15.1, Other Parameters (10^7)=89, Softmax Type=Sampled softmax w. projection2018.04 | 30 | — | |
| Józefowicz et al. (2016)Parameters=1,040M2018.09 | 30 | — | |
| LSTM + CNN Input#Param=1.04B2019.01 | 30 | — | |
| Transformer baselineparameters=53M, embedding size=512, feed-forward module size=20482021.07 | 30.04 | — | |
| 2-LAYER LSTM-8192-1024 (BIG LSTM)Number of Params [Billions]=1.82016.02 | 30.6 | — | |
| 2-layer LSTM-8192-1024Hardware=32 GPUs2016.12 | 30.6 | — | |
| 8192-1024Hid. #=8192, Layer #=2, RNN Parameters (10^7)=15.1, Other Parameters (10^7)=163, Softmax Type=Sampled softmax w. projection2018.04 | 30.6 | — | |
| LSTM#Param=1.8B2019.01 | 30.6 | — | |
| UDLMDiffusion Type=Uniform Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 31.11 | — | |
| GCNN-14 BottleneckHardware=8 GPUs2016.12 | 31.9 | — | |
| Dauphin et al. (2017)Parameters=428M2018.09 | 31.9 | — | |
| GCNN-14 bottleneck2019.01 | 31.9 | — | |
| TransformerType=Autoregressive2023.10 | 31.98 | — | |
| LSTM-8192-2048 (50% DROPOUT)Number of Params [Billions]=3.32016.02 | 32.2 | — | |
| SEDD AbsorbType=Ours (Diffusion)2023.10 | 32.79 | — | |
| Low-Budget MoE#Param=~5B2019.01 | 34.1 | — | |
| BIG LSTM+CNN INPUTS + CNN SOFTMAX + 128-DIM CORRECTIONNumber of Params [Billions]=0.392016.02 | 35.8 | — | |
| G-LSTM-22019.01 | 36 | — | |
| MoSParameters=113M2017.11 | 37.1 | — | |
| LSTM-8192-2048 (NO DROPOUT)Number of Params [Billions]=3.32016.02 | 37.9 | — | |
| GCNN-13Hardware=1 GPU2016.12 | 38.1 | — | |
| CNN-4096-512Hid. #=4096, Layer #=2, RNN Parameters (10^7)=3.8, Other Parameters (10^7)=40.6, Softmax Type=Sampled softmax w. projection2018.04 | 39.7 | — | |
| BIG LSTM+CNN INPUTS + CNN SOFTMAXNumber of Params [Billions]=0.292016.02 | 39.8 | — | |
| 2-layer LSTM-2048Hardware=1 GPU2016.12 | 39.8 | — | |
| 2048-AdaptiveHid. #=2048, Layer #=2, RNN Parameters (10^7)=5.2, Other Parameters (10^7)=26.5, Softmax Type=Adaptive softmax2018.04 | 39.8 | — | |
| SEDD UniformType=Ours (Diffusion)2023.10 | 40.25 | — | |
| vanilla LSTMHid. #=2048, Layer #=2, RNN Parameters (10^7)=5.3, Other Parameters (10^7)=25.6, Softmax Type=Adaptive softmax2018.04 | 40.27 | — | |
| USMoEBackbone=Transformer-XL(420M), Topk=22025.03 | 40.53 | — | |
| T64Type=Byte2018.08 | 40.6 | 1.03 | |
| USMoEBackbone=Transformer-XL(420M), Topk=42025.03 | 40.9 | — | |
| Large Ensemble2016.02 | 41 | — | |
| RNN+SNM10-SKIP2016.02 | 41.3 | — | |
| RNN+KN-52016.02 | 42 | — | |
| RNN+KN-52016.02 | 42.4 | — | |
| SoftmaxParameters=119M2017.11 | 42.77 | — | |
| USMoEBackbone=Transformer-XL(420M), Topk=82025.03 | 43.24 | — | |
| ECBackbone=Transformer-XL(420M), Topk=22025.03 | 43.39 | — | |
| LSTM-2048-512Number of Params [Billions]=0.832016.02 | 43.7 | — | |
| LSTM-2048-512Hardware=32 GPUs2016.12 | 43.7 | — | |
| ECBackbone=Transformer-XL(420M), Topk=42025.03 | 43.7 | — | |
| Large Ensemble2016.02 | 43.8 | — | |
| LSTM-2048Hardware=1 GPU2016.12 | 43.9 | — | |
| ECBackbone=Transformer-XL(420M), Topk=82025.03 | 44.22 | — | |
| TCBackbone=Transformer-XL(420M), Topk=22025.03 | 44.56 | — | |
| LD-NetLayer-wise Dropout=No, Hid. #=300, Layer #=10, RNN Parameters (10^7)=2.3, Other Parameters (10^7)=24.2, Softmax Type=Adaptive softmax2018.04 | 45.14 | — | |
| TCBackbone=Transformer-XL(420M), Topk=42025.03 | 45.52 | — | |
| TCBackbone=Transformer-XL(420M), Topk=82025.03 | 46.36 | — | |
| 2048-512Hid. #=2048, Layer #=1, RNN Parameters (10^7)=0.9, Other Parameters (10^7)=40.6, Softmax Type=Sampled softmax w. projection2018.04 | 47.5 | — | |
| BIG LSTM+CNN INPUTS + CHAR LSTM PREDICTIONSNumber of Params [Billions]=0.232016.02 | 47.9 | — | |
| LSTM-1024-512Number of Params [Billions]=0.822016.02 | 48.2 | — | |
| vanilla LSTMHid. #=1600, Layer #=2, RNN Parameters (10^7)=3.2, Other Parameters (10^7)=24.2, Softmax Type=Adaptive softmax2018.04 | 48.85 | — | |
| LD-NetLayer-wise Dropout=Yes, Hid. #=300, Layer #=10, RNN Parameters (10^7)=2.3, Other Parameters (10^7)=24.2, Softmax Type=Adaptive softmax2018.04 | 50.06 | — | |
| RNN-1024 + MAXENT 9-GRAM FEATURESNumber of Params [Billions]=202016.02 | 51.3 | — | |
| RNN-1024 + MaxEnt 9 Gram FeaturesHardware=24 GPUs2016.12 | 51.3 | — | |
| RNN-1024 + 9 Gram#Param=20B2019.01 | 51.3 | — | |
| SPARSE NON-NEGATIVE MATRIX LMNumber of Params [Billions]=332016.02 | 52.9 | — | |
| Sparse Non-Negative Matrix LM2016.12 | 52.9 | — | |
| Sparse Non-Negative#Param=33B2019.01 | 52.9 | — | |
| LSTM-512-512Number of Params [Billions]=0.822016.02 | 54.1 | — | |
| GPT-2Size=Medium, Zero-shot=true, Unconditional=true2023.10 | 55.72 | — | |
| USMoEBackbone=Transformer-XL(420M), Topk=12025.03 | 56.9 | — | |
| TCBackbone=Transformer-XL(420M), Topk=12025.03 | 58.65 | — | |
| KN+SNMn-gram order=82014.12 | 61.4 | — | |
| KN+SNMn-gram order=72014.12 | 61.7 | — | |
| KNn-gram order=82014.12 | 62.9 | — | |
| KN+SNMn-gram order=62014.12 | 63 | — | |
| KNn-gram order=72014.12 | 63.2 | — | |
| DiffusionBertType=Diffusion2023.10 | 63.78 | — |