Language Modeling on WikiText-103 zero-shot (test)
12.76PPLMegatron-LM GPT-2
Evaluation Results
| Method | Links | |
|---|---|---|
| Megatron-LM GPT-2Model Size=2.5B, Data=same data, Zero-shot=true2021.08 | 12.76 | |
| SLWModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=58.8K, Training tokens=157B, Training time=155Hr, Zero-shot=true, SLW Version=45K2021.08 | 13.72 | |
| SLWModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=428K, Training tokens=157B, Training time=364Hr, Zero-shot=true, SLW Version=270K2021.08 | 13.88 | |
| SLWModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=50K, Training tokens=121B, Training time=121Hr, Zero-shot=true, SLW Version=45K2021.08 | 13.88 | |
| BaselineModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=300K, Training tokens=157B, Training time=341Hr, Zero-shot=true2021.08 | 13.89 | |
| SLWModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=360K, Training tokens=122B, Training time=286Hr, Zero-shot=true, SLW Version=270K2021.08 | 13.89 | |
| ShortformerModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=55K, Training tokens=157B, Training time=162Hr, Zero-shot=true2021.08 | 14.14 | |
| Bsz WarmupModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=58.8K, Training tokens=157B, Training time=165Hr, Zero-shot=true2021.08 | 14.21 | |
| BaselineModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=37.5K, Training tokens=157B, Training time=151Hr, Zero-shot=true2021.08 | 14.76 | |
| Original GPT-2Model Size=1.5B, Data=different data, Zero-shot=true2021.08 | 17.48 | |
| Megatron-LM GPT-2Model Size=355M, Data=same data, Zero-shot=true2021.08 | 19.31 | |
| ARMDModel Size=Medium, Training Steps=300K, Evaluation Protocol=Zero-shot2026.01 | 25.55 | |
| SLWModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=205K, Training tokens=157B, Training time=31Hr, Zero-shot=true, SLW Version=110K2021.08 | 26.03 | |
| SLWModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=330K, Training tokens=157B, Training time=33Hr, Zero-shot=true, SLW Version=60K2021.08 | 27.01 | |
| SLWModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=122.5K, Training tokens=71B, Training time=15Hr, Zero-shot=true, SLW Version=110K2021.08 | 27.06 | |
| ARMDModel Size=Medium, Training Steps=120K, Evaluation Protocol=Zero-shot2026.01 | 27.09 | |
| SLWModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=52.5K, Training tokens=157B, Training time=16Hr, Zero-shot=true, SLW Version=30K2021.08 | 27.15 | |
| SLWModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=200K, Training tokens=89B, Training time=20Hr, Zero-shot=true, SLW Version=60K2021.08 | 27.74 | |
| SLWModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=37K, Training tokens=92B, Training time=10Hr, Zero-shot=true, SLW Version=30K2021.08 | 27.77 | |
| BaselineModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=300K, Training tokens=157B, Training time=37Hr, Zero-shot=true2021.08 | 27.78 | |
| BaselineModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=37.5K, Training tokens=157B, Training time=16Hr, Zero-shot=true2021.08 | 28.09 | |
| BaselineModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=150K, Training tokens=157B, Training time=32Hr, Zero-shot=true2021.08 | 28.19 | |
| RADDModel Size=Medium, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 29.29 | |
| SEDDModel Size=Medium, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 29.98 | |
| GPT-2Model Size=Medium, Evaluation Protocol=Zero-shot2026.01 | 31.39 | |
| ARMDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 35.15 | |
| ARMDModel Size=Small, Training Steps=180K, Evaluation Protocol=Zero-shot2026.01 | 35.66 | |
| Original GPT-2Model Size=117M, Data=different data, Zero-shot=true2021.08 | 37.5 | |
| RADDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 37.98 | |
| SEDDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 40.62 | |
| GPT-2Model Size=Small, Evaluation Protocol=Zero-shot2026.01 | 41.6 | |
| SEDD-UniformModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.01 | 49.6 | |
| PLAIDModel Size=Small, Training Steps=600K, Evaluation Protocol=Zero-shot2026.01 | 50.86 | |
| D3PMModel Size=Small, Evaluation Protocol=Zero-shot2026.01 | 75.16 |