List your company
ResearchFormerly WizWand

Language Modeling on WikiText-103 zero-shot (test)

12.76PPLMegatron-LM GPT-2
10.26427.11243.9660.808Aug 13, 2021May 10, 2022Feb 5, 2023Nov 3, 2023Jul 30, 2024Apr 27, 2025Jan 23, 2026
Updated 8mo ago

Evaluation results

MethodLinks
Megatron-LM GPT-2Model Size=2.5B, Data=same data, Zero-shot=true2021.0812.76
SLWModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=58.8K, Training tokens=157B, Training time=155Hr, Zero-shot=true, SLW Version=45K2021.0813.72
SLWModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=428K, Training tokens=157B, Training time=364Hr, Zero-shot=true, SLW Version=270K2021.0813.88
SLWModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=50K, Training tokens=121B, Training time=121Hr, Zero-shot=true, SLW Version=45K2021.0813.88
BaselineModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=300K, Training tokens=157B, Training time=341Hr, Zero-shot=true2021.0813.89
SLWModel Size=1.5B, Pre-training parameters=bsz512-seqlen1K, Training steps=360K, Training tokens=122B, Training time=286Hr, Zero-shot=true, SLW Version=270K2021.0813.89
ShortformerModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=55K, Training tokens=157B, Training time=162Hr, Zero-shot=true2021.0814.14
Bsz WarmupModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=58.8K, Training tokens=157B, Training time=165Hr, Zero-shot=true2021.0814.21
BaselineModel Size=1.5B, Pre-training parameters=bsz4K-seqlen1K, Training steps=37.5K, Training tokens=157B, Training time=151Hr, Zero-shot=true2021.0814.76
Original GPT-2Model Size=1.5B, Data=different data, Zero-shot=true2021.0817.48
Megatron-LM GPT-2Model Size=355M, Data=same data, Zero-shot=true2021.0819.31
ARMDModel Size=Medium, Training Steps=300K, Evaluation Protocol=Zero-shot2026.0125.55
SLWModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=205K, Training tokens=157B, Training time=31Hr, Zero-shot=true, SLW Version=110K2021.0826.03
SLWModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=330K, Training tokens=157B, Training time=33Hr, Zero-shot=true, SLW Version=60K2021.0827.01
SLWModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=122.5K, Training tokens=71B, Training time=15Hr, Zero-shot=true, SLW Version=110K2021.0827.06
ARMDModel Size=Medium, Training Steps=120K, Evaluation Protocol=Zero-shot2026.0127.09
SLWModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=52.5K, Training tokens=157B, Training time=16Hr, Zero-shot=true, SLW Version=30K2021.0827.15
SLWModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=200K, Training tokens=89B, Training time=20Hr, Zero-shot=true, SLW Version=60K2021.0827.74
SLWModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=37K, Training tokens=92B, Training time=10Hr, Zero-shot=true, SLW Version=30K2021.0827.77
BaselineModel Size=117M, Pre-training parameters=bsz512-seqlen1K, Training steps=300K, Training tokens=157B, Training time=37Hr, Zero-shot=true2021.0827.78
BaselineModel Size=117M, Pre-training parameters=bsz4K-seqlen1K, Training steps=37.5K, Training tokens=157B, Training time=16Hr, Zero-shot=true2021.0828.09
BaselineModel Size=117M, Pre-training parameters=bsz512-seqlen2K, Training steps=150K, Training tokens=157B, Training time=32Hr, Zero-shot=true2021.0828.19
RADDModel Size=Medium, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0129.29
SEDDModel Size=Medium, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0129.98
GPT-2Model Size=Medium, Evaluation Protocol=Zero-shot2026.0131.39
ARMDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0135.15
ARMDModel Size=Small, Training Steps=180K, Evaluation Protocol=Zero-shot2026.0135.66
Original GPT-2Model Size=117M, Data=different data, Zero-shot=true2021.0837.5
RADDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0137.98
SEDDModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0140.62
GPT-2Model Size=Small, Evaluation Protocol=Zero-shot2026.0141.6
SEDD-UniformModel Size=Small, Training Steps=400K, Evaluation Protocol=Zero-shot2026.0149.6
PLAIDModel Size=Small, Training Steps=600K, Evaluation Protocol=Zero-shot2026.0150.86
D3PMModel Size=Small, Evaluation Protocol=Zero-shot2026.0175.16