Language Modeling on OpenWebText (OWT) (val)
7.77PerplexityMDM-Prime-v2
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| MDM-Prime-v2N=286M, D=168B, ℓ=16, compute-optimal=true2026.03 | 7.77 | — | — | — | — | — | |
| MDM-Prime-v2N=92M, D=524B, ℓ=16, compute-optimal=false2026.03 | 8.47 | — | — | — | — | — | |
| ARMN=860M, D=56B, compute-optimal=true2026.03 | 12.99 | — | — | — | — | — | |
| MDM-PrimeN=286M, D=168B, ℓ=6, compute-optimal=true2026.03 | 13.41 | — | — | — | — | — | |
| MDM-PrimeN=92M, D=524B, ℓ=6, compute-optimal=false2026.03 | 15.36 | — | — | — | — | — | |
| MDM-PrimeN=92M, D=524B, ℓ=8, compute-optimal=false2026.03 | 15.48 | — | — | — | — | — | |
| MDM-PrimeN=92M, D=524B, ℓ=4, compute-optimal=false2026.03 | 15.62 | — | — | — | — | — | |
| Autoregressive TransformerDiffusion Type=Autoregressive2026.02 | 17.5 | — | — | — | — | — | |
| ARGradient updates=0.5M, Forward passes=0.5M, Training tokens=262B2025.10 | 17.54 | — | — | — | — | — | |
| ARMN=92M, D=524B, compute-optimal=false2026.03 | 17.54 | — | — | — | — | — | |
| AR TransformerTraining Tokens=524B, Category=Autoregressive2026.07 | 17.54 | — | — | — | — | 96.7 | |
| EDLM-coARN=92M, D=524B, compute-optimal=false2026.03 | 17.58 | — | — | — | — | — | |
| MDM-PrimeN=92M, D=524B, ℓ=2, compute-optimal=false2026.03 | 17.9 | — | — | — | — | — | |
| MDMN=375M, D=128B, compute-optimal=true2026.03 | 18.94 | — | — | — | — | — | |
| BD3-LMsN=92M, D=524B, compute-optimal=false2026.03 | 20.73 | — | — | — | — | — | |
| MDLM+SMregime=iso-update, Gradient updates=1M, Forward passes=2M, Training tokens=262B2025.10 | 21.47 | — | — | — | — | — | |
| EDLM-NCEN=92M, D=524B, compute-optimal=false2026.03 | 21.52 | — | — | — | — | — | |
| GenMD4N=92M, D=524B, compute-optimal=false2026.03 | 21.8 | — | — | — | — | — | |
| LDDM-MGradient updates=1M, Forward passes=2M, Training tokens=262B2025.10 | 21.9 | — | — | — | — | — | |
| GIDD+Gradient updates=1M, Forward passes=1M, Training tokens=262B2025.10 | 22.29 | — | — | — | — | — | |
| MDLM+SMregime=iso-compute, Gradient updates=0.5M, Forward passes=1M, Training tokens=131B2025.10 | 22.36 | — | — | — | — | — | |
| SW-SetDLMTraining Tokens=157B, S (Interpolation Parameter)=≤ 8, Category=AR+Diffusion Hybrid2026.07 | 22.42 | — | — | — | — | 97.9 | |
| MDMN=92M, D=524B, compute-optimal=false2026.03 | 22.98 | — | — | — | — | — | |
| MDLMTraining Tokens=524B, Category=Diffusion2026.07 | 22.98 | — | — | — | — | 7.5 | |
| MDLMDiffusion Type=Masked Diffusion2026.02 | 23.2 | — | — | — | — | — | |
| MDLMGradient updates=1M, Forward passes=1M, Training tokens=262B2025.10 | 23.21 | — | — | — | — | — | |
| MDLM†Total tokens seen=524B*2026.02 | 23.21 | — | — | — | — | — | |
| BD3LMTraining Tokens=524B, S (Interpolation Parameter)=16, Category=AR+Diffusion Hybrid2026.07 | 23.34 | — | — | — | — | 80.2 | |
| SEDD AbsorbDiffusion Type=Masked Diffusion2026.02 | 24.1 | — | — | — | — | — | |
| SEDDGradient updates=1M, Forward passes=1M, Training tokens=262B2025.10 | 24.1 | — | — | — | — | — | |
| SEDDN=92M, D=524B, compute-optimal=false2026.03 | 24.1 | — | — | — | — | — | |
| SEDDTraining Tokens=524B, Category=Diffusion2026.07 | 24.1 | — | — | — | — | — | |
| MDLM2026.02 | 24.72 | — | — | — | — | — | |
| DuoDiffusion Type=Uniform-state Diffusion2026.02 | 25.2 | — | — | — | — | — | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=22026.02 | 25.2 | — | — | — | — | — | |
| DuoN=92M, D=524B, compute-optimal=false2026.03 | 25.2 | — | — | — | — | — | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=32026.02 | 25.3 | — | — | — | — | — | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=52026.02 | 25.4 | — | — | — | — | — | |
| UDLMDiffusion Type=Uniform-state Diffusion2026.02 | 27.4 | — | — | — | — | — | |
| SCDDpu=0.12026.02 | 28.41 | — | — | — | — | — | |
| SEDD UniformDiffusion Type=Uniform-state Diffusion2026.02 | 29.7 | — | — | — | — | — | |
| GIDD+pu=0.12026.02 | 31.54 | — | — | — | — | — | |
| GIDD+pu=0.22026.02 | 32.19 | — | — | — | — | — | |
| SCDDpu=0.22026.02 | 32.49 | — | — | — | — | — | |
| GPTlr=7e-4, min_lr=7e-5, n_layer=12, n_head=12, n_embed=768, n_params=124M2025.12 | 75 | 97.8 | 30.6 | 61.9 | 96.5 | — | |
| RGPT-Plr=6e-4, min_lr=4e-4, n_layer=12, n_head=12, n_embed=768, n_params=85M2025.12 | 99 | 97.6 | 33.6 | 61.5 | 97.5 | — | |
| NRGPTlr=1e-4, min_lr=7e-5, n_layer=12, n_head=12, n_embed=768, n_params=90M2025.12 | 104 | 96.6 | 30.6 | 67.4 | 98.4 | — |