Language Modeling on OpenWebText 1M steps (val)
16.7Validation Perplexity (PPL)Autoregressive Transformer
Evaluation Results
| Method | Links | |
|---|---|---|
| Autoregressive Transformer2026.06 | 16.7 | |
| BlockGenLmax=16, gamma_i=0.2, Diffusion Type=Masked2026.06 | 17.5 | |
| BlockGenLmax=16, gamma_i=0.2, Diffusion Type=Uniform2026.06 | 18.5 | |
| BlockGenLmax=16, gamma1=0.05, gamma16=0.95, Diffusion Type=Masked2026.06 | 19.1 | |
| BlockGenLmax=16, gamma1=0.05, gamma16=0.95, Diffusion Type=Uniform2026.06 | 20.9 | |
| BDMBlock Size (L')=16, Diffusion Type=Masked, Objective=CE2026.06 | 21.6 | |
| BDMBlock Size (L')=16, Category=BDM (single-block)2026.06 | 22.3 | |
| MDLMDiffusion Type=Masked, Category=Sequence Diffusion2026.06 | 23.2 | |
| BDMBlock Size (L')=16, Diffusion Type=Uniform, Objective=ELBO2026.06 | 23.6 | |
| SEDD AbsorbDiffusion Type=Masked, Category=Sequence Diffusion2026.06 | 24.1 | |
| DuoDiffusion Type=Uniform, Category=Sequence Diffusion2026.06 | 25.2 | |
| UDLMDiffusion Type=Uniform, Category=Sequence Diffusion2026.06 | 27.4 | |
| SEDD UniformDiffusion Type=Uniform, Category=Sequence Diffusion2026.06 | 29.7 |