Language Modeling on LM1B zero-shot
51.25PerplexityAR
Evaluation Results
| Method | Links | |
|---|---|---|
| ARZero-shot=true, Training Tokens=524B, Training Dataset=OWT, Status=Retrained2024.06 | 51.25 | |
| Autoregressive TransformerZero-shot=true2026.02 | 51.25 | |
| AR†2026.06 | 51.25 | |
| Uni-EDLM-NCE2026.06 | 58.14 | |
| EDLM-CoAR†2026.06 | 60.23 | |
| Uni-EDLM-AR2026.06 | 60.51 | |
| Block Diffusion*2026.06 | 61.39 | |
| EDLM-NCE†2026.06 | 63.19 | |
| DCD*2026.06 | 66.51 | |
| MDLMZero-shot=true, Training Tokens=524B, Training Dataset=OWT2024.06 | 67.01 | |
| MDLM†2026.06 | 67.01 | |
| GaussianTraining steps=1M, Training dataset=OpenWebText, Evaluation protocol=zero-shot2026.05 | 67.71 | |
| SEDDZero-shot=true, Training Tokens=524B, Training Dataset=OWT, Status=Retrained2024.06 | 68.2 | |
| SEDD†2026.06 | 68.2 | |
| Absorbing Uniform DiffusionLoss=ELBO, Param.=Denoiser, Zero-shot transfer=true2026.05 | 72.02 | |
| BaseTraining steps=1M, Training dataset=OpenWebText, Evaluation protocol=zero-shot2026.05 | 72.06 | |
| Masked DiffusionLoss=ELBO, Param.=Denoiser, Zero-shot transfer=true2026.05 | 73.06 | |
| Duo++Zero-shot=true, Number of Parameters=138M, Model Type=Diffusion, k=22026.02 | 73.8 | |
| DuoZero-shot=true, Number of Parameters=138M, Model Type=Diffusion2026.02 | 73.86 | |
| Duo‡2026.06 | 73.86 | |
| Duo++Zero-shot=true, Number of Parameters=138M, Model Type=Diffusion, k=32026.02 | 74.16 | |
| Duo++Zero-shot=true, Number of Parameters=138M, Model Type=Diffusion, k=52026.02 | 74.91 | |
| UDLMZero-shot=true, Number of Parameters=138M, Model Type=Diffusion2026.02 | 77.59 | |
| Uniform DiffusionLoss=ELBO, Param.=Denoiser, Zero-shot transfer=true2026.05 | 80.33 | |
| Uniform DiffusionLoss=Cross-Entropy, Param.=LOO-Denoiser, Zero-shot transfer=true2026.05 | 81.54 | |
| Uniform DiffusionLoss=ELBO, Param.=LOO-Denoiser, Zero-shot transfer=true2026.05 | 82.03 | |
| SEDD UniformZero-shot=true, Number of Parameters=138M, Model Type=Diffusion2026.02 | 82.62 | |
| Uniform DiffusionLoss=Cross-Entropy, Param.=Denoiser, Zero-shot transfer=true2026.05 | 83.43 | |
| Max Coupling Uniform DiffusionLoss=ELBO, Param.=LOO-Denoiser, Zero-shot transfer=true2026.05 | 112.7 | |
| Max Coupling Uniform DiffusionLoss=ELBO, Param.=Denoiser, Zero-shot transfer=true2026.05 | 117.21 |