Language Modeling on OpenWebText GPT-2 (val)
16.11PerplexityLlama110M (retrain.)
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Llama110M (retrain.)Train. toks.=262B, # params.=110M2025.10 | 16.11 | — | |
| Masked DiffusionLoss=ELBO, Param.=Denoiser2026.05 | 22.9 | — | |
| Absorbing Uniform DiffusionLoss=ELBO, Param.=Denoiser2026.05 | 23.32 | — | |
| GPT2Train. toks.=unk., # params.=117M2025.10 | 23.4 | — | |
| SEDDTrain. toks.=262B, # params.=92.1M2025.10 | 24.1 | — | |
| CCDD-MoEDiT w/RoBERTaTrain. toks.=131B, # params.=104.0M2025.10 | 24.56 | — | |
| Uniform DiffusionLoss=Cross-Entropy, Param.=LOO-Denoiser2026.05 | 25.18 | — | |
| Uniform DiffusionLoss=ELBO, Param.=Denoiser2026.05 | 25.2 | — | |
| Uniform DiffusionLoss=ELBO, Param.=LOO-Denoiser2026.05 | 25.3 | — | |
| Uniform DiffusionLoss=Cross-Entropy, Param.=Denoiser2026.05 | 25.45 | — | |
| GIDD+ (reimpl.)Train. toks.=131B, # params.=92.1M2025.10 | 25.82 | — | |
| MDLM (reimpl.)Train. toks.=131B, # params.=92.1M2025.10 | 27.39 | — | |
| StoSignSGDModel=GPT-2, Precision=FP82026.04 | 30.51 | 3.418 | |
| LionModel=GPT-2, Precision=FP82026.04 | 31.82 | 3.46 | |
| Max Coupling Uniform DiffusionLoss=ELBO, Param.=LOO-Denoiser2026.05 | 32.21 | — | |
| Max Coupling Uniform DiffusionLoss=ELBO, Param.=Denoiser2026.05 | 32.96 | — | |
| SignSGDModel=GPT-2, Precision=FP82026.04 | 34.67 | 3.546 |