Language Modeling on OpenWebText (OWT) (test)
17.54PerplexityAR
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| ARTraining tokens=262B2026.04 | 17.54 | — | — | — | — | — | |
| ARInference Block Size=12026.06 | 17.54 | — | — | — | — | — | |
| ABDInference Block Size=1, Training Tokens=524B2026.06 | 19.57 | — | — | — | — | — | |
| BD3LMInference Block Size=4, Training Block Size (BS)=4, Training Tokens=524B2026.06 | 20.73 | — | — | — | — | — | |
| ABDInference Block Size=4, Training Tokens=524B2026.06 | 21.03 | — | — | — | — | — | |
| LDDM-MDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=false2025.10 | 21.9 | — | — | — | — | — | |
| BD3LMInference Block Size=16, Training Block Size (BS)=16, Training Tokens=524B2026.06 | 22.27 | — | — | — | — | — | |
| ABDInference Block Size=16, Training Tokens=524B2026.06 | 22.56 | — | — | — | — | — | |
| MDLMInference Block Size=10242026.06 | 22.98 | — | — | — | — | — | |
| MDLMDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 23.05 | — | — | — | — | — | |
| SCMDMConcatTraining tokens=262B, Fine-tuning tokens=3.25B2026.04 | 23.1 | — | — | — | — | — | |
| SCMDMAddTraining tokens=262B, Fine-tuning tokens=3.25B2026.04 | 23.11 | — | — | — | — | — | |
| MDLMTraining tokens=262B, Fine-tuning tokens=3.25B2026.04 | 23.73 | — | — | — | — | — | |
| LDDM-UDiffusion Type=Uniform Diffusion, Training Steps=1 million, Retrained status=false2025.10 | 23.82 | — | — | — | — | — | |
| SEDD AbsorbDiffusion Type=Masked Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 24.01 | — | — | — | — | — | |
| SEDDTraining tokens=262B2026.04 | 24.1 | — | — | — | — | — | |
| SEDDInference Block Size=10242026.06 | 24.1 | — | — | — | — | — | |
| ABDInference Block Size=1024, Training Tokens=524B2026.06 | 25.48 | — | — | — | — | — | |
| UDLMDiffusion Type=Uniform Diffusion, Training Steps=1 million, Retrained status=true2025.10 | 25.51 | — | — | — | — | — | |
| GPTlr=7e-4, min_lr=7e-5, n_layer=12, n_head=12, n_embed=768, n_params=124M2025.12 | 75 | 97.8 | 30.6 | 61.9 | 96.5 | — | |
| GPT_Rec_Parallellr=6e-4, min_lr=4e-4, n_layer=12, n_head=12, n_embed=768, n_params=85M2025.12 | 99 | 97.6 | 33.6 | 61.5 | 97.5 | — | |
| NRGPT_H_FF2Wlr=1e-4, min_lr=7e-5, n_layer=12, n_head=12, n_embed=768, n_params=90M2025.12 | 104 | 96.6 | 30.6 | 67.4 | 98.4 | — | |
| (mode collapse)Generation steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 100 | |
| Dataset (Real Data Reference)Generation steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 4.6 | |
| Duo + DCDGeneration steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 29.7 | |
| Duo + Di4CGeneration steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 27.2 | |
| FMLMGeneration steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 12.1 | |
| MDLM + Di4CGeneration steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 3.1 | |
| MDLM + SDTTGeneration steps=1, Number of generated samples=10242026.02 | — | — | — | — | — | 3.6 |