Language Modeling on LM1B
22.8PerplexityAutoregressive Transformer
Evaluation Results
| Method | Links | |
|---|---|---|
| Autoregressive TransformerSteps=1M2026.05 | 22.8 | |
| CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 29.61 | |
| MDLM (low var.)Steps=1M, Retrained from scratch=true2026.05 | 30.8 | |
| REPLAIDSteps=1M, s.c.=true2026.05 | 31.6 | |
| MDLMSteps=1M2026.05 | 31.8 | |
| SEDD AbsorbSteps=1M, Sentence packing=false2026.05 | 32.7 | |
| REPLAIDSteps=1M, s.c.=false2026.05 | 33.3 | |
| DuoSteps=1M2026.05 | 33.7 | |
| Eso-LMs (α0 = 1)Steps=1M2026.05 | 36.1 | |
| SemDLM+Sampling Steps=5122026.06 | 36.6 | |
| UDLMSteps=1M2026.05 | 36.7 | |
| PlaidSteps=1M, s.c.=true2026.05 | 37.7 | |
| ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 39.14 | |
| PlaidSteps=1M, s.c.=false2026.05 | 39.4 | |
| BD3LMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 40.04 | |
| SEDD UniformSteps=1M, Sentence packing=false2026.05 | 40.3 | |
| MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.01 | 48.68 | |
| ARZero-shot=true2026.06 | 51.14 | |
| Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 51.25 | |
| SemDLM+Sampling Steps=2562026.06 | 54.08 | |
| DID-FSize=Medium, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 58.05 | |
| ABDZero-shot=true, Block Size (BS)=12026.06 | 58.34 | |
| GPT2Sequence Length (L)=10242024.11 | 58.92 | |
| DID-SSize=Medium, Zero-shot=true, Alignment=Steps-aligned2026.03 | 59.88 | |
| RADDSize=Medium, Zero-shot=true2026.03 | 60.32 | |
| BD3LMZero-shot=true, Block Size (BS)=42026.06 | 60.88 | |
| ABDZero-shot=true, Block Size (BS)=42026.06 | 62.07 | |
| DiffusionBertSteps=1M2026.05 | 63.8 | |
| MDLMZero-shot=true2026.06 | 64.94 | |
| SEDD (Masking)Sampling Steps=2562026.06 | 65.63 | |
| MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 67.01 | |
| SEDDZero-shot=true2026.06 | 68.14 | |
| SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 68.2 | |
| LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 68.21 | |
| SEDD (Masking)Sampling Steps=5122026.06 | 70.74 | |
| DID-FSize=Small, Zero-shot=true, Alignment=FLOPs-aligned2026.03 | 72.04 | |
| RADDSize=Small, Zero-shot=true2026.03 | 72.99 | |
| DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 73.86 | |
| DID-SSize=Small, Zero-shot=true, Alignment=Steps-aligned2026.03 | 74.04 | |
| SEDD (Masking)Sampling Steps=1282026.06 | 74.48 | |
| SemDLM+Sampling Steps=1282026.06 | 74.52 | |
| ReMDM (MDLM+remask)Sampling Steps=5122026.06 | 75.98 | |
| D3PM AbsorbSteps=1M2026.05 | 76.9 | |
| UDLMModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 77.59 | |
| DFM-NSequence Length (L)=10242024.11 | 77.87 | |
| MDLM (no remasking)Sampling Steps=5122026.06 | 78.02 | |
| SEDDSequence Length (L)=10242024.11 | 80.79 | |
| SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.04 | 82.62 | |
| GIDD (a = 1, b = −2)Sampling Steps=1282026.06 | 85.09 | |
| ReMDM (MDLM+remask)Sampling Steps=2562026.06 | 85.21 | |
| MDLM (no remasking)Sampling Steps=2562026.06 | 85.24 | |
| GIDD (a = 1, b = −2)Sampling Steps=5122026.06 | 89.76 | |
| PlaidModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04 | 91.12 | |
| ReMDM (MDLM+remask)Sampling Steps=1282026.06 | 91.19 | |
| GIDD (a = 1, b = −2)Sampling Steps=2562026.06 | 92.44 | |
| MDLM (no remasking)Sampling Steps=1282026.06 | 93.69 | |
| SEDD (uniform)Sampling Steps=2562026.06 | 104.51 | |
| SEDD (uniform)Sampling Steps=5122026.06 | 106.06 | |
| SEDD (uniform)Sampling Steps=1282026.06 | 117.33 | |
| Diffusion-LMSteps=1M2026.05 | 118.6 | |
| D3PM (Uniform)Sampling Steps=5122026.06 | 133.92 | |
| D3PM UniformSteps=1M2026.05 | 137.9 | |
| D3PM AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04 | 138.92 | |
| BERT-MouthSteps=1M2026.05 | 142.9 | |
| D3PM (Uniform)Sampling Steps=2562026.06 | 150.37 |