List your company
ResearchFormerly WizWand

Language Modeling on LM1B

22.8PerplexityAutoregressive Transformer
17.697252.141186.585121.0289Nov 1, 2024Feb 9, 2025May 21, 2025Aug 30, 2025Dec 8, 2025Mar 19, 2026Jun 28, 2026
Updated 4mo ago

Evaluation results

MethodLinks
Autoregressive TransformerSteps=1M2026.0522.8
CARDzero-shot=true, number of parameters=1B, training tokens=300B2026.0129.61
MDLM (low var.)Steps=1M, Retrained from scratch=true2026.0530.8
REPLAIDSteps=1M, s.c.=true2026.0531.6
MDLMSteps=1M2026.0531.8
SEDD AbsorbSteps=1M, Sentence packing=false2026.0532.7
REPLAIDSteps=1M, s.c.=false2026.0533.3
DuoSteps=1M2026.0533.7
Eso-LMs (α0 = 1)Steps=1M2026.0536.1
SemDLM+Sampling Steps=5122026.0636.6
UDLMSteps=1M2026.0536.7
PlaidSteps=1M, s.c.=true2026.0537.7
ARMzero-shot=true, number of parameters=1B, training tokens=300B2026.0139.14
PlaidSteps=1M, s.c.=false2026.0539.4
BD3LMzero-shot=true, number of parameters=1B, training tokens=300B2026.0140.04
SEDD UniformSteps=1M, Sentence packing=false2026.0540.3
MDLMzero-shot=true, number of parameters=1B, training tokens=300B2026.0148.68
ARZero-shot=true2026.0651.14
Autoregressive TransformerModel Category=Autoregressive, Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0451.25
SemDLM+Sampling Steps=2562026.0654.08
DID-FSize=Medium, Zero-shot=true, Alignment=FLOPs-aligned2026.0358.05
ABDZero-shot=true, Block Size (BS)=12026.0658.34
GPT2Sequence Length (L)=10242024.1158.92
DID-SSize=Medium, Zero-shot=true, Alignment=Steps-aligned2026.0359.88
RADDSize=Medium, Zero-shot=true2026.0360.32
BD3LMZero-shot=true, Block Size (BS)=42026.0660.88
ABDZero-shot=true, Block Size (BS)=42026.0662.07
DiffusionBertSteps=1M2026.0563.8
MDLMZero-shot=true2026.0664.94
SEDD (Masking)Sampling Steps=2562026.0665.63
MDLMModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0467.01
SEDDZero-shot=true2026.0668.14
SEDD AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0468.2
LangFlowModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0468.21
SEDD (Masking)Sampling Steps=5122026.0670.74
DID-FSize=Small, Zero-shot=true, Alignment=FLOPs-aligned2026.0372.04
RADDSize=Small, Zero-shot=true2026.0372.99
DuoModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0473.86
DID-SSize=Small, Zero-shot=true, Alignment=Steps-aligned2026.0374.04
SEDD (Masking)Sampling Steps=1282026.0674.48
SemDLM+Sampling Steps=1282026.0674.52
ReMDM (MDLM+remask)Sampling Steps=5122026.0675.98
D3PM AbsorbSteps=1M2026.0576.9
UDLMModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0477.59
DFM-NSequence Length (L)=10242024.1177.87
MDLM (no remasking)Sampling Steps=5122026.0678.02
SEDDSequence Length (L)=10242024.1180.79
SEDD UniformModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1M2026.0482.62
GIDD (a = 1, b = −2)Sampling Steps=1282026.0685.09
ReMDM (MDLM+remask)Sampling Steps=2562026.0685.21
MDLM (no remasking)Sampling Steps=2562026.0685.24
GIDD (a = 1, b = −2)Sampling Steps=5122026.0689.76
PlaidModel Category=Diffusion (Uniform-state / Continuous), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.0491.12
ReMDM (MDLM+remask)Sampling Steps=1282026.0691.19
GIDD (a = 1, b = −2)Sampling Steps=2562026.0692.44
MDLM (no remasking)Sampling Steps=1282026.0693.69
SEDD (uniform)Sampling Steps=2562026.06104.51
SEDD (uniform)Sampling Steps=5122026.06106.06
SEDD (uniform)Sampling Steps=1282026.06117.33
Diffusion-LMSteps=1M2026.05118.6
D3PM (Uniform)Sampling Steps=5122026.06133.92
D3PM UniformSteps=1M2026.05137.9
D3PM AbsorbModel Category=Diffusion (Absorbing-state), Zero-shot=true, Training Data=OWT, Training Steps=1.3M2026.04138.92
BERT-MouthSteps=1M2026.05142.9
D3PM (Uniform)Sampling Steps=2562026.06150.37