List your company
ResearchFormerly WizWand

Language Modeling on Text8 (test)

1.23BPCTransformer AR
1.2041.37951.5551.7305Oct 28, 2024Jan 29, 2025May 3, 2025Aug 5, 2025Nov 6, 2025Feb 8, 2026May 13, 2026
Updated 5mo ago

Evaluation results

MethodLinks
Transformer ARModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.23
AR Discrete FlowModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.23
Transformer ARCategory=Autoregressive2026.021.23
AR Discrete FlowCategory=Autoregressive2026.021.23
Transformer ARModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.051.23
AR Discrete FlowModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.051.23
EDLMModel Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.24
MD4Model Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.37
MD4Category=Discrete Diffusion, Training steps=1,000,0002026.021.37
MD4Model Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.37
AR Argmax FlowModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.39
AR Argmax FlowCategory=Autoregressive2026.021.39
AR Argmax FlowModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.051.39
MDLMModel Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.4
MDLMCategory=Discrete Diffusion, Training steps=1,000,0002026.021.4
MACCategory=Any-order Autoregressive2026.021.4
MDLMModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.4
MACModel Category=Any-order Autoregressive, Likelihood Estimator Type=exact NLL2026.051.4
SEDD AbsorbCategory=Discrete Diffusion, Training steps=1,000,0002026.021.41
SEDD AbsorbModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.41
ARDMCategory=Any-order Autoregressive2026.021.43
ARDMModel Category=Any-order Autoregressive, Likelihood Estimator Type=exact NLL2026.051.43
DSLCategory=Continuous Diffusion, Training steps=1,000,0002026.021.45
D3PM AbsorbCategory=Discrete Diffusion, Training steps=1,000,0002026.021.45
DSLModel Category=Continuous Diffusion, Likelihood Estimator Type=ELBO upper bound2026.051.45
D3PM AbsorbModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.45
PlaidCategory=Continuous Diffusion, Implementation=MD4, Training steps=1,000,0002026.021.48
Plaid (MD4 impl.)Model Category=Continuous Diffusion, Likelihood Estimator Type=ELBO upper bound2026.051.48
D3PM UniformCategory=Discrete Diffusion, Training steps=1,000,0002026.021.61
D3PM UniformModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.61
Mult. DiffusionCategory=Discrete Diffusion, Training steps=1,000,0002026.021.72
Mult. DiffusionModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.051.72
IAF/SCFModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.101.88
IAF/SCFCategory=Autoregressive2026.021.88
IAF/SCFModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.051.88