Language Modeling on Text8 (test)
1.23BPCTransformer AR
Evaluation Results
| Method | Links | |
|---|---|---|
| Transformer ARModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.23 | |
| AR Discrete FlowModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.23 | |
| Transformer ARCategory=Autoregressive2026.02 | 1.23 | |
| AR Discrete FlowCategory=Autoregressive2026.02 | 1.23 | |
| Transformer ARModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.23 | |
| AR Discrete FlowModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.23 | |
| EDLMModel Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.24 | |
| MD4Model Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.37 | |
| MD4Category=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.37 | |
| MD4Model Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.37 | |
| AR Argmax FlowModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.39 | |
| AR Argmax FlowCategory=Autoregressive2026.02 | 1.39 | |
| AR Argmax FlowModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.39 | |
| MDLMModel Category=Discrete Diffusion, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.4 | |
| MDLMCategory=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.4 | |
| MACCategory=Any-order Autoregressive2026.02 | 1.4 | |
| MDLMModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.4 | |
| MACModel Category=Any-order Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.4 | |
| SEDD AbsorbCategory=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.41 | |
| SEDD AbsorbModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.41 | |
| ARDMCategory=Any-order Autoregressive2026.02 | 1.43 | |
| ARDMModel Category=Any-order Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.43 | |
| DSLCategory=Continuous Diffusion, Training steps=1,000,0002026.02 | 1.45 | |
| D3PM AbsorbCategory=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.45 | |
| DSLModel Category=Continuous Diffusion, Likelihood Estimator Type=ELBO upper bound2026.05 | 1.45 | |
| D3PM AbsorbModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.45 | |
| PlaidCategory=Continuous Diffusion, Implementation=MD4, Training steps=1,000,0002026.02 | 1.48 | |
| Plaid (MD4 impl.)Model Category=Continuous Diffusion, Likelihood Estimator Type=ELBO upper bound2026.05 | 1.48 | |
| D3PM UniformCategory=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.61 | |
| D3PM UniformModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.61 | |
| Mult. DiffusionCategory=Discrete Diffusion, Training steps=1,000,0002026.02 | 1.72 | |
| Mult. DiffusionModel Category=Discrete Diffusion, Likelihood Estimator Type=exact NLL2026.05 | 1.72 | |
| IAF/SCFModel Category=Autoregressive, Chunk Length=256, Backbone=12-layer transformer (GPT2-small scale)2024.10 | 1.88 | |
| IAF/SCFCategory=Autoregressive2026.02 | 1.88 | |
| IAF/SCFModel Category=Autoregressive, Likelihood Estimator Type=exact NLL2026.05 | 1.88 |