Language modelling on LM1B (test)
20.86PerplexityTRANSFORMER
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TRANSFORMERParameters=110M, Training tokens=327B, Category=AR2024.05 | 20.86 | — | |
| TransformerModel Category=Autoregressive (Retrained), Parameters=110M, Training Tokens=327B2024.06 | 20.86 | — | |
| OmniNetModel Category=Autoregressive, Parameters=100M2024.06 | 21.5 | — | |
| Autoregressive TransformerDiffusion Type=Autoregressive2026.02 | 22.3 | — | |
| TransformerArchitecture=Autoregressive2026.04 | 22.3 | — | |
| TRANSFORMERParameters=110M, Training tokens=33B, Category=AR2024.05 | 22.32 | — | |
| TransformerModel Category=Autoregressive (Retrained), Parameters=110M, Training Tokens=33B2024.06 | 22.32 | — | |
| ARMDModel type=Diffusion, Training steps=1M2026.01 | 22.36 | — | |
| FISHER-FLOWParameters=110M, Training tokens=327B, Category=DM/FM2024.05 | 22.42 | — | |
| TransformerModel type=Autoregressive2026.01 | 22.83 | — | |
| TransformerModel Type=Autoregressive, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 22.83 | 120.1 | |
| ARInference Block Size=12026.06 | 22.88 | — | |
| MDLMParameters=110M, Training tokens=327B, Category=DM/FM2024.05 | 23 | — | |
| MDLMModel Category=Diffusion (Ours), Parameters=110M, Training Tokens=327B2024.06 | 23 | — | |
| Transformer-X BaseModel Category=Autoregressive, Parameters=0.46B2024.06 | 23.5 | — | |
| Transformer-XBaseModel type=Autoregressive2026.01 | 23.5 | — | |
| Transformer-X BaseModel Type=Autoregressive, Number of Parameters=0.5B, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 23.5 | — | |
| ARMDModel type=Diffusion, Training steps=300K2026.01 | 23.64 | — | |
| ABDInference Block Size=1, Training Tokens=65B2026.06 | 24.76 | — | |
| ABDInference Block Size=2, Training Tokens=65B2026.06 | 25.86 | — | |
| FISHER-FLOWParameters=110M, Training tokens=33B, Category=DM/FM2024.05 | 26.51 | — | |
| ScatterModel scale=350M, L=162026.04 | 26.58 | — | |
| MDLMDiffusion Type=Masked Diffusion2026.02 | 27 | — | |
| MDLMArchitecture=Diffusion (Absorbing State / Gaussian)2026.04 | 27 | — | |
| MDLMParameters=110M, Training tokens=33B, Category=DM/FM2024.05 | 27.04 | — | |
| MDLMModel Category=Diffusion (Ours), Parameters=110M, Training Tokens=33B2024.06 | 27.04 | — | |
| ScatterModel scale=110M, L=162026.04 | 27.1 | — | |
| SemDLM+2026.06 | 27.19 | — | |
| BD3LMInference Block Size=2, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 27.34 | — | |
| ABDInference Block Size=4, Training Tokens=65B2026.06 | 27.5 | — | |
| SW-SetDLMModel Type=AR+Diffusion Hybrid, S (segment size)=≤ 8, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 27.83 | 98.5 | |
| BD3LMInference Block Size=1, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 28.13 | — | |
| BD3LMInference Block Size=4, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 28.15 | — | |
| BD3-LMsModel type=Diffusion, L_prime=4, Training steps=1M2026.01 | 28.23 | — | |
| BD3-LMsModel scale=110M, L=42026.04 | 28.23 | — | |
| BD3LMModel Type=AR+Diffusion Hybrid, S (segment size)=4, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 28.23 | 76.9 | |
| BD3LMModel Type=AR+Diffusion Hybrid, S (segment size)=8, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 28.95 | 75 | |
| ABDInference Block Size=8, Training Tokens=65B2026.06 | 29.3 | — | |
| BD3-LMsModel scale=110M, L=162026.04 | 29.83 | — | |
| DuoDiffusion Type=Uniform-state Diffusion2026.02 | 29.9 | — | |
| DuoArchitecture=Diffusion (Absorbing State / Gaussian)2026.04 | 29.9 | — | |
| SW-SetDLMModel Type=AR+Diffusion Hybrid, S (segment size)=≤ 16, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 29.9 | 96.7 | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=22026.02 | 30 | — | |
| ScatterModel scale=350M, L=82026.04 | 30.02 | — | |
| BD3LMInference Block Size=8, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 30.06 | — | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=32026.02 | 30.1 | — | |
| Duo++Diffusion Type=Uniform-state Diffusion, k=52026.02 | 30.2 | — | |
| BD3LMInference Block Size=8, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 30.25 | — | |
| BD3LMInference Block Size=4, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 30.33 | — | |
| ScatterModel scale=110M, L=82026.04 | 30.5 | — | |
| BD3LMInference Block Size=16, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 30.56 | — | |
| BD3-LMsModel type=Diffusion, L_prime=16, Training steps=1M2026.01 | 30.6 | — | |
| BD3-LMsModel scale=110M, L=82026.04 | 30.6 | — | |
| BD3LMModel Type=AR+Diffusion Hybrid, S (segment size)=16, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 30.6 | 70.4 | |
| ABDInference Block Size=16, Training Tokens=65B2026.06 | 31.04 | — | |
| BD3LMInference Block Size=2, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 31.09 | — | |
| BD3LMInference Block Size=32, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 31.16 | — | |
| UDLMDiffusion Type=Uniform-state Diffusion2026.02 | 31.3 | — | |
| ScatterModel scale=350M, L=42026.04 | 31.63 | — | |
| MDLMModel type=Diffusion, Training steps=1M2026.01 | 31.78 | — | |
| Masking DLMkernel=Masking2026.06 | 31.78 | — | |
| BD3LMInference Block Size=64, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 31.78 | — | |
| MDLMModel Type=Diffusion, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 31.78 | 37.7 | |
| SW-SetDLMModel Type=AR+Diffusion Hybrid, S (segment size)=≤ 32, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 31.78 | 92.8 | |
| MDLMInference Block Size=1282026.06 | 31.92 | — | |
| ScatterModel scale=110M, L=42026.04 | 32.1 | — | |
| BD3LMInference Block Size=128, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 32.28 | — | |
| ABDInference Block Size=32, Training Tokens=65B2026.06 | 32.46 | — | |
| BD3LMInference Block Size=16, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 32.65 | — | |
| SEDDModel type=Diffusion, Training steps=1M2026.01 | 32.68 | — | |
| SEDD (Masking)kernel=Masking2026.06 | 32.68 | — | |
| SEDDInference Block Size=1282026.06 | 32.68 | — | |
| SEDDModel Type=Diffusion, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 32.68 | — | |
| SEDD AbsorbDiffusion Type=Masked Diffusion2026.02 | 32.7 | — | |
| SEDD AbsorbArchitecture=Diffusion (Absorbing State / Gaussian)2026.04 | 32.7 | — | |
| SEDDParameters=110M, Training tokens=33B, Category=Diffusion2024.05 | 32.79 | — | |
| SEDDModel Category=Diffusion, Parameters=110M, Training Tokens=33B2024.06 | 32.79 | — | |
| MDLMInference Block Size=642026.06 | 32.82 | — | |
| BD3LMInference Block Size=1, Training Block Size (BS)=16, Training Tokens=65B2026.06 | 33.09 | — | |
| ABDInference Block Size=64, Training Tokens=65B2026.06 | 33.31 | — | |
| DuoModel Type=Diffusion, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 33.68 | — | |
| ABDInference Block Size=128, Training Tokens=65B2026.06 | 33.73 | — | |
| GIDDkernel=mixture2026.06 | 34.34 | — | |
| MDLMInference Block Size=322026.06 | 34.49 | — | |
| BD3LMInference Block Size=32, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 35.2 | — | |
| UDLMModel Type=Diffusion, Number of Parameters=110M, Decoding Hardware=four RTX A6000 48GB GPUs2026.07 | 36.71 | — | |
| BD3LMInference Block Size=64, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 37.15 | — | |
| MDLMInference Block Size=162026.06 | 37.21 | — | |
| BD3LMInference Block Size=128, Training Block Size (BS)=4, Training Tokens=65B2026.06 | 38.64 | — | |
| SEDD UniformDiffusion Type=Uniform-state Diffusion2026.02 | 40.3 | — | |
| SEDD UniformArchitecture=Diffusion (Uniform)2026.04 | 40.3 | — | |
| SEDD (uniform)kernel=uniform2026.06 | 40.68 | — | |
| IDDMArchitecture=Diffusion (Absorbing State / Gaussian)2026.04 | 42.2 | — | |
| MDLMInference Block Size=82026.06 | 42.35 | — | |
| IDDMArchitecture=Diffusion (Uniform)2026.04 | 51.1 | — | |
| ARzero-shot=true2024.10 | 51.25 | — | |
| GPT-2Zero-shot=true, Trained on=OpenWebText, Training iterations=1M2025.05 | 51.25 | — | |
| MDLMInference Block Size=42026.06 | 52.48 | — | |
| EDLM-coARzero-shot=true2024.10 | 60.23 | — | |
| EDLM-ARzero-shot=true2024.10 | 60.8 | — |