Language Modeling on OpenWebText (OWT) (Gen. PPL, Ent.)
15.06Gen. PPLReal data
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Real data2026.05 | 15.06 | 5.44 | |
| Bitstream DiffusionNFE=256, Sampling Strategy=Stochastic2026.05 | 27.06 | 5.26 | |
| AR Transformer2026.05 | 35.9 | 5.58 | |
| LangFlowNFE=10242026.05 | 36.53 | 5.25 | |
| CoFReBudget=768, Train=≈123h + 30k, VRAM=93.44 GiB/GPU2026.05 | 37.846 | 5.142 | |
| Bitstream DiffusionNFE=256, Sampling Strategy=Deterministic2026.05 | 46.32 | 5.13 | |
| MDLM + SDTTBudget=768, Train=≈139h + SDTT, VRAM=112.4 GiB/GPU2026.05 | 47.04 | 5.45 | |
| Duo + DCDSteps=12026.02 | 47.13 | 2.8 | |
| CoFReBudget=384, Train=≈123h + 30k, VRAM=93.44 GiB/GPU2026.05 | 48.755 | 5.283 | |
| SCDDSampling Steps=1024, pu=0.2, Context Length=5122026.02 | 55.7 | — | |
| SCDDSampling Steps=512, pu=0.2, Context Length=5122026.02 | 58.2 | — | |
| SCDDSampling Steps=256, pu=0.2, Context Length=5122026.02 | 59.6 | — | |
| SCDDSampling Steps=128, pu=0.2, Context Length=5122026.02 | 60.7 | — | |
| SCDDSampling Steps=1024, pu=0.1, Context Length=5122026.02 | 61.3 | — | |
| FLMSampling steps=10242026.02 | 62.23 | 5.33 | |
| FLM2026.05 | 62.23 | 5.33 | |
| MDLM + SDTTBudget=384, Train=≈139h + SDTT, VRAM=112.4 GiB/GPU2026.05 | 62.29 | 5.49 | |
| SCDDSampling Steps=512, pu=0.1, Context Length=5122026.02 | 63.6 | — | |
| GIDD+Sampling Steps=1024, pu=0.1, Context Length=5122026.02 | 63.8 | — | |
| GIDD+Sampling Steps=512, pu=0.1, Context Length=5122026.02 | 64.8 | — | |
| GIDD+Sampling Steps=256, pu=0.1, Context Length=5122026.02 | 65 | — | |
| CoFReBudget=192, Train=≈123h + 30k, VRAM=93.44 GiB/GPU2026.05 | 65.182 | 5.38 | |
| SCDDSampling Steps=256, pu=0.1, Context Length=5122026.02 | 66 | — | |
| GIDD+Sampling Steps=128, pu=0.1, Context Length=5122026.02 | 66.7 | — | |
| SCDDSampling Steps=64, pu=0.2, Context Length=5122026.02 | 67.1 | — | |
| SCDDSampling Steps=128, pu=0.1, Context Length=5122026.02 | 67.6 | — | |
| ReMDM-CapSampling Steps=1024, cap=0.01, Context Length=5122026.02 | 68.3 | — | |
| GIDD+Sampling Steps=1024, pu=0.2, Context Length=5122026.02 | 71.2 | — | |
| GIDD+Sampling Steps=64, pu=0.1, Context Length=5122026.02 | 71.4 | — | |
| SCDDSampling Steps=64, pu=0.1, Context Length=5122026.02 | 71.8 | — | |
| GIDD+Sampling Steps=512, pu=0.2, Context Length=5122026.02 | 72 | — | |
| GIDD+Sampling Steps=256, pu=0.2, Context Length=5122026.02 | 73.2 | — | |
| ReMDM-CapSampling Steps=512, cap=0.01, Context Length=5122026.02 | 73.9 | — | |
| SCDDSampling Steps=32, pu=0.2, Context Length=5122026.02 | 74.5 | — | |
| GIDD+Sampling Steps=128, pu=0.2, Context Length=5122026.02 | 75.1 | — | |
| FMLMSteps=42026.02 | 76.37 | 5.05 | |
| DuoSampling steps=10242026.02 | 77.69 | 5.55 | |
| Duo2026.05 | 77.69 | 5.55 | |
| SCDDSampling Steps=32, pu=0.1, Context Length=5122026.02 | 78.6 | — | |
| GIDD+Sampling Steps=64, pu=0.2, Context Length=5122026.02 | 79 | — | |
| ReMDM-ConfidenceSampling Steps=1024, Context Length=5122026.02 | 80.5 | — | |
| ReMDM-CapSampling Steps=256, cap=0.01, Context Length=5122026.02 | 81.7 | — | |
| GIDD+Sampling Steps=32, pu=0.1, Context Length=5122026.02 | 82.1 | — | |
| ReMDM-ConfidenceSampling Steps=512, Context Length=5122026.02 | 83.9 | — | |
| ReMDM-ConfidenceSampling Steps=256, Context Length=5122026.02 | 87.9 | — | |
| MDLMSampling Steps=1024, Context Length=5122026.02 | 88.5 | — | |
| MDLM + SDTTBudget=192, Train=≈139h + SDTT, VRAM=112.4 GiB/GPU2026.05 | 89.17 | 5.53 | |
| GIDD+Sampling Steps=32, pu=0.2, Context Length=5122026.02 | 90.5 | — | |
| MDLMSampling Steps=512, Context Length=5122026.02 | 91.9 | — | |
| MDLMSampling Steps=256, Context Length=5122026.02 | 94.8 | — | |
| ReMDM-CapSampling Steps=128, cap=0.01, Context Length=5122026.02 | 95.9 | — | |
| Duo + DCDSteps=22026.02 | 96.59 | 3.77 | |
| Duo + Di4CSteps=12026.02 | 97.77 | 3.36 | |
| ReMDM-ConfidenceSampling Steps=128, Context Length=5122026.02 | 98.1 | — | |
| CoFReBudget=96, Train=≈123h + 30k, VRAM=93.44 GiB/GPU2026.05 | 101.791 | 5.434 | |
| MDLMSampling Steps=128, Context Length=5122026.02 | 104.7 | — | |
| MDLM2026.05 | 105.15 | 5.63 | |
| Duo + DCDSteps=42026.02 | 108.21 | 4.82 | |
| ReMDM-ConfidenceSampling Steps=64, Context Length=5122026.02 | 118.3 | — | |
| ReMDM-CapSampling Steps=64, cap=0.01, Context Length=5122026.02 | 120.9 | — | |
| MDLMSampling steps=10242026.02 | 121.09 | 5.65 | |
| MDLMSampling Steps=64, Context Length=5122026.02 | 123.6 | — | |
| FMLMSteps=12026.02 | 129.32 | 4.53 | |
| FMLMSteps=22026.02 | 134.26 | 5.07 | |
| CANDISampling steps=10242026.02 | 143.13 | 5.71 | |
| CANDI2026.05 | 143.13 | 5.71 | |
| Duo + Di4CSteps=42026.02 | 150.67 | 4.81 | |
| Duo + Di4CSteps=22026.02 | 165.81 | 4.65 | |
| ReMDM-CapSampling Steps=32, cap=0.01, Context Length=5122026.02 | 166.3 | — | |
| ReMDM-ConfidenceSampling Steps=32, Context Length=5122026.02 | 167.6 | — | |
| MDLMSampling Steps=32, Context Length=5122026.02 | 169.9 | — | |
| MDLM + SDTTBudget=96, Train=≈139h + SDTT, VRAM=112.4 GiB/GPU2026.05 | 193.05 | 5.58 | |
| MDLM + Di4CSteps=42026.02 | 239.27 | 5.4 | |
| MDLM + SDTTSteps=42026.02 | 339.73 | 5.38 | |
| MDLM + Di4CSteps=22026.02 | 758.23 | 5.35 | |
| MDLM + SDTTSteps=22026.02 | 877.22 | 5.34 | |
| MDLM + SDTTSteps=12026.02 | 1,260.86 | 5.26 | |
| MDLM + Di4CSteps=12026.02 | 1,298.8 | 5.29 |