Text Generation on OpenWebText
3.18PerplexitySDTT
Evaluation results
| Method | Links | ||
|---|---|---|---|
| SDTTNFE=2562025.10 | 3.18 | — | |
| SDTTNFE=1282025.10 | 3.3 | — | |
| SDTTNFE=642025.10 | 3.46 | — | |
| SDTTNFE=322025.10 | 3.7 | — | |
| SpeculativeNFE=2562025.10 | 5.02 | — | |
| 10nc-2c layersNFE=2562025.10 | 5.03 | — | |
| Masked DiffusionNFE=2562025.10 | 5.05 | — | |
| SpeculativeNFE=1282025.10 | 5.05 | — | |
| No output residualNFE=2562025.10 | 5.05 | — | |
| 10nc-2c layersNFE=1282025.10 | 5.06 | — | |
| No output residualNFE=1282025.10 | 5.1 | — | |
| SpeculativeNFE=642025.10 | 5.12 | — | |
| Masked DiffusionNFE=1282025.10 | 5.13 | — | |
| No output residualNFE=642025.10 | 5.16 | — | |
| 10nc-2c layersNFE=642025.10 | 5.16 | — | |
| Masked DiffusionNFE=642025.10 | 5.27 | — | |
| SpeculativeNFE=322025.10 | 5.28 | — | |
| 10nc-2c layersNFE=322025.10 | 5.34 | — | |
| No output residualNFE=322025.10 | 5.36 | — | |
| Masked DiffusionNFE=322025.10 | 5.5 | — | |
| AutoregressiveGeneration steps (T)=10242026.04 | 12.1 | 5.22 | |
| GaussianNFE=2048, Training steps=1M, Number of seeds=52026.05 | 33.89 | — | |
| BaseNFE=2048, Training steps=1M, Number of seeds=52026.05 | 36.04 | — | |
| GaussianNFE=1024, Training steps=1M, Number of seeds=52026.05 | 41.55 | — | |
| BaseNFE=1024, Training steps=1M, Number of seeds=52026.05 | 42.81 | — | |
| LaDiffN_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=1024, Temperature sampling (zeta)=0.72026.05 | 47.8 | 5.26 | |
| GaussianNFE=512, Training steps=1M, Number of seeds=52026.05 | 55.5 | — | |
| BaseNFE=512, Training steps=1M, Number of seeds=52026.05 | 56.51 | — | |
| LaDiffN_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=10242026.05 | 62.9 | 5.4 | |
| DUON_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=10242026.05 | 71.7 | 5.22 | |
| SDDM (ours)N_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=10242026.05 | 75.3 | 5.31 | |
| DuoGeneration steps (T)=10242026.04 | 77.7 | 5.55 | |
| DuoGeneration steps (T)=5122026.04 | 78.1 | 5.55 | |
| DuoGeneration steps (T)=2562026.04 | 78.6 | 5.55 | |
| MDLM (ours)N_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=10242026.05 | 90.5 | 5.5 | |
| IDDMGeneration steps (T)=2562026.04 | 98.5 | 5.51 | |
| SEDD UniformGeneration steps (T)=10242026.04 | 99.9 | 5.56 | |
| SEDD UniformGeneration steps (T)=5122026.04 | 100.4 | 5.56 | |
| IDDMGeneration steps (T)=10242026.04 | 100.9 | 5.6 | |
| VADD+TTD (Ours)Timesteps=1282026.07 | 101.7 | 6.76 | |
| IDDMGeneration steps (T)=5122026.04 | 103.2 | 5.64 | |
| SEDD UniformGeneration steps (T)=2562026.04 | 103.4 | 5.56 | |
| MDLMGeneration steps (T)=5122026.04 | 104.4 | 5.63 | |
| SEDD AbsorbGeneration steps (T)=5122026.04 | 104.5 | 5.62 | |
| MDLMN_cont (continuous diffusion steps)=200, N_disc (discrete decoding steps)=10242026.05 | 104.5 | 5.63 | |
| MDLMGeneration steps (T)=10242026.04 | 104.9 | 5.63 | |
| SEDD AbsorbGeneration steps (T)=10242026.04 | 105 | 5.62 | |
| VADD+CPD (Ours)Timesteps=1282026.07 | 108.57 | 6.84 | |
| SEDD AbsorbGeneration steps (T)=2562026.04 | 109.8 | 5.63 | |
| VADD+TTD (Ours)Timesteps=642026.07 | 109.8 | 6.78 | |
| MDLMGeneration steps (T)=2562026.04 | 112.7 | 5.66 | |
| VADD+CPD (Ours)Timesteps=642026.07 | 118.11 | 6.86 | |
| MDLM+TTD (Ours)Timesteps=1282026.07 | 119.2 | 7.54 | |
| MDLMTimesteps=1282026.07 | 122.08 | 7.56 | |
| VADD+TTD (Ours)Timesteps=322026.07 | 122.34 | 6.78 | |
| VADDTimesteps=1282026.07 | 123.09 | 6.97 | |
| MDLM+CPD (Ours)Timesteps=1282026.07 | 126.85 | 7.6 | |
| SEDD (mask)Train Toks=682B, Sampling Steps=1282026.04 | 127.2 | — | |
| Di4CTimesteps=1282026.07 | 127.31 | 7.57 | |
| EDLMTimesteps=1282026.07 | 129.38 | 7.6 | |
| VADDTimesteps=642026.07 | 132.09 | 6.98 | |
| DFM-MMF-EXACT-EMAGeneration Steps=10242024.11 | 132.55 | — | |
| VADD+CPD (Ours)Timesteps=322026.07 | 135.97 | 6.86 | |
| MDLM+TTD (Ours)Timesteps=642026.07 | 140.97 | 7.57 | |
| DFM-MMF-EXACTGeneration Steps=10242024.11 | 141.46 | — | |
| DFM-MMF-OTGeneration Steps=10242024.11 | 141.92 | — | |
| DFM-MMF-EXACT-EMAGeneration Steps=1282024.11 | 143.08 | — | |
| DFM-MMFGeneration Steps=10242024.11 | 143.48 | — | |
| MDLMTimesteps=642026.07 | 146.09 | 7.58 | |
| DFM-OGeneration Steps=10242024.11 | 146.54 | — | |
| DFM-NGeneration Steps=10242024.11 | 147.07 | — | |
| MDLM+CPD (Ours)Timesteps=642026.07 | 148.1 | 7.6 | |
| VADDTimesteps=322026.07 | 151.84 | 6.98 | |
| Di4CTimesteps=642026.07 | 152.63 | 7.6 | |
| DFM-MMF-OTGeneration Steps=1282024.11 | 153.51 | — | |
| DFM-MMF-EXACTGeneration Steps=1282024.11 | 153.51 | — | |
| EDLMTimesteps=642026.07 | 153.79 | 7.62 | |
| VADD+TTD (Ours)Timesteps=162026.07 | 153.85 | 6.8 | |
| DFM-MMFGeneration Steps=1282024.11 | 155.45 | — | |
| DFM-MMF-EXACT-EMAGeneration Steps=642024.11 | 156.62 | — | |
| DFM-S (MD4 LOSS)Generation Steps=10242024.11 | 156.81 | — | |
| DFM-OGeneration Steps=1282024.11 | 159.03 | — | |
| DFM-B-EXACT-EMAGeneration Steps=10242024.11 | 159.87 | — | |
| DFM-NGeneration Steps=1282024.11 | 160.17 | — | |
| DFM-B-EXACT-EMAGeneration Steps=1282024.11 | 164.42 | — | |
| DFM-MMF-OTGeneration Steps=642024.11 | 167.86 | — | |
| DFM-MMF-EXACTGeneration Steps=642024.11 | 168.12 | — | |
| DFM-S (MD4 LOSS)Generation Steps=1282024.11 | 169.81 | — | |
| DFM-B-EXACT-EMAGeneration Steps=642024.11 | 169.89 | — | |
| DFM-MMFGeneration Steps=642024.11 | 170.61 | — | |
| DFM-OGeneration Steps=642024.11 | 175.59 | — | |
| DFM-NGeneration Steps=642024.11 | 176.34 | — | |
| DCDTimesteps=1282026.07 | 176.74 | 8.89 | |
| DFM-B-OTGeneration Steps=10242024.11 | 178.53 | — | |
| VADD+CPD (Ours)Timesteps=162026.07 | 180.72 | 6.9 | |
| DFM-B-EXACTGeneration Steps=10242024.11 | 180.91 | — | |
| DFM-B-EXACT-EMAGeneration Steps=322024.11 | 182.4 | — | |
| MDLM+TTD (Ours)Timesteps=322026.07 | 183.43 | 7.6 | |
| Neural CTMC – EulerTrain Toks=262B, Sampling Steps=1282026.04 | 183.6 | — | |
| Neural CTMC – τ-leapingTrain Toks=262B, Sampling Steps=1282026.04 | 184.8 | — |