Text Generation on OpenWebText (OWT)
5.33GPT2 PerplexityDFM (ESD)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| DFM (ESD)NFEs=12026.04 | 5.33 | — | 0.26 | |
| Datatop-p=0.852026.03 | 15.4 | 0 | 5.44 | |
| Masked D-MMDNFE=16, top-p=0.852026.03 | 17.2 | 23.6 | 5 | |
| Masked D-MMDNFE=32, top-p=0.852026.03 | 19.4 | 22.5 | 5.05 | |
| Masked D-MMDNFE=4, top-p=0.852026.03 | 20.3 | 82 | 4.6 | |
| Masked TeacherNFE=256, top-p=0.852026.03 | 22.5 | 27.5 | 5.13 | |
| Masked TeacherNFE=128, top-p=0.852026.03 | 23.9 | 29.5 | 5.17 | |
| Masked TeacherNFE=64, top-p=0.852026.03 | 26 | 30.7 | 5.19 | |
| SDTT (reimpl.)NFE=64, top-p=0.852026.03 | 26.9 | 29.3 | 5.17 | |
| SDTT (reimpl.)NFE=32, top-p=0.852026.03 | 30.4 | 34 | 5.18 | |
| Masked D-MMDNFE=4, top-p=1.02026.03 | 66.1 | 71.9 | 5.44 | |
| Masked D-MMDNFE=16, top-p=1.02026.03 | 67.7 | 55.8 | 5.57 | |
| Masked D-MMDNFE=32, top-p=1.02026.03 | 72.1 | 57.8 | 5.57 | |
| FMLMNFE=4, top-p=0.852026.03 | 76.4 | — | 5.05 | |
| DFM (ESD)NFEs=42026.04 | 77.08 | — | 5.27 | |
| Masked TeacherNFE=256, top-p=1.02026.03 | 85.9 | 67.2 | 5.59 | |
| Masked TeacherNFE=128, top-p=1.02026.03 | 91.1 | 71.1 | 5.61 | |
| Masked TeacherNFE=64, top-p=1.02026.03 | 101 | 78.1 | 5.63 | |
| Duo + DCDNFE=4, top-p=0.852026.03 | 108.2 | — | 4.82 | |
| DFM (ESD)NFEs=22026.04 | 108.91 | — | 5.15 | |
| FMLMNFEs=42026.04 | 111.31 | — | 5.26 | |
| DFM (PSD)NFEs=42026.04 | 122.32 | — | 5.1 | |
| FMLMNFEs=22026.04 | 133.29 | — | 5.25 | |
| Duo + Di4CNFE=4, top-p=0.852026.03 | 150.7 | — | 4.81 | |
| DFM (PSD)NFEs=22026.04 | 152.83 | — | 5.03 | |
| Duo + Di4CNFEs=42026.04 | 154.67 | — | 4.85 | |
| FMLMNFEs=12026.04 | 168.3 | — | 5.17 | |
| DFM (PSD)NFEs=12026.04 | 180.29 | — | 4.91 | |
| Duo + Di4CNFEs=22026.04 | 210.22 | — | 4.63 | |
| MDLM + Di4CNFEs=42026.04 | 239.27 | — | 5.4 | |
| MDLM + Di4CNFE=4, top-p=0.852026.03 | 239.3 | — | 5.4 | |
| Duo + DCDNFEs=42026.04 | 250.86 | — | 5.37 | |
| MDLM + SDTTNFE=4, top-p=0.852026.03 | 339.7 | — | 5.38 | |
| MDLM + SDTTNFEs=42026.04 | 339.73 | — | 5.38 | |
| Duo + Di4CNFEs=12026.04 | 370.51 | — | 3.92 | |
| MDLM + Di4CNFEs=22026.04 | 758.23 | — | 5.35 | |
| MDLM + SDTTNFEs=22026.04 | 877.22 | — | 5.34 | |
| Duo + DCDNFEs=22026.04 | 891.16 | — | 5.41 | |
| MDLM + SDTTNFEs=12026.04 | 1,260.86 | — | 5.26 | |
| MDLM + Di4CNFEs=12026.04 | 1,298.8 | — | 5.29 | |
| Duo + DCDNFEs=12026.04 | 5,743.29 | — | 6.02 |