Unconditional Generation on OpenWebText (OWT) L=1024 (held-out)
1MAUVEData (reference)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Data (reference)T=128, L=10242026.02 | 1 | 14.8 | 5.44 | |
| Data (reference)T=256, L=10242026.02 | 1 | 14.8 | 5.44 | |
| Data (reference)T=512, L=10242026.02 | 1 | 14.8 | 5.44 | |
| Data (reference)T=1024, L=10242026.02 | 1 | 14.8 | 5.44 | |
| AR (reference)T=1024, L=10242026.02 | 0.76 | 12.1 | 5.22 | |
| DSL-FT + ReMDM-loop (principled ηcap(T))T=1024, L=10242026.02 | 0.722 | 45.3 | 5.2 | |
| ADLMT=1024, L=10242026.02 | 0.699 | 25.4 | 5.35 | |
| DSL-FT + ReMDM-loop (principled ηcap(T))T=256, L=10242026.02 | 0.661 | 49 | 5.27 | |
| DSL-FT + ReMDM-loop (principled ηcap(T))T=512, L=10242026.02 | 0.651 | 41 | 5.18 | |
| DSL-FT + ReMDM-loop (principled ηcap(T))T=128, L=10242026.02 | 0.639 | 59.7 | 5.34 | |
| ADLMT=512, L=10242026.02 | 0.573 | 31.6 | 5.4 | |
| DSL-FT + ReMDM (official schedule)T=256, L=10242026.02 | 0.557 | 25 | 4.91 | |
| DSL-FT + ReMDM (official schedule)T=128, L=10242026.02 | 0.529 | 38.1 | 5.14 | |
| PRISM-loopT=1024, L=10242026.02 | 0.527 | 15.3 | 5.1 | |
| PRISM-loopT=512, L=10242026.02 | 0.423 | 16.4 | 5.12 | |
| DSL-FT + ReMDM (official schedule)T=1024, L=10242026.02 | 0.413 | 23.4 | 4.93 | |
| ReMDMT=1024, L=10242026.02 | 0.403 | 28.6 | 5.38 | |
| ReMDMT=512, L=10242026.02 | 0.35 | 21.1 | 5.21 | |
| ADLMT=256, L=10242026.02 | 0.349 | 39.9 | 5.46 | |
| DSL-FT + ReMDM (official schedule)T=512, L=10242026.02 | 0.335 | 13.4 | 4.4 | |
| PRISM-loopT=256, L=10242026.02 | 0.294 | 18 | 5.15 | |
| PRISMT=512, L=10242026.02 | 0.281 | 15.4 | 5.04 | |
| PRISMT=256, L=10242026.02 | 0.268 | 16.2 | 5.07 | |
| PRISMT=1024, L=10242026.02 | 0.26 | 15 | 5.02 | |
| MDLM+DFMT=1024, L=10242026.02 | 0.254 | 21.7 | 5.2 | |
| ReMDMT=256, L=10242026.02 | 0.216 | 30.5 | 5.34 | |
| PRISMT=128, L=10242026.02 | 0.175 | 17.9 | 5.1 | |
| MDLM+DFMT=512, L=10242026.02 | 0.168 | 24.2 | 5.26 | |
| ADLMT=128, L=10242026.02 | 0.14 | 52.5 | 5.52 | |
| MDLM+FBT=1024, L=10242026.02 | 0.133 | 33.8 | 5.35 | |
| PRISM-loopT=128, L=10242026.02 | 0.118 | 21.5 | 5.18 | |
| MDLM+FBT=512, L=10242026.02 | 0.103 | 37 | 5.4 | |
| MDLM+DFMT=256, L=10242026.02 | 0.098 | 31.2 | 5.3 | |
| MDLM+FBT=256, L=10242026.02 | 0.086 | 39 | 5.41 | |
| MDLM+FBT=128, L=10242026.02 | 0.064 | 42.8 | 5.44 | |
| ReMDMT=128, L=10242026.02 | 0.057 | 42.5 | 5.43 | |
| MDLMT=1024, L=10242026.02 | 0.042 | 51.3 | 5.46 | |
| MDLM+DFMT=128, L=10242026.02 | 0.041 | 37.9 | 5.31 | |
| MDLMT=512, L=10242026.02 | 0.031 | 53 | 5.48 | |
| MDLMT=256, L=10242026.02 | 0.023 | 55.8 | 5.49 | |
| MDLMT=128, L=10242026.02 | 0.015 | 61.5 | 5.52 | |
| SEDD (absorb)T=512, L=10242026.02 | 0.009 | 107.8 | 5.62 | |
| SEDD (absorb)T=256, L=10242026.02 | 0.008 | 112.9 | 5.63 | |
| SEDD (absorb)T=1024, L=10242026.02 | 0.008 | 104.7 | 5.62 | |
| SEDD (absorb)T=128, L=10242026.02 | 0.007 | 119.2 | 5.65 |