Unconditional Generation on FineWeb-Edu Mask source 170M-parameter (train)
8.5EntropyDFM (Teacher)
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| DFM (Teacher)Steps=42026.05 | 8.5 | 1,473.3 | 561.2 | 1,441 | |
| DFM (Teacher)Steps=82026.05 | 8.4 | 761.3 | 307 | 754.9 | |
| DFM (Teacher)Steps=162026.05 | 8.4 | 359.3 | 152.7 | 348.9 | |
| DFM (Teacher)Steps=322026.05 | 8.3 | 190.3 | 87 | 184.2 | |
| FS-DFMSteps=42026.05 | 8.2 | 1,015.7 | 367.1 | 943.5 | |
| FS-DFMSteps=82026.05 | 8.2 | 516.6 | 204.9 | 492.4 | |
| FS-DFMSteps=162026.05 | 8.2 | 284.7 | 122.7 | 271.6 | |
| FS-DFMSteps=322026.05 | 8.2 | 189 | 86.3 | 182.5 | |
| DFM (Teacher)Steps=10242026.05 | 8.1 | 93.6 | 45.7 | 87.9 | |
| TS-DFMInitialization=FS-DFM init, Steps=162026.05 | 8 | 109.9 | 55.2 | 101.9 | |
| TS-DFMInitialization=FS-DFM init, Steps=322026.05 | 8 | 91.4 | 45.4 | 84.5 | |
| TS-DFMInitialization=FS-DFM init, Steps=82026.05 | 7.9 | 188.7 | 94.8 | 177.4 | |
| TS-DFMInitialization=DFM init, Steps=322026.05 | 7.7 | 52.4 | 37 | 51.3 | |
| TS-DFMInitialization=FS-DFM init, Steps=42026.05 | 7.7 | 386.7 | 201.2 | 363.4 | |
| TS-DFMInitialization=DFM init, Steps=162026.05 | 7.6 | 61.9 | 45.6 | 61.1 | |
| TS-DFMInitialization=DFM init, Steps=82026.05 | 7.4 | 91.4 | 73.1 | 91.3 | |
| TS-DFMInitialization=DFM init, Steps=42026.05 | 7.1 | 157.4 | 132.2 | 149.8 |