Unconditional Generation on FineWeb-Edu Uniform source 170M-parameter (train)
7EntropyFS-DFM
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| FS-DFMSteps=42026.05 | 7 | 169.7 | 99.8 | 188.9 | |
| TS-DFMInitialization=FS-DFM init, Steps=42026.05 | 7 | 100.3 | 57.1 | 98.5 | |
| TS-DFMInitialization=DFM init, Steps=42026.05 | 7.1 | 121.2 | 73.4 | 120.6 | |
| TS-DFMInitialization=FS-DFM init, Steps=82026.05 | 7.2 | 56.1 | 32.2 | 54.6 | |
| TS-DFMInitialization=DFM init, Steps=82026.05 | 7.3 | 79 | 47.7 | 79.9 | |
| FS-DFMSteps=82026.05 | 7.4 | 87.6 | 52 | 98.5 | |
| FS-DFMSteps=162026.05 | 7.4 | 73.2 | 45.1 | 81.6 | |
| TS-DFMInitialization=FS-DFM init, Steps=162026.05 | 7.4 | 47.2 | 27.1 | 46.4 | |
| TS-DFMInitialization=FS-DFM init, Steps=322026.05 | 7.5 | 46.7 | 28.5 | 47.4 | |
| TS-DFMInitialization=DFM init, Steps=162026.05 | 7.6 | 70.1 | 41.5 | 69.5 | |
| FS-DFMSteps=322026.05 | 7.7 | 67.8 | 39 | 69.5 | |
| DFM (Teacher)Steps=10242026.05 | 7.8 | 82.8 | 47.8 | 78.9 | |
| TS-DFMInitialization=DFM init, Steps=322026.05 | 7.8 | 71 | 41 | 69.7 | |
| DFM (Teacher)Steps=162026.05 | 8 | 176.9 | 100.8 | 177.2 | |
| DFM (Teacher)Steps=322026.05 | 8 | 117.7 | 67 | 114.1 | |
| DFM (Teacher)Steps=42026.05 | 8.1 | 749.9 | 413.3 | 772.9 | |
| DFM (Teacher)Steps=82026.05 | 8.1 | 328.9 | 186.7 | 340.4 |