Language Modeling on Fineweb-edu distillation 8B to 300M
2.74LM LossRandom Sampling KD
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Random Sampling KD2025.03 | 2.74 | 100 | 0.9 | |
| FullKD2025.03 | 2.75 | 100 | 0.7 | |
| Raman et al. (2023)sampling method only=false2025.03 | 2.77 | 57 | 1.9 | |
| Peng et al. (2024)sampling method only=true2025.03 | 2.78 | -47 | 1.7 | |
| Raman et al. (2023)sampling method only=true2025.03 | 2.8 | 5 | 2.2 | |
| CE2025.03 | 2.81 | 0 | 1.2 | |
| Peng et al. (2024)sampling method only=false2025.03 | 2.85 | -78 | 1.4 |