Language Modeling on Fineweb-edu 1.0 (test)
2.32LM LossRandom Sampling KD (Ours 12+)
Evaluation Results
| Method | Links | |
|---|---|---|
| Random Sampling KD (Ours 12+)Student Size=3B, Teacher Size=8B, Training Tokens=100B, Unique Tokens=12, Adaptive Weighting=true2025.03 | 2.32 | |
| FullKDStudent Size=3B, Teacher Size=8B, Training Tokens=100B2025.03 | 2.34 | |
| Random Sampling KD (Ours 12)Student Size=3B, Teacher Size=8B, Training Tokens=100B, Unique Tokens=122025.03 | 2.35 | |
| CEStudent Size=3B, Teacher Size=8B, Training Tokens=100B2025.03 | 2.37 | |
| Top-K 50Student Size=3B, Teacher Size=8B, Training Tokens=100B, K=502025.03 | 2.4 | |
| Top-K 12Student Size=3B, Teacher Size=8B, Training Tokens=100B, K=122025.03 | 2.5 |