Language Modeling on Pre-training dataset
2.48LM LossCross-Entropy
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Cross-EntropyTraining Tokens=100B, Student Model=300M, Teacher Model=3B2025.03 | 2.48 | 0.7 | |
| Random Sampling KDTraining Tokens=100B, Student Model=300M, Teacher Model=3B2025.03 | 2.48 | 0.3 | |
| Full Knowledge DistillationTraining Tokens=100B, Student Model=300M, Teacher Model=3B2025.03 | 2.48 | 0.4 |