ResearchBenchmarksKnowledge Distillation on The PileKnowledge Distillation on The PileFollow1,200Raw KL DivergenceStudent1,139.881,545.691,951.52,357.31Apr 5, 2026Evaluation resultsMethodMethodLinksRaw KL DivergencePer-Token KL DivergenceNormalized KL ScoreStudentdS=1024, Params=405MdS=1024, Params=405M2026.041,2000.58644.4StudentdS=768, Params=247MdS=768, Params=247M2026.041,3350.65249.4StudentdS=512, Params=127MdS=512, Params=127M2026.041,5010.73355.5StudentdS=256, Params=45MdS=256, Params=45M2026.042,0641.00876.4StudentdS=128, Params=18MdS=128, Params=18M2026.042,7031.32100