Instruction Following on Super-NI (test)
36.82ROUGE-LDistiLLM
Evaluation Results
| Method | Links | |
|---|---|---|
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 36.82 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 36.49 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 36.3 | |
| ImitKD + DLD-meanLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 35.85 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 35.31 | |
| TAIDLoss=tKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 31.93 | |
| Teacher (OpenLLaMA-7B)Teacher Backbone=OpenLLaMA-7B2025.09 | 31.05 | |
| TeacherModel=GPT-2-1.5B2025.09 | 26.46 | |
| Teacher (GPT-2-1.5B)Teacher Backbone=GPT-2-1.5B2025.09 | 26.46 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 26.29 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 26.28 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 26.28 | |
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 26.12 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 25.51 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 25.49 | |
| CSDLoss=CSD, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 24.6 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 23.91 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 23.42 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 22.96 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 22.91 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 22.91 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 22.69 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 22.13 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 21.58 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 21.39 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 21.04 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 20.87 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 20.2 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 20.13 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 19.6 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 18.91 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 17.35 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 17.35 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 17.13 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 16.76 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 13.88 |