Instruction Following on Self-Instruct (test)
21.81ROUGE-LImitKD + Ours
Evaluation Results
| Method | Links | |
|---|---|---|
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 21.81 | |
| AMiD (Ours)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 20.99 | |
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 20.76 | |
| DistiLLM (SRKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 20.76 | |
| ABKDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 20.46 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 20.2 | |
| DistiLLM (SKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 20.2 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 20.07 | |
| ImitKD + DLD-meanLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 19.91 | |
| Teacher (OpenLLaMA-7B)Teacher Backbone=OpenLLaMA-7B2025.09 | 18.17 | |
| TeacherModel Name=OpenLLaMA2-7B2025.10 | 18.17 | |
| TAIDLoss=tKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 17.73 | |
| TAIDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 17.73 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 14.85 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 14.85 | |
| TeacherModel=GPT-2-1.5B2025.09 | 14.07 | |
| Teacher (GPT-2-1.5B)Teacher Backbone=GPT-2-1.5B2025.09 | 14.07 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 13.96 | |
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 13.75 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 13.71 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 13.64 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 13.51 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 12.64 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 12.51 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 12.4 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 12.4 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 12.37 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 12.23 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 12.1 | |
| CSDLoss=CSD, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 12.06 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 12.04 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 12.03 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 11.93 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 11.86 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 11.67 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 11.22 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 11.15 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 11.03 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 10.56 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 10.25 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 10.08 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 10.02 |