Instruction Following on Vicuna Eval (test)
20.37ROUGE-LImitKD + Ours
Evaluation Results
| Method | Links | |
|---|---|---|
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 20.37 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 20.05 | |
| ImitKD + DLD-meanLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 19.58 | |
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 19.37 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 19.15 | |
| TAIDLoss=tKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.09 | 18.14 | |
| Teacher (OpenLLaMA-7B)Teacher Backbone=OpenLLaMA-7B2025.09 | 17.85 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 17.69 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 17.55 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 17.18 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 17.18 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 17.05 | |
| DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 16.99 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.98 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.94 | |
| DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.91 | |
| DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.9 | |
| CSDLoss=CSD, D=On, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 16.88 | |
| ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.88 | |
| ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 16.77 | |
| DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 16.66 | |
| GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 16.65 | |
| GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 16.59 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 16.45 | |
| TeacherModel=GPT-2-1.5B2025.09 | 16.31 | |
| Teacher (GPT-2-1.5B)Teacher Backbone=GPT-2-1.5B2025.09 | 16.31 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 16.24 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 15.83 | |
| CSDLoss=CSD, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 15.78 | |
| GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 15.61 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.09 | 15.53 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.09 | 15.27 | |
| SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 15.09 | |
| SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 14.73 | |
| ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.09 | 14.65 | |
| KDLoss=KL, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.09 | 14.57 |