List your company
ResearchFormerly WizWand

Instruction Following on Vicuna Eval (test)

20.37ROUGE-LImitKD + Ours
14.33815.90417.4719.036Sep 30, 2025
Updated 8mo ago

Evaluation results

MethodLinks
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0920.37
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0920.05
ImitKD + DLD-meanLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0919.58
DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0919.37
DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0919.15
TAIDLoss=tKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0918.14
Teacher (OpenLLaMA-7B)Teacher Backbone=OpenLLaMA-7B2025.0917.85
DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0917.69
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0917.55
CSDLoss=CSD, D=On, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0917.18
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0917.18
DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0917.05
DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0916.99
GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.98
GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.94
DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.91
DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.9
CSDLoss=CSD, D=On, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0916.88
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.88
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0916.77
DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0916.66
GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0916.65
GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0916.59
GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0916.45
TeacherModel=GPT-2-1.5B2025.0916.31
Teacher (GPT-2-1.5B)Teacher Backbone=GPT-2-1.5B2025.0916.31
SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0916.24
KDLoss=KL, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0915.83
CSDLoss=CSD, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0915.78
GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0915.61
ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0915.53
SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0915.27
SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0915.09
SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0914.73
ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0914.65
KDLoss=KL, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0914.57