List your company
ResearchFormerly WizWand

Instruction Following on Dolly Eval (test)

29.69ROUGE-LAMiD (Ours)
21.47423.60725.7427.873Sep 30, 2025Oct 2, 2025Oct 4, 2025Oct 6, 2025Oct 8, 2025Oct 10, 2025Oct 13, 2025
Updated 8mo ago

Evaluation results

MethodLinks
AMiD (Ours)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.1029.69
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0929.63
ABKDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.1029.43
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0929.07
DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0928.83
DistiLLM (SRKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.1028.83
DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0928.63
DistiLLM (SKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.1028.63
ImitKD + DLD-meanLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0928.13
Teacher (OpenLLaMA-7B)Teacher Backbone=OpenLLaMA-7B2025.0927.6
TeacherModel Name=OpenLLaMA2-7B2025.1027.6
CSDLoss=CSD, D=On, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0927.14
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0927.14
GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0927.11
TeacherModel=GPT-2-1.5B2025.0927
Teacher (GPT-2-1.5B)Teacher Backbone=GPT-2-1.5B2025.0927
DistiLLMLoss=SRKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0926.92
DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0926.77
TAIDLoss=tKL, Distillation dataset protocol=Ada, Teacher Backbone=OpenLLaMA-7B, Student Backbone=OpenLLaMA-3B2025.0926.53
TAIDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.1026.53
GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0926.06
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0925.82
CSDLoss=CSD, D=On, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0925.7
ImitKD + OursLoss=CSD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0925.7
GKD + OursLoss=CSD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0925.5
DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0925.43
KDLoss=KL, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0925.41
DistiLLM + OursLoss=CSD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0925.34
GKD + DLDLoss=DLD, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0925.29
DistiLLMLoss=SKL, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0925.28
GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0925.15
SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0925.09
CSDLoss=CSD, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0924.94
SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.3B, Teacher Model=GPT-2-1.5B2025.0924.79
ImitKD + DLDLoss=DLD, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0924.69
DistiLLM + DLDLoss=DLD, Distillation dataset protocol=Ada, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0924.23
SeqKDLoss=SFT, D=pT, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0923.86
ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.3B2025.0923.61
KDLoss=KL, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0923.52
SFTLoss=SFT, D=Fix, Student Model=GPT-2-0.1B, Teacher Model=GPT-2-1.5B2025.0923.49
GKDLoss=GJS, Distillation dataset protocol=Mix, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0922.48
ImitKDLoss=KL, Distillation dataset protocol=On, Teacher Backbone=GPT-2-1.5B, Student Backbone=GPT-2-0.1B2025.0921.79