List your company
ResearchFormerly WizWand

Instruction Following on Self-Instruct (ROUGE-L)

16.5ROUGE-LMiniLLM
9.344811.202413.0614.9176Sep 30, 2025Oct 2, 2025Oct 4, 2025Oct 6, 2025Oct 8, 2025Oct 10, 2025Oct 13, 2025
Updated 8mo ago

Evaluation results

MethodLinks
MiniLLMDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1016.5
AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1016.46
DistiLLM (SKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1015.69
ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1015.46
AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1015.26
TAIDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1015.07
MiniLLMDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1014.87
DistiLLM (SRKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1014.61
TeacherModel=GPT-2 XL (1.5B)2025.1014.55
TAIDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1014.53
GKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1014.44
DistiLLM (SKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1014.11
TeacherModel=GPT-2-1.5B2025.0914.07
SeqKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1013.93
AKLDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1013.83
DistiLLM (SRKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1013.79
SFTDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1013.78
AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1013.74
KDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1013.72
ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1013.69
ImitKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.1013.26
SeqKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1013.01
TAIDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1012.91
MiniLLMDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1012.83
SFTDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1012.6
ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1012.52
GKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1012.36
DistiLLM (SKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1012.35
SRKLLoss=SRKL, Parameter=0.12025.0912.19
DistiLLM (SRKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1012.13
CSDLoss=CSD2025.0912.06
AKLDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1012.06
ImitKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1012
ABLoss=AB, Parameters=0.2, 0.72025.0911.82
GKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1011.78
GJSLoss=GJS, Parameter=0.92025.0911.4
SKLLoss=SKL, Parameter=0.12025.0911.21
RKLLoss=RKL2025.0911.19
AKLDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1011.18
SeqKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1011.12
TVLoss=TV2025.0911.03
JeffreyLoss=Jeffrey2025.0910.82
KDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.1010.58
ImitKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1010.34
Sym-KLLoss=Sym-KL2025.0910.24
KDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.1010.12
KLLoss=KL2025.0910.02
SFTDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.109.62