List your company
ResearchFormerly WizWand

Instruction Following on SelfInst

23.4R-L ScoreTeacher
9.46413.08216.720.318Jun 14, 2023Nov 23, 2023May 4, 2024Oct 14, 2024Mar 25, 2025Sep 4, 2025Feb 14, 2026
Updated 8mo ago

Evaluation results

MethodLinks
TeacherBackbone=LLaMA, Parameters=13B2023.0623.4—75.5
MINILLMBackbone=LLaMA, Parameters=7B2023.0623.2—73.1
SFT w/o KDBackbone=LLaMA, Parameters=7B2023.0620.8—69.2
SeqKDBackbone=LLaMA, Parameters=7B2023.0620.8—71.5
KDBackbone=LLaMA, Parameters=7B2023.0620.2—70.5
TeacherBackbone=OPT, Parameters=13B2023.0618.4—56.1
MINILLMBackbone=OPT, Parameters=6.7B2023.0617.5—58.5
MINILLMBackbone=OPT, Parameters=2.7B2023.0617.2—52.7
KDBackbone=OPT, Parameters=6.7B2023.0617—58
SeqKDBackbone=OPT, Parameters=6.7B2023.0617—54
SFT w/o KDBackbone=OPT, Parameters=6.7B2023.0616.4—56.4
MINILLMBackbone=GPT-2, Parameters=760M2023.0615.9—44.6
MINILLMBackbone=GPT-2, Parameters=340M2023.0615.6—40.5
MINILLMBackbone=OPT, Parameters=1.3B2023.0614.8—47
LumiNetModel Architecture=GPT-2, Parameters=760M2023.1014.7——
TeacherModel Architecture=Teacher, Parameters=1.5B2023.1014.3——
TeacherBackbone=GPT-2, Parameters=1.5B2023.0614.3—42.9
SeqKDModel Architecture=GPT-2, Parameters=760M2023.1014——
SeqKDBackbone=GPT-2, Parameters=760M2023.0614—38.9
SFT w/o KDBackbone=OPT, Parameters=2.7B2023.0613.9—38.9
LumiNetModel Architecture=GPT-2, Parameters=340M2023.1013.8——
KDBackbone=OPT, Parameters=2.7B2023.0613.8—48.6
KDModel Architecture=GPT-2, Parameters=760M2023.1013.4——
KDBackbone=GPT-2, Parameters=760M2023.0613.4—40.4
SeqKDBackbone=OPT, Parameters=2.7B2023.0613.3—40.5
DISTILLLENS + MiniLLMPolicy=Hybrid2026.0213.3——
MINILLMBackbone=GPT-2, Parameters=120M2023.0613.2—29.2
MiniLLMPolicy=On2026.0213.2——
SFT w/o KDModel Architecture=GPT-2, Parameters=340M2023.1013——
SFT w/o KDBackbone=GPT-2, Parameters=340M2023.0613—39.6
SeqKDBackbone=OPT, Parameters=1.3B2023.0612.7—36.6
SeqKDModel Architecture=GPT-2, Parameters=340M2023.1012.6——
SeqKDBackbone=GPT-2, Parameters=340M2023.0612.6—39
SFT w/o KDModel Architecture=GPT-2, Parameters=760M2023.1012.4——
SFT w/o KDBackbone=GPT-2, Parameters=760M2023.0612.4—38.3
DISTILLLENSPolicy=Off2026.0212.4——
KDBackbone=OPT, Parameters=1.3B2023.0612.2—36
KDModel Architecture=GPT-2, Parameters=340M2023.1012——
KDBackbone=GPT-2, Parameters=340M2023.0612—39.2
LumiNetModel Architecture=GPT-2, Parameters=120M2023.1011.4——
SFT w/o KDBackbone=OPT, Parameters=1.3B2023.0611.4—37.7
KDModel Architecture=GPT-2, Parameters=120M2023.1010.8——
KDBackbone=GPT-2, Parameters=120M2023.0610.8—27.8
DistiLLMPolicy=On2026.0210.8——
GKDPolicy=On2026.0210.4——
KD (FKL)Policy=Off2026.0210.3——
SeqKDModel Architecture=GPT-2, Parameters=120M2023.1010.1——
SeqKDBackbone=GPT-2, Parameters=120M2023.0610.1—26.2
SFT w/o KDModel Architecture=GPT-2, Parameters=120M2023.1010——
SFT w/o KDBackbone=GPT-2, Parameters=120M2023.0610—26.3
CEDistillation Method=None (Cross-Entropy), Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03—64.6—
FullKDDistillation Method=Full KD, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03—66.1—
Ours 12Distillation Method=Random Sampling KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03—73.1—
Top-K 12Distillation Method=Top-K KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03—60.9—
Top-K 50Distillation Method=Top-K KD, Number of Unique Tokens=50, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03—63.4—