List your company
ResearchFormerly WizWand

Instruction Following on Vicuna

58.2ScoreOurs 12
11.60823.70435.847.896Oct 5, 2023Jan 1, 2024Mar 30, 2024Jun 27, 2024Sep 24, 2024Dec 22, 2024Mar 21, 2025
Updated 8mo ago

Evaluation results

MethodLinks
Ours 12Distillation Method=Random Sampling KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.0358.2
FullKDDistillation Method=Full KD, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.0356.9
Top-K 50Distillation Method=Top-K KD, Number of Unique Tokens=50, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.0353.1
CEDistillation Method=None (Cross-Entropy), Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.0349.1
Top-K 12Distillation Method=Top-K KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.0348.9
LumiNetModel Architecture=GPT-2, Parameters=760M2023.1017.5
LumiNetModel Architecture=GPT-2, Parameters=340M2023.1017.1
SeqKDModel Architecture=GPT-2, Parameters=340M2023.1016.9
KDModel Architecture=GPT-2, Parameters=760M2023.1016.9
TeacherModel Architecture=Teacher, Parameters=1.5B2023.1016.3
SFT w/o KDModel Architecture=GPT-2, Parameters=760M2023.1016.1
SFT w/o KDModel Architecture=GPT-2, Parameters=340M2023.1016
SeqKDModel Architecture=GPT-2, Parameters=760M2023.1015.9
KDModel Architecture=GPT-2, Parameters=340M2023.1015.4
LumiNetModel Architecture=GPT-2, Parameters=120M2023.1014.9
SFT w/o KDModel Architecture=GPT-2, Parameters=120M2023.1014.7
SeqKDModel Architecture=GPT-2, Parameters=120M2023.1014.3
KDModel Architecture=GPT-2, Parameters=120M2023.1013.4