Instruction Following on Vicuna
58.2ScoreOurs 12
Evaluation Results
| Method | Links | |
|---|---|---|
| Ours 12Distillation Method=Random Sampling KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 58.2 | |
| FullKDDistillation Method=Full KD, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 56.9 | |
| Top-K 50Distillation Method=Top-K KD, Number of Unique Tokens=50, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 53.1 | |
| CEDistillation Method=None (Cross-Entropy), Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 49.1 | |
| Top-K 12Distillation Method=Top-K KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 48.9 | |
| LumiNetModel Architecture=GPT-2, Parameters=760M2023.10 | 17.5 | |
| LumiNetModel Architecture=GPT-2, Parameters=340M2023.10 | 17.1 | |
| SeqKDModel Architecture=GPT-2, Parameters=340M2023.10 | 16.9 | |
| KDModel Architecture=GPT-2, Parameters=760M2023.10 | 16.9 | |
| TeacherModel Architecture=Teacher, Parameters=1.5B2023.10 | 16.3 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=760M2023.10 | 16.1 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=340M2023.10 | 16 | |
| SeqKDModel Architecture=GPT-2, Parameters=760M2023.10 | 15.9 | |
| KDModel Architecture=GPT-2, Parameters=340M2023.10 | 15.4 | |
| LumiNetModel Architecture=GPT-2, Parameters=120M2023.10 | 14.9 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=120M2023.10 | 14.7 | |
| SeqKDModel Architecture=GPT-2, Parameters=120M2023.10 | 14.3 | |
| KDModel Architecture=GPT-2, Parameters=120M2023.10 | 13.4 |