Instruction Following on SelfInst
23.4R-L ScoreTeacher
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| TeacherBackbone=LLaMA, Parameters=13B2023.06 | 23.4 | — | 75.5 | |
| MINILLMBackbone=LLaMA, Parameters=7B2023.06 | 23.2 | — | 73.1 | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2023.06 | 20.8 | — | 69.2 | |
| SeqKDBackbone=LLaMA, Parameters=7B2023.06 | 20.8 | — | 71.5 | |
| KDBackbone=LLaMA, Parameters=7B2023.06 | 20.2 | — | 70.5 | |
| TeacherBackbone=OPT, Parameters=13B2023.06 | 18.4 | — | 56.1 | |
| MINILLMBackbone=OPT, Parameters=6.7B2023.06 | 17.5 | — | 58.5 | |
| MINILLMBackbone=OPT, Parameters=2.7B2023.06 | 17.2 | — | 52.7 | |
| KDBackbone=OPT, Parameters=6.7B2023.06 | 17 | — | 58 | |
| SeqKDBackbone=OPT, Parameters=6.7B2023.06 | 17 | — | 54 | |
| SFT w/o KDBackbone=OPT, Parameters=6.7B2023.06 | 16.4 | — | 56.4 | |
| MINILLMBackbone=GPT-2, Parameters=760M2023.06 | 15.9 | — | 44.6 | |
| MINILLMBackbone=GPT-2, Parameters=340M2023.06 | 15.6 | — | 40.5 | |
| MINILLMBackbone=OPT, Parameters=1.3B2023.06 | 14.8 | — | 47 | |
| LumiNetModel Architecture=GPT-2, Parameters=760M2023.10 | 14.7 | — | — | |
| TeacherModel Architecture=Teacher, Parameters=1.5B2023.10 | 14.3 | — | — | |
| TeacherBackbone=GPT-2, Parameters=1.5B2023.06 | 14.3 | — | 42.9 | |
| SeqKDModel Architecture=GPT-2, Parameters=760M2023.10 | 14 | — | — | |
| SeqKDBackbone=GPT-2, Parameters=760M2023.06 | 14 | — | 38.9 | |
| SFT w/o KDBackbone=OPT, Parameters=2.7B2023.06 | 13.9 | — | 38.9 | |
| LumiNetModel Architecture=GPT-2, Parameters=340M2023.10 | 13.8 | — | — | |
| KDBackbone=OPT, Parameters=2.7B2023.06 | 13.8 | — | 48.6 | |
| KDModel Architecture=GPT-2, Parameters=760M2023.10 | 13.4 | — | — | |
| KDBackbone=GPT-2, Parameters=760M2023.06 | 13.4 | — | 40.4 | |
| SeqKDBackbone=OPT, Parameters=2.7B2023.06 | 13.3 | — | 40.5 | |
| DISTILLLENS + MiniLLMPolicy=Hybrid2026.02 | 13.3 | — | — | |
| MINILLMBackbone=GPT-2, Parameters=120M2023.06 | 13.2 | — | 29.2 | |
| MiniLLMPolicy=On2026.02 | 13.2 | — | — | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=340M2023.10 | 13 | — | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2023.06 | 13 | — | 39.6 | |
| SeqKDBackbone=OPT, Parameters=1.3B2023.06 | 12.7 | — | 36.6 | |
| SeqKDModel Architecture=GPT-2, Parameters=340M2023.10 | 12.6 | — | — | |
| SeqKDBackbone=GPT-2, Parameters=340M2023.06 | 12.6 | — | 39 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=760M2023.10 | 12.4 | — | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=760M2023.06 | 12.4 | — | 38.3 | |
| DISTILLLENSPolicy=Off2026.02 | 12.4 | — | — | |
| KDBackbone=OPT, Parameters=1.3B2023.06 | 12.2 | — | 36 | |
| KDModel Architecture=GPT-2, Parameters=340M2023.10 | 12 | — | — | |
| KDBackbone=GPT-2, Parameters=340M2023.06 | 12 | — | 39.2 | |
| LumiNetModel Architecture=GPT-2, Parameters=120M2023.10 | 11.4 | — | — | |
| SFT w/o KDBackbone=OPT, Parameters=1.3B2023.06 | 11.4 | — | 37.7 | |
| KDModel Architecture=GPT-2, Parameters=120M2023.10 | 10.8 | — | — | |
| KDBackbone=GPT-2, Parameters=120M2023.06 | 10.8 | — | 27.8 | |
| DistiLLMPolicy=On2026.02 | 10.8 | — | — | |
| GKDPolicy=On2026.02 | 10.4 | — | — | |
| KD (FKL)Policy=Off2026.02 | 10.3 | — | — | |
| SeqKDModel Architecture=GPT-2, Parameters=120M2023.10 | 10.1 | — | — | |
| SeqKDBackbone=GPT-2, Parameters=120M2023.06 | 10.1 | — | 26.2 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=120M2023.10 | 10 | — | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=120M2023.06 | 10 | — | 26.3 | |
| CEDistillation Method=None (Cross-Entropy), Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | — | 64.6 | — | |
| FullKDDistillation Method=Full KD, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | — | 66.1 | — | |
| Ours 12Distillation Method=Random Sampling KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | — | 73.1 | — | |
| Top-K 12Distillation Method=Top-K KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | — | 60.9 | — | |
| Top-K 50Distillation Method=Top-K KD, Number of Unique Tokens=50, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | — | 63.4 | — |