Instruction Following on SelfInst (GPT-4 Feedback Score & Rouge-L)
21.7Rouge-LAdversarial Moment-Matching Distillation
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Adversarial Moment-Matching DistillationBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 21.7 | 54.2 | — | |
| OpenLLaMA2-7BBackbone=OpenLLaMA2-7B, Role=Teacher, Training=Fine-tuned2024.06 | 21.6 | 56.7 | — | |
| GKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 20.9 | 52.4 | — | |
| DistiLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 20.8 | 53.4 | — | |
| TeacherModel=Llama, #Params=6.7B2026.02 | 20.8 | — | 25.1 | |
| MiniLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 20.2 | 51.8 | — | |
| SFTModel=LLaMA-3, Role=Teacher, #Param=1.23B2026.04 | 18.8 | — | — | |
| ImitKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 18.4 | 45 | — | |
| DISTILLLENSModel=Llama, #Params=1.1B2026.02 | 18.2 | — | 19 | |
| FKL + RKLModel=Llama, #Params=1.1B2026.02 | 17.5 | — | 17.2 | |
| KDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 17.4 | 43.5 | — | |
| AKLModel=Llama, #Params=1.1B2026.02 | 17.4 | — | 16.9 | |
| TeacherModel=GPT-J-6B2023.06 | 17.3 | 57.4 | — | |
| MiniLLMStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 17.1 | 52.5 | — | |
| SFTModel=Llama, #Params=1.1B2026.02 | 17.1 | — | 15.7 | |
| KD (FKL)Model=Llama, #Params=1.1B2026.02 | 17 | — | 16.8 | |
| JSDModel=Llama, #Params=1.1B2026.02 | 16.9 | — | 16.9 | |
| MiniLLMStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 16.6 | 48.5 | — | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 16.46 | — | — | |
| MiniLLMStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 16.3 | 43.7 | — | |
| SFTBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 16.3 | 40.8 | — | |
| RKLModel=Llama, #Params=1.1B2026.02 | 16.3 | — | 16 | |
| SeqKDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 16.2 | 47.2 | — | |
| SeqKDModel=Llama, #Params=1.1B2026.02 | 16.2 | — | 14.8 | |
| KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 16 | 47 | — | |
| SFT w/o KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 15.8 | 45.4 | — | |
| SeqKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 15.8 | 46.8 | — | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 15.26 | — | — | |
| DISTILLLENSModel=GPT-2, #Params=340M2026.02 | 14.6 | — | 10.3 | |
| GPT-2 XL (Teacher)Model Role=Teacher, Parameters=1.5B2025.10 | 14.55 | — | — | |
| KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 14.5 | 46 | — | |
| SFT w/o KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 14.3 | 42.9 | — | |
| GPT-2 XL (teacher)Model Size=1.5B2024.06 | 14.3 | 34.7 | — | |
| TeacherModel=GPT-2, #Params=1.5B2026.02 | 14.3 | — | 12.5 | |
| MMKD (ours)Model Size=0.1B2024.06 | 14.2 | 22.7 | — | |
| SeqKDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 14 | 39.2 | — | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 13.74 | — | — | |
| SeqKDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 13.6 | 43.2 | — | |
| KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 13.4 | 38.9 | — | |
| RKLModel=GPT-2, #Params=340M2026.02 | 13.3 | — | 9.4 | |
| MiniLLMModel Size=0.1B2024.06 | 13.2 | 20.5 | — | |
| FKL + RKLModel=GPT-2, #Params=340M2026.02 | 12.9 | — | 10 | |
| GKDModel Size=0.1B2024.06 | 12.7 | 20.7 | — | |
| AKLModel=GPT-2, #Params=340M2026.02 | 12.7 | — | 9.1 | |
| DistiLLMModel Size=0.1B2024.06 | 12.5 | 21.7 | — | |
| SFT w/o KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 12.4 | 38.3 | — | |
| DISTILLLENSModel=GPT-2, #Params=120M2026.02 | 12.4 | — | 7 | |
| KDModel Size=0.1B2024.06 | 12.3 | 18 | — | |
| SFTModel=GPT-2, #Params=340M2026.02 | 12.3 | — | 9.2 | |
| KD (FKL)Model=GPT-2, #Params=340M2026.02 | 12.1 | — | 8.5 | |
| SeqKDModel=GPT-2, #Params=340M2026.02 | 12.1 | — | 8.9 | |
| JSDModel=GPT-2, #Params=340M2026.02 | 12 | — | 9.3 | |
| SeqKDModel Size=0.1B2024.06 | 11.6 | 18.2 | — | |
| ImitKDModel Size=0.1B2024.06 | 11.5 | 18.2 | — | |
| RKLModel=GPT-2, #Params=120M2026.02 | 10.6 | — | 5.9 | |
| FKL + RKLModel=GPT-2, #Params=120M2026.02 | 10.4 | — | 6.1 | |
| JSDModel=GPT-2, #Params=120M2026.02 | 10.4 | — | 6.2 | |
| AKLModel=GPT-2, #Params=120M2026.02 | 10.4 | — | 6.4 | |
| SFT (student)Model Size=0.1B2024.06 | 10.3 | 20.2 | — | |
| KD (FKL)Model=GPT-2, #Params=120M2026.02 | 10.3 | — | 5.7 | |
| SeqKDModel=GPT-2, #Params=120M2026.02 | 10.1 | — | 4.8 | |
| SFTModel=GPT-2, #Params=120M2026.02 | 10 | — | 4.9 | |
| MaKDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 9.38 | — | — | |
| SFTModel=GPT-2, Role=Teacher, #Param=124M2026.04 | 9.05 | — | — | |
| MaKDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 8.26 | — | — | |
| SFTModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 7.82 | — | — | |
| KDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 7.23 | — | — | |
| SeqKDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 6.76 | — | — | |
| SeqKDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 6.33 | — | — | |
| SFTModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 6.31 | — | — | |
| KDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 6.18 | — | — | |
| NoneModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 4.19 | — | — | |
| NoneModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 0.55 | — | — |