Instruction Following on UnNI (Rouge-L)
40.2Rouge-LMINILLM
Evaluation results
| Method | Links | |
|---|---|---|
| MINILLMBackbone=LLaMA, Parameters=7B2023.06 | 40.2 | |
| Adversarial Moment-Matching DistillationBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 39.1 | |
| TeacherBackbone=LLaMA, Parameters=13B2023.06 | 38.5 | |
| OpenLLaMA2-7BBackbone=OpenLLaMA2-7B, Role=Teacher, Training=Fine-tuned2024.06 | 38.5 | |
| DistiLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 38.2 | |
| KDBackbone=LLaMA, Parameters=7B2023.06 | 37.9 | |
| MINILLMBackbone=GPT-2, Parameters=760M2023.06 | 37.7 | |
| SeqKDBackbone=LLaMA, Parameters=7B2023.06 | 37.6 | |
| MiniLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 37.5 | |
| MINILLMBackbone=OPT, Parameters=6.7B2023.06 | 36.7 | |
| TeacherBackbone=OPT, Parameters=13B2023.06 | 36.1 | |
| Teacher (Qwen1.5)Distillation Setting=Cross-Tokenizer KD2026.03 | 35.98 | |
| MiniLLMStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 35.9 | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2023.06 | 35.8 | |
| TeacherModel=Llama, #Params=6.7B2026.02 | 35.8 | |
| AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 35.64 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 35.64 | |
| MiniLLMStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 35.5 | |
| MiniLLMStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 35.4 | |
| MINILLMBackbone=OPT, Parameters=2.7B2023.06 | 35.1 | |
| KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 34.9 | |
| GKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 34.8 | |
| SFT w/o KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 34.6 | |
| MINILLMBackbone=GPT-2, Parameters=340M2023.06 | 34.5 | |
| MiniLLMDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 34.4 | |
| DISTILLLENSModel=Llama, #Params=1.1B2026.02 | 34.3 | |
| SeqKDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 34.2 | |
| TeacherModel=GPT-J-6B2023.06 | 33.6 | |
| MINILLMBackbone=OPT, Parameters=1.3B2023.06 | 33.4 | |
| KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 33.2 | |
| AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 33.15 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 33.15 | |
| ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 33.11 | |
| DISTILLLENSModel=GPT-2, #Params=340M2026.02 | 33 | |
| SeqKDBackbone=GPT-2, Parameters=760M2023.06 | 32.9 | |
| SeqKDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 32.9 | |
| FKL + RKLModel=Llama, #Params=1.1B2026.02 | 32.8 | |
| KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 32.7 | |
| SeqKDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 32.5 | |
| JSDModel=Llama, #Params=1.1B2026.02 | 32.5 | |
| SFT w/o KDBackbone=OPT, Parameters=2.7B2023.06 | 32.3 | |
| SeqKDBackbone=OPT, Parameters=2.7B2023.06 | 32.3 | |
| GPT-2 XL (teacher)Model Size=1.5B2024.06 | 32.2 | |
| AKLModel=Llama, #Params=1.1B2026.02 | 32.1 | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2023.06 | 32 | |
| RKLModel=Llama, #Params=1.1B2026.02 | 32 | |
| TeacherBackbone=GPT-2, Parameters=1.5B2023.06 | 31.8 | |
| TeacherModel=GPT-2, #Params=1.5B2026.02 | 31.8 | |
| SFTModel=Llama, #Params=1.1B2026.02 | 31.8 | |
| KDBackbone=GPT-2, Parameters=760M2023.06 | 31.7 | |
| SFT w/o KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 31.6 | |
| TeacherModel=GPT-2 XL (1.5B)2025.10 | 31.41 | |
| GPT-2 XL (Teacher)Model Role=Teacher, Parameters=1.5B2025.10 | 31.41 | |
| Teacher (GPT-2 XL)Distillation Setting=Same-Tokenizer KD2026.03 | 31.41 | |
| OPT 6.7BRole=Teacher2026.05 | 31.39 | |
| KD (FKL)Model=Llama, #Params=1.1B2026.02 | 31.1 | |
| TeacherModel=GPT-2-1.5B2025.09 | 31.1 | |
| GPT-2 1.5BRole=Teacher2026.05 | 31.1 | |
| TAIDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 31.09 | |
| DistiLLM (SRKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 31.04 | |
| KDBackbone=GPT-2, Parameters=340M2023.06 | 31 | |
| GKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 30.99 | |
| ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 30.94 | |
| DistiLLM (SKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 30.73 | |
| MiniLLMDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 30.7 | |
| SeqKDModel=Llama, #Params=1.1B2026.02 | 30.6 | |
| AMiDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.10 | 30.35 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 30.35 | |
| SeqKDBackbone=GPT-2, Parameters=340M2023.06 | 30.2 | |
| KDBackbone=OPT, Parameters=2.7B2023.06 | 30.2 | |
| MMKD (ours)Model Size=0.1B2024.06 | 29.9 | |
| TAIDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 29.79 | |
| DistiLLM (SRKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 29.31 | |
| KDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 28.94 | |
| AKLDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 28.91 | |
| DistiLLM (SKL)Distillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 28.84 | |
| ImitKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 28.7 | |
| SFT w/o KDBackbone=OPT, Parameters=6.7B2023.06 | 28.6 | |
| SFTBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 28.6 | |
| SeqKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 28.58 | |
| CISTStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 28.41 | |
| SFT w/o KDBackbone=OPT, Parameters=1.3B2023.06 | 28.4 | |
| SeqKDBackbone=OPT, Parameters=1.3B2023.06 | 28.2 | |
| SeqKDBackbone=OPT, Parameters=6.7B2023.06 | 28.2 | |
| DISTILLLENSModel=GPT-2, #Params=120M2026.02 | 27.9 | |
| DISTILLLENSPolicy=Off2026.02 | 27.9 | |
| SeqKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 27.8 | |
| SeqKDModel=GPT-2, #Params=340M2026.02 | 27.8 | |
| DistiLLMModel Size=0.1B2024.06 | 27.6 | |
| ImitKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Large (0.8B), Student Model Parameters=0.8B2025.10 | 27.59 | |
| SFTDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 27.51 | |
| RKLModel=GPT-2, #Params=340M2026.02 | 27.5 | |
| ABKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 (0.1B), Student Model Parameters=0.1B2025.10 | 27.36 | |
| AKLModel=GPT-2, #Params=340M2026.02 | 27.3 | |
| DISTILLLENS + MiniLLMPolicy=Hybrid2026.02 | 27.2 | |
| GKDDistillation Setup=GPT-2 XL (1.5B) → GPT-2 Medium (0.3B), Student Model Parameters=0.3B2025.10 | 27.14 | |
| SFT w/o KDBackbone=GPT-2, Parameters=760M2023.06 | 27.1 | |
| SFT w/o KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 27.1 | |
| KDBackbone=OPT, Parameters=1.3B2023.06 | 27 | |
| FKL + RKLModel=GPT-2, #Params=340M2026.02 | 27 |