Instruction Following on S-NI (Rouge-L)
38.7Rouge-LAdversarial Moment-Matching Distillation
Evaluation Results
| Method | Links | |
|---|---|---|
| Adversarial Moment-Matching DistillationBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 38.7 | |
| MiniLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 37.4 | |
| DistiLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 37.2 | |
| GKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 36.8 | |
| OpenLLaMA2-7BBackbone=OpenLLaMA2-7B, Role=Teacher, Training=Fine-tuned2024.06 | 36.3 | |
| TeacherBackbone=LLaMA, Parameters=13B2023.06 | 35.8 | |
| MINILLMBackbone=LLaMA, Parameters=7B2023.06 | 35.5 | |
| Teacher (Qwen1.5)Distillation Setting=Cross-Tokenizer KD2026.03 | 34.74 | |
| KDBackbone=LLaMA, Parameters=7B2023.06 | 33.7 | |
| SeqKDBackbone=LLaMA, Parameters=7B2023.06 | 33.7 | |
| ImitKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 33.1 | |
| MINILLMBackbone=OPT, Parameters=6.7B2023.06 | 32.5 | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2023.06 | 32.4 | |
| TeacherModel=Llama, #Params=6.7B2026.02 | 32.4 | |
| DISTILLLENSModel=Llama, #Params=1.1B2026.02 | 30.8 | |
| MINILLMBackbone=OPT, Parameters=2.7B2023.06 | 30.7 | |
| KDBackbone=OPT, Parameters=6.7B2023.06 | 30.7 | |
| TeacherBackbone=OPT, Parameters=13B2023.06 | 30.4 | |
| SeqKDBackbone=OPT, Parameters=6.7B2023.06 | 30.4 | |
| SFTBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 30.4 | |
| SFT w/o KDBackbone=OPT, Parameters=6.7B2023.06 | 30.3 | |
| JSDModel=Llama, #Params=1.1B2026.02 | 30 | |
| FKL + RKLModel=Llama, #Params=1.1B2026.02 | 29.9 | |
| MiniLLMStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 29.8 | |
| SeqKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 29.7 | |
| SFTModel=Llama, #Params=1.1B2026.02 | 29.5 | |
| AKLModel=Llama, #Params=1.1B2026.02 | 29.5 | |
| MINILLMBackbone=GPT-2, Parameters=760M2023.06 | 29.3 | |
| KDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 29.3 | |
| KD (FKL)Model=Llama, #Params=1.1B2026.02 | 28.8 | |
| MINILLMBackbone=OPT, Parameters=1.3B2023.06 | 28.6 | |
| MMKD (ours)Model Size=0.1B2024.06 | 28.6 | |
| MiniLLMStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 28.5 | |
| RKLModel=Llama, #Params=1.1B2026.02 | 28.4 | |
| DISTILLLENSModel=GPT-2, #Params=340M2026.02 | 28.1 | |
| TeacherModel=GPT-J-6B2023.06 | 28 | |
| SeqKDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 28 | |
| DistiLLMModel Size=0.1B2024.06 | 27.7 | |
| SeqKDModel=Llama, #Params=1.1B2026.02 | 27.7 | |
| TeacherBackbone=GPT-2, Parameters=1.5B2023.06 | 27.6 | |
| SFT w/o KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 27.6 | |
| KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 27.6 | |
| GPT-2 XL (teacher)Model Size=1.5B2024.06 | 27.6 | |
| TeacherModel=GPT-2, #Params=1.5B2026.02 | 27.6 | |
| MINILLMBackbone=GPT-2, Parameters=340M2023.06 | 27.4 | |
| KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 27.2 | |
| Teacher (GPT-2 XL)Distillation Setting=Same-Tokenizer KD2026.03 | 27.12 | |
| MiniLLMStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 27.1 | |
| SFT w/o KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 26.5 | |
| TeacherModel=GPT-2-1.5B2025.09 | 26.46 | |
| KDBackbone=OPT, Parameters=2.7B2023.06 | 26.3 | |
| SeqKDBackbone=GPT-2, Parameters=760M2023.06 | 26.1 | |
| SeqKDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 26.1 | |
| DISTILLLENS + MiniLLMPolicy=Hybrid2026.02 | 26.1 | |
| KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 25.9 | |
| SeqKDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 25.5 | |
| MINILLMBackbone=GPT-2, Parameters=120M2023.06 | 25.3 | |
| KDBackbone=GPT-2, Parameters=760M2023.06 | 25.3 | |
| SeqKDBackbone=OPT, Parameters=2.7B2023.06 | 25.3 | |
| MiniLLMPolicy=On2026.02 | 25.3 | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2023.06 | 25.1 | |
| GKDModel Size=0.1B2024.06 | 25.1 | |
| SFT w/o KDBackbone=OPT, Parameters=2.7B2023.06 | 24.9 | |
| CSDLoss=CSD2025.09 | 24.6 | |
| TVLoss=TV2025.09 | 24.58 | |
| DISTILLLENSModel=GPT-2, #Params=120M2026.02 | 24.3 | |
| DISTILLLENSPolicy=Off2026.02 | 24.3 | |
| SeqKDModel=GPT-2, #Params=340M2026.02 | 23.9 | |
| KDBackbone=GPT-2, Parameters=340M2023.06 | 23.7 | |
| RKLModel=GPT-2, #Params=340M2026.02 | 23.4 | |
| SRKLLoss=SRKL, Parameter=0.12025.09 | 23.37 | |
| FKL + RKLModel=GPT-2, #Params=340M2026.02 | 23.2 | |
| JSDModel=GPT-2, #Params=340M2026.02 | 23.2 | |
| AKLModel=GPT-2, #Params=340M2026.02 | 23.2 | |
| SFT w/o KDBackbone=OPT, Parameters=1.3B2023.06 | 23.1 | |
| SeqKDBackbone=GPT-2, Parameters=340M2023.06 | 22.9 | |
| MiniLLMModel Size=0.1B2024.06 | 22.7 | |
| SFTModel=GPT-2, #Params=340M2026.02 | 22.7 | |
| SKLLoss=SKL, Parameter=0.12025.09 | 22.65 | |
| KDBackbone=OPT, Parameters=1.3B2023.06 | 21.9 | |
| KD (FKL)Model=GPT-2, #Params=340M2026.02 | 21.9 | |
| SFT w/o KDBackbone=GPT-2, Parameters=760M2023.06 | 21.5 | |
| SFT w/o KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 21.5 | |
| ABLoss=AB, Parameters=0.2, 0.72025.09 | 21.44 | |
| SeqKDBackbone=OPT, Parameters=1.3B2023.06 | 21.4 | |
| KDModel Size=0.1B2024.06 | 20.8 | |
| GJSLoss=GJS, Parameter=0.92025.09 | 20.28 | |
| RKLLoss=RKL2025.09 | 20.17 | |
| KDBackbone=GPT-2, Parameters=120M2023.06 | 19.7 | |
| DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=SKL2026.03 | 19.44 | |
| DSKD-CLPDistillation Setting=Cross-Tokenizer KD2026.03 | 19.26 | |
| DSKDDistillation Setting=Same-Tokenizer KD2026.03 | 19.22 | |
| DSKD-CLADistillation Setting=Cross-Tokenizer KD2026.03 | 18.53 | |
| DSKD-CMADistillation Setting=Cross-Tokenizer KD, Divergence Function=SRKL2026.03 | 18.4 | |
| ImitKDModel Size=0.1B2024.06 | 18.2 | |
| JeffreyLoss=Jeffrey2025.09 | 18.19 | |
| DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.03 | 18.19 | |
| RKLModel=GPT-2, #Params=120M2026.02 | 17.9 | |
| DSKD-CMA-CTDistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.03 | 17.87 | |
| FKL + RKLModel=GPT-2, #Params=120M2026.02 | 17.6 |