Instruction Following on SelfInst (OOD)
23.2Rouge-LTeacher
Evaluation results
| Method | Links | |
|---|---|---|
| TeacherBackbone=LLaMA, Parameters=13B2026.06 | 23.2 | |
| ARKDBackbone=LLaMA, Parameters=7B2026.06 | 22.6 | |
| FKL+RKLBackbone=LLaMA, Parameters=7B2026.06 | 21.8 | |
| RKLBackbone=LLaMA, Parameters=7B2026.06 | 21 | |
| SeqKDBackbone=LLaMA, Parameters=7B2026.06 | 20.8 | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2026.06 | 20.6 | |
| FKLBackbone=LLaMA, Parameters=7B2026.06 | 20.5 | |
| TeacherBackbone=GPT-2, Parameters=1.5B2026.06 | 15.1 | |
| ARKDBackbone=GPT-2, Parameters=340M2026.06 | 15 | |
| FKL+RKLBackbone=GPT-2, Parameters=340M2026.06 | 14.6 | |
| RKLBackbone=GPT-2, Parameters=340M2026.06 | 14.4 | |
| FKLBackbone=GPT-2, Parameters=340M2026.06 | 13.8 | |
| ARKDBackbone=GPT-2, Parameters=120M2026.06 | 13.6 | |
| FKL+RKLBackbone=GPT-2, Parameters=120M2026.06 | 13.2 | |
| SeqKDBackbone=GPT-2, Parameters=340M2026.06 | 13.1 | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2026.06 | 13 | |
| RKLBackbone=GPT-2, Parameters=120M2026.06 | 12.5 | |
| FKLBackbone=GPT-2, Parameters=120M2026.06 | 12.1 | |
| SFT w/o KDBackbone=GPT-2, Parameters=120M2026.06 | 11.9 | |
| SeqKDBackbone=GPT-2, Parameters=120M2026.06 | 11.6 |