Instruction Following on S-NI (OOD)
36.1Rouge-LTeacher
Evaluation Results
| Method | Links | |
|---|---|---|
| TeacherBackbone=LLaMA, Parameters=13B2026.06 | 36.1 | |
| ARKDBackbone=LLaMA, Parameters=7B2026.06 | 35.5 | |
| FKL+RKLBackbone=LLaMA, Parameters=7B2026.06 | 34.7 | |
| RKLBackbone=LLaMA, Parameters=7B2026.06 | 34.1 | |
| FKLBackbone=LLaMA, Parameters=7B2026.06 | 33.8 | |
| SeqKDBackbone=LLaMA, Parameters=7B2026.06 | 33.3 | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2026.06 | 32.4 | |
| TeacherBackbone=GPT-2, Parameters=1.5B2026.06 | 27.2 | |
| ARKDBackbone=GPT-2, Parameters=340M2026.06 | 26.8 | |
| FKL+RKLBackbone=GPT-2, Parameters=340M2026.06 | 26.3 | |
| RKLBackbone=GPT-2, Parameters=340M2026.06 | 26.1 | |
| FKLBackbone=GPT-2, Parameters=340M2026.06 | 25.7 | |
| SeqKDBackbone=GPT-2, Parameters=340M2026.06 | 25.4 | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2026.06 | 24.9 | |
| ARKDBackbone=GPT-2, Parameters=120M2026.06 | 19.1 | |
| FKL+RKLBackbone=GPT-2, Parameters=120M2026.06 | 17.8 | |
| RKLBackbone=GPT-2, Parameters=120M2026.06 | 17.2 | |
| FKLBackbone=GPT-2, Parameters=120M2026.06 | 16.6 | |
| SFT w/o KDBackbone=GPT-2, Parameters=120M2026.06 | 16.4 | |
| SeqKDBackbone=GPT-2, Parameters=120M2026.06 | 16.3 |