Instruction-following on (val)
34.39ROUGE-LAMiD (Ours)
Evaluation Results
| Method | Links | |
|---|---|---|
| AMiD (Ours)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 34.39 | |
| ABKDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 33.91 | |
| DistiLLM (SRKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 33.18 | |
| DistiLLM (SKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 33.07 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 31.1 | |
| TAIDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 30.85 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 30.83 | |
| DistiLLM (SRKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 30.59 | |
| ABKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 30.49 | |
| TAIDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 29.83 | |
| DistiLLM (SRKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 29.72 | |
| DistiLLM (SKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 29.69 | |
| DistiLLM (SKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 29.65 | |
| ABKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 29.64 | |
| TAIDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 29.45 | |
| GKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Large (0.8B), Student Parameters=0.8B2025.10 | 29.36 | |
| AMiDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 29.24 | |
| ABKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 28.61 | |
| TAIDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 28.37 | |
| DistiLLM (SRKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 28.21 | |
| GKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 Medium (0.3B), Student Parameters=0.3B2025.10 | 27.9 | |
| DistiLLM (SKL)Teacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 27.88 | |
| GKDTeacher Model=GPT-2 XL (1.5B), Student Model=GPT-2 (0.1B), Student Parameters=0.1B2025.10 | 27.06 |