Language Generation on Self-Instruct (test)
23.4ROUGE-LTeacher
Evaluation results
| Method | Links | |
|---|---|---|
| TeacherModel=LLaMA, #Params=13B2026.03 | 23.4 | |
| SFT w/o KDModel=LLaMA, #Params=7B2026.03 | 20.8 | |
| SeqKDModel=LLaMA, #Params=7B2026.03 | 20.8 | |
| VQAEModel=LLaMA, #Params=7B2026.03 | 20.5 | |
| KDModel=LLaMA, #Params=7B2026.03 | 20.2 | |
| TeacherModel=GPT-2, #Params=1.5B2026.03 | 14.3 | |
| SeqKDModel=GPT-2, #Params=760M2026.03 | 14 | |
| KDModel=GPT-2, #Params=760M2026.03 | 13.4 | |
| VQAEModel=GPT-2, #Params=760M2026.03 | 13.3 | |
| SFT w/o KDModel=GPT-2, #Params=760M2026.03 | 12.4 | |
| KDModel=GPT-2, #Params=120M2026.03 | 10.8 | |
| VQAEModel=GPT-2, #Params=120M2026.03 | 10.4 | |
| SeqKDModel=GPT-2, #Params=120M2026.03 | 10.1 | |
| SFT w/o KDModel=GPT-2, #Params=120M2026.03 | 10 |