Language Generation on Dolly databricks 15k (test)
29.7ROUGE-LTeacher
Evaluation results
| Method | Links | |
|---|---|---|
| TeacherModel=LLaMA, #Params=13B2026.03 | 29.7 | |
| VQAEModel=LLaMA, #Params=7B2026.03 | 28.2 | |
| TeacherModel=GPT-2, #Params=1.5B2026.03 | 27.6 | |
| SeqKDModel=LLaMA, #Params=7B2026.03 | 27.5 | |
| KDModel=LLaMA, #Params=7B2026.03 | 27.4 | |
| SFT w/o KDModel=LLaMA, #Params=7B2026.03 | 26.3 | |
| KDModel=GPT-2, #Params=760M2026.03 | 25.9 | |
| VQAEModel=GPT-2, #Params=760M2026.03 | 25.7 | |
| SeqKDModel=GPT-2, #Params=760M2026.03 | 25.6 | |
| SFT w/o KDModel=GPT-2, #Params=760M2026.03 | 25.4 | |
| VQAEModel=GPT-2, #Params=120M2026.03 | 23.5 | |
| SFT w/o KDModel=GPT-2, #Params=120M2026.03 | 23.3 | |
| KDModel=GPT-2, #Params=120M2026.03 | 22.8 | |
| SeqKDModel=GPT-2, #Params=120M2026.03 | 22.7 |