Summarization on TL;DR (distillation set)
27.24Word CountInitial Student
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Initial StudentBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 27.24 | 26.68 | 8.98 | 20.6 | — | 33.42 | |
| Standard KDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 26.5 | 29.15 | 8.51 | 21.84 | — | 12.47 | |
| Initial StudentBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 26.35 | 29.04 | 8.34 | 21.66 | — | 11.51 | |
| Standard KDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 26.32 | 31.45 | 10.04 | 23.78 | — | 37.7 | |
| SeqKDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 26.14 | 29.23 | 8.57 | 21.82 | — | 13.18 | |
| PLaDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 26.02 | 31.46 | 10.79 | 24.91 | — | 40.46 | |
| f-distillBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 25.98 | 32.09 | 10.65 | 24.42 | — | 38.35 | |
| SFT StudentBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 25.88 | 29.08 | 8.44 | 21.76 | — | 11.4 | |
| MiniLLMBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 25.86 | 29.37 | 8.62 | 22.03 | — | 13.73 | |
| TeacherBackbone=GPT-Neo-2.7B, Model Group=GPT-Neo2024.06 | 25.81 | 29.64 | 8.82 | 22.22 | — | 15.35 | |
| MiniLLMBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 25.8 | 31.32 | 10.48 | 24.25 | — | 38.07 | |
| SFTBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 25.77 | 30.06 | 9.95 | 22.97 | — | 35.13 | |
| PLaDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 25.77 | 29.5 | 8.69 | 22.08 | — | 14.76 | |
| f-distillBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 25.73 | 29.34 | 8.62 | 21.91 | — | 13.55 | |
| SeqKDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 25.67 | 31.67 | 10.42 | 24.2 | — | 37.92 | |
| TeacherBackbone=LLaMA-2-13B, Model Group=LLaMA-22024.06 | 25.51 | 33.14 | 11.4 | 25.31 | — | 45.28 |