Dialogue on Anthropic-HH (distillation set)
73.53Response Word CountInitial Student
Evaluation Results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Initial StudentBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 73.53 | 19.77 | 3.36 | 12.67 | — | 8.73 | |
| SFT StudentBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 73.49 | 18.51 | 2.76 | 11.93 | — | 8.79 | |
| SeqKDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 73.11 | 20.74 | 3.75 | 13.05 | — | 9.25 | |
| Standard KDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 73.09 | 20.35 | 3.56 | 12.78 | — | 8.94 | |
| PLaDBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 72.76 | 21.12 | 3.82 | 12.83 | — | 10.38 | |
| f-distillBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 72.68 | 20.67 | 3.84 | 13.74 | — | 9.66 | |
| TeacherBackbone=GPT-Neo-2.7B, Model Group=GPT-Neo2024.06 | 72.55 | 21.47 | 4.12 | 13.66 | — | 15.52 | |
| MiniLLMBackbone=GPT-Neo-1.3B, Model Group=GPT-Neo2024.06 | 72.38 | 20.9 | 3.8 | 13.56 | — | 9.81 | |
| f-distillBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 68.24 | 25.23 | 6.72 | 16.4 | — | 25.98 | |
| MiniLLMBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 68.05 | 25.42 | 6.76 | 16.53 | — | 26.61 | |
| PLaDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 67.93 | 25.54 | 6.82 | 16.7 | — | 27.74 | |
| Initial StudentBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 67.8 | 24.96 | 6.48 | 16.26 | — | 25.63 | |
| TeacherBackbone=LLaMA-2-13B, Model Group=LLaMA-22024.06 | 67.78 | 25.15 | 6.5 | 16.32 | — | 26.95 | |
| SeqKDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 67.73 | 25.02 | 6.51 | 16.28 | — | 26.24 | |
| SFTBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 67.64 | 25.33 | 6.66 | 16.51 | — | 25.52 | |
| Standard KDBackbone=LLaMA-2-7B, Model Group=LLaMA-22024.06 | 67.6 | 25.06 | 6.57 | 16.31 | — | 25.83 |