Instruction Following on VicunaEval
35Rouge-LGoal Prioritization
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Goal PrioritizationModel=Llama2-13B, Setting=w/ training2023.11 | 35 | — | 85 | |
| Aligned SFTModel=Llama2-13B, Setting=w/ training2023.11 | 34.9 | — | 82.5 | |
| Goal Prioritization w/o harmModel=Llama2-13B, Setting=w/ training2023.11 | 34.5 | — | 83.8 | |
| Vanilla SFTModel=Llama2-13B, Setting=w/ training2023.11 | 34.4 | — | 85 | |
| Goal Prioritization w/o thoughtsModel=Llama2-13B, Setting=w/ training2023.11 | 34.4 | — | 86.3 | |
| Goal Prioritization w/o harmModel=Llama2-7B, Setting=w/ training2023.11 | 34.1 | — | 82.5 | |
| Goal PrioritizationModel=Llama2-7B, Setting=w/ training2023.11 | 33.9 | — | 78.8 | |
| Vanilla SFTModel=Llama2-7B, Setting=w/ training2023.11 | 33.8 | — | 80 | |
| Aligned SFTModel=Llama2-7B, Setting=w/ training2023.11 | 33.8 | — | 81.3 | |
| Goal Prioritization w/o thoughtsModel=Llama2-7B, Setting=w/ training2023.11 | 33.4 | — | 81.3 | |
| OpenLLaMA2-7BBackbone=OpenLLaMA2-7B, Role=Teacher, Training=Fine-tuned2024.06 | 22.6 | 46.2 | — | |
| Adversarial Moment-Matching DistillationBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 21.4 | 47.8 | — | |
| MINILLMBackbone=LLaMA, Parameters=7B2023.06 | 20.7 | 64.1 | — | |
| MiniLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 20.7 | 44.2 | — | |
| MMKD (ours)Model Size=0.1B2024.06 | 20.5 | 23.6 | — | |
| DistiLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 20.4 | 46.3 | — | |
| TeacherBackbone=LLaMA, Parameters=13B2023.06 | 19.4 | 65.1 | — | |
| MiniLLMStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 19.4 | 48.9 | — | |
| GKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 19.3 | 45.5 | — | |
| DistiLLMModel Size=0.1B2024.06 | 19.2 | 22.5 | — | |
| MINILLMBackbone=OPT, Parameters=2.7B2023.06 | 19.1 | 55.9 | — | |
| MiniLLMStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 19.1 | 44.3 | — | |
| ImitKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 19.1 | 41.7 | — | |
| MINILLMBackbone=OPT, Parameters=6.7B2023.06 | 18.7 | 60.1 | — | |
| MiniLLMStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 18.6 | 54.1 | — | |
| MiniLLMModel Size=0.1B2024.06 | 18.5 | 20.5 | — | |
| KDBackbone=LLaMA, Parameters=7B2023.06 | 18.4 | 62.7 | — | |
| MINILLMBackbone=GPT-2, Parameters=760M2023.06 | 18.3 | 45.7 | — | |
| SeqKDBackbone=LLaMA, Parameters=7B2023.06 | 18.1 | 62.6 | — | |
| SeqKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 18 | 38.8 | — | |
| MINILLMBackbone=OPT, Parameters=1.3B2023.06 | 17.9 | 50.6 | — | |
| SeqKDBackbone=OPT, Parameters=6.7B2023.06 | 17.9 | 57.6 | — | |
| TeacherBackbone=OPT, Parameters=13B2023.06 | 17.8 | 58 | — | |
| SFT w/o KDBackbone=OPT, Parameters=6.7B2023.06 | 17.8 | 57.3 | — | |
| MINILLMBackbone=GPT-2, Parameters=340M2023.06 | 17.7 | 42.6 | — | |
| GKDModel Size=0.1B2024.06 | 17.7 | 20.2 | — | |
| KDBackbone=OPT, Parameters=6.7B2023.06 | 17.5 | 57 | — | |
| SFT w/o KDBackbone=LLaMA, Parameters=7B2023.06 | 17.5 | 61.6 | — | |
| TeacherModel=GPT-J-6B2023.06 | 17.4 | 55.8 | — | |
| SFTBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 17.3 | 34.8 | — | |
| SFT w/o KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 17 | 51.5 | — | |
| MINILLMBackbone=GPT-2, Parameters=120M2023.06 | 16.9 | 34.1 | — | |
| SeqKDBackbone=GPT-2, Parameters=340M2023.06 | 16.9 | 43 | — | |
| KDBackbone=GPT-2, Parameters=760M2023.06 | 16.9 | 43.4 | — | |
| SeqKDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 16.9 | 46.6 | — | |
| KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 16.9 | 52.1 | — | |
| SeqKDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.06 | 16.9 | 53 | — | |
| KDBackbone=OPT, Parameters=2.7B2023.06 | 16.7 | 51.3 | — | |
| SeqKDBackbone=OPT, Parameters=1.3B2023.06 | 16.6 | 42.6 | — | |
| SFT w/o KDBackbone=OPT, Parameters=2.7B2023.06 | 16.6 | 44.8 | — | |
| SeqKDBackbone=OPT, Parameters=2.7B2023.06 | 16.5 | 44.5 | — | |
| KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 16.5 | 47.2 | — | |
| KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 16.4 | 43.4 | — | |
| KDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.06 | 16.4 | 33.7 | — | |
| TeacherBackbone=GPT-2, Parameters=1.5B2023.06 | 16.3 | 48.6 | — | |
| SFT w/o KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.06 | 16.3 | 48.6 | — | |
| GPT-2 XL (teacher)Model Size=1.5B2024.06 | 16.2 | 32.7 | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=760M2023.06 | 16.1 | 43.1 | — | |
| SFT w/o KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 16.1 | 43.1 | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=340M2023.06 | 16 | 42.3 | — | |
| SeqKDBackbone=GPT-2, Parameters=760M2023.06 | 15.9 | 42.4 | — | |
| SFT w/o KDBackbone=OPT, Parameters=1.3B2023.06 | 15.6 | 40.5 | — | |
| SeqKDModel Size=0.1B2024.06 | 15.5 | 18.2 | — | |
| KDBackbone=GPT-2, Parameters=340M2023.06 | 15.4 | 42.8 | — | |
| SeqKDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.06 | 15.3 | 42 | — | |
| KDModel Size=0.1B2024.06 | 15.2 | 17.2 | — | |
| KDBackbone=OPT, Parameters=1.3B2023.06 | 14.9 | 40.8 | — | |
| SFT w/o KDBackbone=GPT-2, Parameters=120M2023.06 | 14.7 | 32.8 | — | |
| SFT (student)Model Size=0.1B2024.06 | 14.6 | 17.8 | — | |
| ImitKDModel Size=0.1B2024.06 | 14.5 | 18.6 | — | |
| SeqKDBackbone=GPT-2, Parameters=120M2023.06 | 14.3 | 31 | — | |
| KDBackbone=GPT-2, Parameters=120M2023.06 | 13.4 | 31.9 | — |