Instruction Following on Super-Natural Instructions (test)
68.5ROUGE-LMHR
Evaluation results
| Method | Links | |
|---|---|---|
| MHRBackbone=T5-XL2022.11 | 68.5 | |
| MHR-zBackbone=T5-XL, restricted to routing parameters=true2022.11 | 68 | |
| PolyBackbone=T5-XL2022.11 | 67.8 | |
| LoRABackbone=T5-XL2022.11 | 67.6 | |
| LoRA-bigBackbone=T5-XL2022.11 | 67.2 | |
| Poly-zBackbone=T5-XL, restricted to routing parameters=true2022.11 | 64.6 | |
| ABKDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 39.51 | |
| AMiD (Ours)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 39.06 | |
| PICLParameters=770M2023.05 | 37.6 | |
| VanillaICLParameters=2.7B2023.05 | 37.3 | |
| DistiLLM (SRKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 36.82 | |
| DistiLLM (SKL)Distillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 35.31 | |
| MetaICLParameters=770M2023.05 | 35.3 | |
| VanillaICLParameters=1.5B2023.05 | 34.9 | |
| ExtraLMParameters=770M2023.05 | 34.6 | |
| VanillaICLParameters=770M2023.05 | 34.3 | |
| LLaMA + PEQA#Params=13B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, Quantization=4-bit2023.05 | 34.1 | |
| TAIDDistillation Pair=OpenLLaMA2-7B → OpenLLaMA2-3B2025.10 | 31.93 | |
| LLaMA + LoRA#Params=13B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, LoRA configuration=QKVO162023.05 | 31.3 | |
| TeacherModel Name=OpenLLaMA2-7B2025.10 | 31.05 | |
| Self-SupParameters=770M2023.05 | 30.5 | |
| LLaMA + LoRA w/ OPTQ#Params=13B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, LoRA configuration=QKVO16, Quantization=4-bit2023.05 | 29.2 | |
| LLaMA + PEQA#Params=7B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, Quantization=4-bit2023.05 | 27.1 | |
| LLaMA + LoRA w/ OPTQ#Params=7B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, LoRA configuration=QKVO16, Quantization=4-bit2023.05 | 25 | |
| LLaMA + LoRA#Params=7B, Evaluation protocol=Zero-shot, Training dataset=Alpaca, LoRA configuration=QKVO162023.05 | 24.4 | |
| LLaMA#Params=7B, Evaluation protocol=Zero-shot, Training dataset=Alpaca2023.05 | 9.4 | |
| LLaMA#Params=13B, Evaluation protocol=Zero-shot, Training dataset=Alpaca2023.05 | 8.9 |