Instruction Following on Dolly
71.3ScoreOurs 12
Evaluation Results
| Method | Links | |
|---|---|---|
| Ours 12Distillation Method=Random Sampling KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 71.3 | |
| FullKDDistillation Method=Full KD, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 66.1 | |
| Top-K 50Distillation Method=Top-K KD, Number of Unique Tokens=50, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 65.4 | |
| CEDistillation Method=None (Cross-Entropy), Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 64.2 | |
| Top-K 12Distillation Method=Top-K KD, Number of Unique Tokens=12, Teacher Model Size=8B, Student Model Size=3B, Evaluation Judge=Llama 3.1 405B Instruct2025.03 | 59 | |
| FedAvg+AlignFedBackbone=Llama3-8B2026.06 | 58.52 | |
| FedAvgBackbone=Llama3-8B2026.06 | 58.49 | |
| FFA-LoRA+AlignFedBackbone=Llama3-8B2026.06 | 58.32 | |
| FFA-LoRABackbone=Llama3-8B2026.06 | 58.21 | |
| FedSA-LoRA+AlignFedBackbone=Llama3-8B2026.06 | 58.13 | |
| FedSA-LoRABackbone=Llama3-8B2026.06 | 58 | |
| FFA-LoRA+FedBuffBackbone=Llama3-8B2026.06 | 57.94 | |
| FedAvg+FedBuffBackbone=Llama3-8B2026.06 | 57.89 | |
| FedSA-LoRA+FedBuffBackbone=Llama3-8B2026.06 | 57.71 | |
| LumiNetModel Architecture=GPT-2, Parameters=760M2023.10 | 28.6 | |
| LumiNetModel Architecture=GPT-2, Parameters=340M2023.10 | 27.8 | |
| TeacherModel Architecture=Teacher, Parameters=1.5B2023.10 | 27.6 | |
| KDModel Architecture=GPT-2, Parameters=760M2023.10 | 25.9 | |
| SeqKDModel Architecture=GPT-2, Parameters=760M2023.10 | 25.6 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=340M2023.10 | 25.5 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=760M2023.10 | 25.4 | |
| SeqKDModel Architecture=GPT-2, Parameters=340M2023.10 | 25.3 | |
| KDModel Architecture=GPT-2, Parameters=340M2023.10 | 25 | |
| LumiNetModel Architecture=GPT-2, Parameters=120M2023.10 | 23.8 | |
| SFT w/o KDModel Architecture=GPT-2, Parameters=120M2023.10 | 23.3 | |
| CBDParams=134.002026.05 | 22.99 | |
| KDModel Architecture=GPT-2, Parameters=120M2023.10 | 22.8 | |
| SeqKDModel Architecture=GPT-2, Parameters=120M2023.10 | 22.7 | |
| CBDParams=277.002026.05 | 20.82 | |
| Van-LearngeneParams=297.242026.05 | 20.67 | |
| Auto-LearngeneParams=297.242026.05 | 20.43 | |
| Auto-LearngeneParams=174.282026.05 | 20.3 | |
| Auto-LearngeneParams=235.762026.05 | 20.03 | |
| Van-LearngeneParams=235.762026.05 | 19.92 | |
| CBDParams=220.002026.05 | 19.82 | |
| Van-LearngeneParams=174.282026.05 | 19.62 |