Instruction Following on Dolly (Rouge-L)
27.74Rouge-LCIST
Evaluation Results
| Method | Links | |
|---|---|---|
| CISTStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 27.74 | |
| OPT 6.7BRole=Teacher2026.05 | 27.52 | |
| SFTModel=LLaMA-3, Role=Teacher, #Param=1.23B2026.04 | 27.47 | |
| Teacher (Qwen1.5)Distillation Setting=Cross-Tokenizer KD2026.03 | 27.45 | |
| Teacher (GPT-2 XL)Distillation Setting=Same-Tokenizer KD2026.03 | 27.07 | |
| GPT-2 1.5BRole=Teacher2026.05 | 27 | |
| ABStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.86 | |
| RKLStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.58 | |
| JSStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.39 | |
| AKLStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.2 | |
| FKLStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.07 | |
| SRKLStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 26.04 | |
| DSKDDistillation Setting=Same-Tokenizer KD2026.03 | 25.98 | |
| Sym-KLStudent=OPT 1.3B, Teacher=OPT 6.7B2026.05 | 25.73 | |
| CISTStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 25.42 | |
| DISTILLLENSPolicy=Off2026.02 | 25.2 | |
| DISTILLLENS + MiniLLMPolicy=Hybrid2026.02 | 25.1 | |
| DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.03 | 24.92 | |
| DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=SKL2026.03 | 24.79 | |
| RKLStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 24.67 | |
| DSKD-CMADistillation Setting=Cross-Tokenizer KD, Divergence Function=SRKL2026.03 | 24.64 | |
| DSKD-CMA-CTDistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.03 | 24.63 | |
| MiniLLMPolicy=On2026.02 | 24.6 | |
| DSKD-CLPDistillation Setting=Cross-Tokenizer KD2026.03 | 24.52 | |
| SRKLStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 24.48 | |
| GKDPolicy=On2026.02 | 24.4 | |
| Sym-KLStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 24.38 | |
| ABStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 24.32 | |
| DSKD-CLADistillation Setting=Cross-Tokenizer KD2026.03 | 24.2 | |
| DSKD-CMADistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.03 | 24.13 | |
| DistiLLMPolicy=On2026.02 | 24.1 | |
| JSStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 24.08 | |
| FKLStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 23.8 | |
| KD (FKL)Policy=Off2026.02 | 23.5 | |
| SFTDistillation Setting=Cross-Tokenizer KD2026.03 | 22.83 | |
| SFTModel=GPT-2, Role=Teacher, #Param=124M2026.04 | 22.5 | |
| AKLStudent=GPT-2 0.1B, Teacher=GPT-2 1.5B2026.05 | 21.83 | |
| ULDDistillation Setting=Cross-Tokenizer KD2026.03 | 21.73 | |
| MinEDDistillation Setting=Cross-Tokenizer KD2026.03 | 21.69 | |
| MaKDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 20.74 | |
| MaKDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 18.56 | |
| SeqKDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 17.05 | |
| SeqKDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 16.56 | |
| KDModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 16.25 | |
| SFTModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 16.1 | |
| KDModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 16.04 | |
| SFTModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 15.47 | |
| Student (GPT-2)Distillation Setting=Baseline2026.03 | 6.39 | |
| NoneModel=GPT-2, Role=Student, #Param=77.4M2026.04 | 5.69 | |
| NoneModel=LLaMA-3, Role=Student, #Param=0.69B2026.04 | 0.5 |