List your company
ResearchFormerly WizWand

Instruction Following on S-NI (Rouge-L)

38.7Rouge-LAdversarial Moment-Matching Distillation
16.75622.45328.1533.847Jun 14, 2023Nov 29, 2023May 16, 2024Nov 1, 2024Apr 19, 2025Oct 5, 2025Mar 23, 2026
Updated 7mo ago

Evaluation results

MethodLinks
Adversarial Moment-Matching DistillationBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0638.7
MiniLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0637.4
DistiLLMBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0637.2
GKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0636.8
OpenLLaMA2-7BBackbone=OpenLLaMA2-7B, Role=Teacher, Training=Fine-tuned2024.0636.3
TeacherBackbone=LLaMA, Parameters=13B2023.0635.8
MINILLMBackbone=LLaMA, Parameters=7B2023.0635.5
Teacher (Qwen1.5)Distillation Setting=Cross-Tokenizer KD2026.0334.74
KDBackbone=LLaMA, Parameters=7B2023.0633.7
SeqKDBackbone=LLaMA, Parameters=7B2023.0633.7
ImitKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0633.1
MINILLMBackbone=OPT, Parameters=6.7B2023.0632.5
SFT w/o KDBackbone=LLaMA, Parameters=7B2023.0632.4
TeacherModel=Llama, #Params=6.7B2026.0232.4
DISTILLLENSModel=Llama, #Params=1.1B2026.0230.8
MINILLMBackbone=OPT, Parameters=2.7B2023.0630.7
KDBackbone=OPT, Parameters=6.7B2023.0630.7
TeacherBackbone=OPT, Parameters=13B2023.0630.4
SeqKDBackbone=OPT, Parameters=6.7B2023.0630.4
SFTBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0630.4
SFT w/o KDBackbone=OPT, Parameters=6.7B2023.0630.3
JSDModel=Llama, #Params=1.1B2026.0230
FKL + RKLModel=Llama, #Params=1.1B2026.0229.9
MiniLLMStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.0629.8
SeqKDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0629.7
SFTModel=Llama, #Params=1.1B2026.0229.5
AKLModel=Llama, #Params=1.1B2026.0229.5
MINILLMBackbone=GPT-2, Parameters=760M2023.0629.3
KDBackbone=OpenLLaMA-3B, Role=Student, Training=Fine-tuned2024.0629.3
KD (FKL)Model=Llama, #Params=1.1B2026.0228.8
MINILLMBackbone=OPT, Parameters=1.3B2023.0628.6
MMKD (ours)Model Size=0.1B2024.0628.6
MiniLLMStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.0628.5
RKLModel=Llama, #Params=1.1B2026.0228.4
DISTILLLENSModel=GPT-2, #Params=340M2026.0228.1
TeacherModel=GPT-J-6B2023.0628
SeqKDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.0628
DistiLLMModel Size=0.1B2024.0627.7
SeqKDModel=Llama, #Params=1.1B2026.0227.7
TeacherBackbone=GPT-2, Parameters=1.5B2023.0627.6
SFT w/o KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.0627.6
KDStudent Model=GPT-2-1.5B, Teacher Model=GPT-J-6B2023.0627.6
GPT-2 XL (teacher)Model Size=1.5B2024.0627.6
TeacherModel=GPT-2, #Params=1.5B2026.0227.6
MINILLMBackbone=GPT-2, Parameters=340M2023.0627.4
KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.0627.2
Teacher (GPT-2 XL)Distillation Setting=Same-Tokenizer KD2026.0327.12
MiniLLMStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.0627.1
SFT w/o KDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.0626.5
TeacherModel=GPT-2-1.5B2025.0926.46
KDBackbone=OPT, Parameters=2.7B2023.0626.3
SeqKDBackbone=GPT-2, Parameters=760M2023.0626.1
SeqKDStudent Model=GPT-Neo-2.7B, Teacher Model=GPT-J-6B2023.0626.1
DISTILLLENS + MiniLLMPolicy=Hybrid2026.0226.1
KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.0625.9
SeqKDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.0625.5
MINILLMBackbone=GPT-2, Parameters=120M2023.0625.3
KDBackbone=GPT-2, Parameters=760M2023.0625.3
SeqKDBackbone=OPT, Parameters=2.7B2023.0625.3
MiniLLMPolicy=On2026.0225.3
SFT w/o KDBackbone=GPT-2, Parameters=340M2023.0625.1
GKDModel Size=0.1B2024.0625.1
SFT w/o KDBackbone=OPT, Parameters=2.7B2023.0624.9
CSDLoss=CSD2025.0924.6
TVLoss=TV2025.0924.58
DISTILLLENSModel=GPT-2, #Params=120M2026.0224.3
DISTILLLENSPolicy=Off2026.0224.3
SeqKDModel=GPT-2, #Params=340M2026.0223.9
KDBackbone=GPT-2, Parameters=340M2023.0623.7
RKLModel=GPT-2, #Params=340M2026.0223.4
SRKLLoss=SRKL, Parameter=0.12025.0923.37
FKL + RKLModel=GPT-2, #Params=340M2026.0223.2
JSDModel=GPT-2, #Params=340M2026.0223.2
AKLModel=GPT-2, #Params=340M2026.0223.2
SFT w/o KDBackbone=OPT, Parameters=1.3B2023.0623.1
SeqKDBackbone=GPT-2, Parameters=340M2023.0622.9
MiniLLMModel Size=0.1B2024.0622.7
SFTModel=GPT-2, #Params=340M2026.0222.7
SKLLoss=SKL, Parameter=0.12025.0922.65
KDBackbone=OPT, Parameters=1.3B2023.0621.9
KD (FKL)Model=GPT-2, #Params=340M2026.0221.9
SFT w/o KDBackbone=GPT-2, Parameters=760M2023.0621.5
SFT w/o KDStudent Model=GPT-2-760M, Teacher Model=GPT-J-6B2023.0621.5
ABLoss=AB, Parameters=0.2, 0.72025.0921.44
SeqKDBackbone=OPT, Parameters=1.3B2023.0621.4
KDModel Size=0.1B2024.0620.8
GJSLoss=GJS, Parameter=0.92025.0920.28
RKLLoss=RKL2025.0920.17
KDBackbone=GPT-2, Parameters=120M2023.0619.7
DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=SKL2026.0319.44
DSKD-CLPDistillation Setting=Cross-Tokenizer KD2026.0319.26
DSKDDistillation Setting=Same-Tokenizer KD2026.0319.22
DSKD-CLADistillation Setting=Cross-Tokenizer KD2026.0318.53
DSKD-CMADistillation Setting=Cross-Tokenizer KD, Divergence Function=SRKL2026.0318.4
ImitKDModel Size=0.1B2024.0618.2
JeffreyLoss=Jeffrey2025.0918.19
DSKD-CMA-GADistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.0318.19
RKLModel=GPT-2, #Params=120M2026.0217.9
DSKD-CMA-CTDistillation Setting=Cross-Tokenizer KD, Divergence Function=KL2026.0317.87
FKL + RKLModel=GPT-2, #Params=120M2026.0217.6