Code Generation on HumanEval and MBPP EvalPlus
70.1HumanEval+ Pass@kMistral-7B-Instruct-v0.2
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| Mistral-7B-Instruct-v0.2Size=7B, Version=v0.2, Mode=Instruct2024.05 | 70.1 | 75 | 44.7 | 37 | 59.9 | 53.6 | |
| MAmmoTH2-7B-PlusSize=7B2024.05 | 65.9 | 72.1 | 60.1 | 50.4 | 66.1 | 58.2 | |
| DeRS-SMMoE Model=Coder-MoE [6], Added Params.=0.81B+325M2025.03 | 62.8 | 66.5 | 63.4 | 51.4 | 61 | — | |
| DeRS-LMMoE Model=Coder-MoE [6], Added Params.=0.81B+9.09M2025.03 | 62.8 | 65.9 | 62.9 | 51.9 | 60.9 | — | |
| Vanilla UpcyclingMoE Model=Coder-MoE [6], Added Params.=0.81B+2.43B2025.03 | 61 | 64.6 | 63.9 | 51.4 | 60.2 | — | |
| MAmmoTH2-8B-PlusSize=8B2024.05 | 57.9 | 63.4 | 60.4 | 48.6 | 61.9 | 53.3 | |
| Llama-3-8B-InstructSize=8B, Mode=Instruct2024.05 | 56.7 | 61.6 | 70.1 | 59.3 | 65.9 | 58 | |
| MAmmoTH2-8×7B-PlusSize=8x7B2024.05 | 53.7 | 57.9 | 68.7 | 56.9 | 63.3 | 55.3 | |
| SFTModel=Phi3-mini-3.8B2025.02 | 51.83 | — | — | 48.68 | — | — | |
| Mixtral-8×7B-Instruct-v0.1Size=8x7B, Version=v0.1, Mode=Instruct2024.05 | 39.6 | 45.1 | 59.5 | 49.7 | 52.3 | 44.7 | |
| SFTModel=Qwen2-7B2025.02 | 39.02 | — | — | 39.42 | — | — | |
| Gemma-1.1-7B-InstructSize=7B, Version=1.1, Mode=Instruct2024.05 | 35.4 | 42.7 | 57.1 | 45 | 49.9 | 40.2 | |
| SFTModel=Llama-3-8B2025.02 | 34.76 | — | — | 50.26 | — | — | |
| Llama-3-8BSize=8B2024.05 | 29.3 | 33.5 | 61.4 | 51.6 | 47.5 | 40.5 | |
| SFTModel=Gemma-2-9B2025.02 | 24.39 | — | — | 27.51 | — | — | |
| Mistral-7BSize=7B2024.05 | 23.8 | 28.7 | 51.9 | 42.1 | 40.3 | 33 | |
| CDMStudent=Gemma-2-2B, Teacher=Llama-3-8B, Category=Cross Tokenizer KD2025.02 | 23.78 | — | — | 21.69 | — | — | |
| SFTModel=Gemma-2-2B2025.02 | 21.34 | — | — | 21.34 | — | — | |
| RKLStudent=Qwen2-0.5B, Teacher=Qwen2-7B, Category=Same Tokenizer KD2025.02 | 20.73 | — | — | 22.75 | — | — | |
| ULDStudent=Gemma-2-2B, Teacher=Llama-3-8B, Category=Cross Tokenizer KD2025.02 | 20.4 | — | — | 17.7 | — | — | |
| MinEDStudent=Gemma-2-2B, Teacher=Llama-3-8B, Category=Cross Tokenizer KD2025.02 | 20.12 | — | — | 22.22 | — | — | |
| Gemma-7BSize=7B2024.05 | 20.1 | 26.8 | 52.6 | 43.4 | 39.7 | 31.8 | |
| FKLStudent=Gemma-2-2B, Teacher=Gemma-2-9B, Category=Same Tokenizer KD2025.02 | 18.9 | — | — | 23 | — | — | |
| RKLStudent=Gemma-2-2B, Teacher=Gemma-2-9B, Category=Same Tokenizer KD2025.02 | 18.9 | — | — | 21.42 | — | — | |
| CDMStudent=Qwen2-0.5B, Teacher=Phi3-mini-3.8B, Category=Cross Tokenizer KD2025.02 | 18.9 | — | — | 23.81 | — | — | |
| MinEDStudent=Qwen2-0.5B, Teacher=Phi3-mini-3.8B, Category=Cross Tokenizer KD2025.02 | 17.1 | — | — | 22.2 | — | — | |
| FKLStudent=Qwen2-0.5B, Teacher=Qwen2-7B, Category=Same Tokenizer KD2025.02 | 17.07 | — | — | 23.38 | — | — | |
| ULDStudent=Qwen2-0.5B, Teacher=Phi3-mini-3.8B, Category=Cross Tokenizer KD2025.02 | 17.07 | — | — | 22.49 | — | — | |
| SFTModel=Qwen2-0.5B2025.02 | 15.85 | — | — | 22.22 | — | — | |
| Forward KL2026.05 | — | — | — | — | 46.76 | — | |
| Forward KL + Ours2026.05 | — | — | — | — | 47.89 | — | |
| Hard KD2026.05 | — | — | — | — | 44.66 | — | |
| Reverse KL2026.05 | — | — | — | — | 46.34 | — | |
| Reverse KL + Ours2026.05 | — | — | — | — | 46.82 | — | |
| Skew FKL2026.05 | — | — | — | — | 47.29 | — | |
| Skew FKL + Ours2026.05 | — | — | — | — | 47.92 | — | |
| Skew RKL2026.05 | — | — | — | — | 47.31 | — | |
| Skew RKL + Ours2026.05 | — | — | — | — | 47.68 | — | |
| Student (No Distill)2026.05 | — | — | — | — | 42.26 | — | |
| Teacher2026.05 | — | — | — | — | 72.56 | — | |
| Total Variation2026.05 | — | — | — | — | 46.54 | — | |
| Total Variation + Ours2026.05 | — | — | — | — | 47.06 | — | |
| α-β divergence2026.05 | — | — | — | — | 47.09 | — | |
| α-β divergence + Ours2026.05 | — | — | — | — | 49.74 | — |