Code Generation on MBPP (Accuracy, Avg.)
70.42Average ScoreEevee
Evaluation results
| Method | Links | ||
|---|---|---|---|
| EeveeCondition=Prompt learning generalized from main four-benchmark suite2026.06 | 70.42 | — | |
| BaselineCondition=Shared unadapted baseline2026.06 | 69.29 | — | |
| GEPACondition=Prompt learning generalized from main four-benchmark suite2026.06 | 68.2 | — | |
| ACECondition=Prompt learning generalized from main four-benchmark suite2026.06 | 67.47 | — | |
| HybKDTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 47.89 | 63.12 | |
| Skew F(R)KLTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 47.29 | 60.85 | |
| α-β divergenceTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 47.09 | 60.05 | |
| Forward KLTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 46.76 | 63.49 | |
| Reverse KLTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 46.34 | 62.17 | |
| Hard KDTeacher-Student=DeepSeek-Coder-6.7B -> 1.3B2026.05 | 44.66 | 60.32 | |
| GDSD w/ TLCSequence Length=Average, k-shot=32026.05 | 43.3 | — | |
| ESPOSequence Length=Average, k-shot=32026.05 | 42.7 | — | |
| GDSDSequence Length=Average, k-shot=32026.05 | 42 | — | |
| SPGSequence Length=Average, k-shot=32026.05 | 40.5 | — | |
| Base ModelBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 38.84 | — | |
| LLaDA-8BSequence Length=Average, k-shot=32026.05 | 38.8 | — | |
| diffu-GRPO (d1)Sequence Length=Average, k-shot=32026.05 | 36.5 | — | |
| wd1Sequence Length=Average, k-shot=32026.05 | 36.5 | — | |
| Frozen RouterBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 35.2 | — | |
| ProbMoEBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 35 | — | |
| DenseMixerBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 34 | — | |
| DefaultMoEBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 33.2 | — | |
| ReMoEBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 33 | — | |
| ConventionalBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 32.8 | — | |
| DenseMixerBackbone=OLMoE-1B-7B, Top-k=82026.06 | 24.4 | — | |
| ConventionalBackbone=OLMoE-1B-7B, Top-k=82026.06 | 23.2 | — | |
| ProbMoEBackbone=OLMoE-1B-7B, Top-k=82026.06 | 22.8 | — | |
| Base ModelBackbone=OLMoE-1B-7B, Top-k=82026.06 | 19.8 | — | |
| Frozen RouterBackbone=OLMoE-1B-7B, Top-k=82026.06 | 17.8 | — | |
| SparseMixerBackbone=Qwen1.5-MoE-A2.7B, Top-k=42026.06 | 0 | — | |
| diffu-GRPO (d1)Sequence Length=128, k-shot=32026.05 | — | 34.8 | |
| diffu-GRPO (d1)Sequence Length=256, k-shot=32026.05 | — | 36.6 | |
| diffu-GRPO (d1)Sequence Length=512, k-shot=32026.05 | — | 38 | |
| ESPOSequence Length=128, k-shot=32026.05 | — | 43.6 | |
| ESPOSequence Length=256, k-shot=32026.05 | — | 43.2 | |
| ESPOSequence Length=512, k-shot=32026.05 | — | 41.2 | |
| GDSDSequence Length=128, k-shot=32026.05 | — | 40.6 | |
| GDSDSequence Length=256, k-shot=32026.05 | — | 41.8 | |
| GDSDSequence Length=512, k-shot=32026.05 | — | 43.6 | |
| GDSD w/ TLCSequence Length=128, k-shot=32026.05 | — | 43 | |
| GDSD w/ TLCSequence Length=256, k-shot=32026.05 | — | 43.6 | |
| GDSD w/ TLCSequence Length=512, k-shot=32026.05 | — | 43.2 | |
| LLaDA-8BSequence Length=128, k-shot=32026.05 | — | 36.2 | |
| LLaDA-8BSequence Length=256, k-shot=32026.05 | — | 42 | |
| LLaDA-8BSequence Length=512, k-shot=32026.05 | — | 38.1 | |
| SPGSequence Length=128, k-shot=32026.05 | — | 40.4 | |
| SPGSequence Length=256, k-shot=32026.05 | — | 40.8 | |
| SPGSequence Length=512, k-shot=32026.05 | — | 40.4 | |
| wd1Sequence Length=128, k-shot=32026.05 | — | 38 | |
| wd1Sequence Length=256, k-shot=32026.05 | — | 37.2 | |
| wd1Sequence Length=512, k-shot=32026.05 | — | 34.4 |