Code Generation on MBPP standard (test)
94.2Pass@1QualityFlow
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| QualityFlowLLM Backbone=Claude Sonnet-3.5-v22025.01 | 94.2 | — | — | 4.8 | |
| DeepSeek-Coder-V2-Instruct2025.01 | 89.4 | — | — | — | |
| LEDEX RLBackbone=CodeLlama-13B, Stage=Refine2024.05 | 61.98 | — | 79.95 | — | |
| LEDEX SFTBackbone=CodeLlama-13B, Stage=Refine2024.05 | 61.78 | — | 79.96 | — | |
| LEDEX RLBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.05 | 61.63 | — | 80.27 | — | |
| LEDEX SFTBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.05 | 61.59 | — | 80.21 | — | |
| LEDEX RLBackbone=StarCoder-15B, Stage=Expl. + Refine2024.05 | 58.19 | — | 77.96 | — | |
| LEDEX SFTBackbone=CodeLlama-7B, Stage=Refine2024.05 | 58.07 | — | 77.34 | — | |
| LEDEX RLBackbone=StarCoder-15B, Stage=Refine2024.05 | 58 | — | 78.12 | — | |
| LEDEX SFTBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.05 | 57.98 | — | 77.92 | — | |
| LEDEX RLBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.05 | 57.92 | — | 78.97 | — | |
| LEDEX RLBackbone=CodeLlama-7B, Stage=Refine2024.05 | 57.35 | — | 78.12 | — | |
| LEDEX SFTBackbone=StarCoder-15B, Stage=Expl. + Refine2024.05 | 57.11 | — | 76.7 | — | |
| LEDEX SFTBackbone=StarCoder-15B, Stage=Refine2024.05 | 56.66 | — | 75.78 | — | |
| LEDEX SFTBackbone=CodeLlama-13B, Stage=Init2024.05 | 52.43 | — | 73.66 | — | |
| LEDEX RLBackbone=CodeLlama-13B, Stage=Init2024.05 | 51.19 | — | 73.16 | — | |
| PromptBackbone=CodeLlama-13B, Stage=Expl. + Refine2024.05 | 49.97 | — | 76.39 | — | |
| PromptBackbone=CodeLlama-13B, Stage=Refine2024.05 | 49.68 | — | 75.85 | — | |
| LEDEX SFTBackbone=CodeLlama-7B, Stage=Init2024.05 | 48.87 | — | 70.89 | — | |
| LEDEX RLBackbone=StarCoder-15B, Stage=Init2024.05 | 48.62 | — | 70.68 | — | |
| LEDEX SFTBackbone=StarCoder-15B, Stage=Init2024.05 | 47.41 | — | 70.27 | — | |
| LEDEX RLBackbone=CodeLlama-7B, Stage=Init2024.05 | 46.54 | — | 71.54 | — | |
| PromptBackbone=CodeLlama-7B, Stage=Refine2024.05 | 43.98 | — | 71.94 | — | |
| PromptBackbone=CodeLlama-7B, Stage=Expl. + Refine2024.05 | 43.42 | — | 72.09 | — | |
| PromptBackbone=CodeLlama-13B, Stage=Init2024.05 | 42.88 | — | 70.85 | — | |
| PromptBackbone=StarCoder-15B, Stage=Refine2024.05 | 40.64 | — | 71.04 | — | |
| PromptBackbone=StarCoder-15B, Stage=Expl. + Refine2024.05 | 40.37 | — | 71.6 | — | |
| PromptBackbone=CodeLlama-7B, Stage=Init2024.05 | 38.21 | — | 67.24 | — | |
| PromptBackbone=StarCoder-15B, Stage=Init2024.05 | 37.7 | — | 69.6 | — | |
| Source Model baselineModel=Llama-2-7b, PEFT=LoRA, Protocol=Zero-shot2024.05 | — | 27.2 | — | — | |
| Source Model baselineModel=Gemma-2b, PEFT=LoRA, Protocol=Zero-shot2024.05 | — | 41.1 | — | — | |
| Target Model baselineModel=Llama-2-13b, PEFT=None, Protocol=Zero-shot2024.05 | — | 37.1 | — | — | |
| Target Model baselineModel=Gemma-7b, PEFT=None, Protocol=Zero-shot2024.05 | — | 37.9 | — | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Llama-2-13b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05 | — | 39.7 | — | — | |
| Trans-LoRASource Model=Gemma-2b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05 | — | 50 | — | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Gemma-2b, Protocol=Zero-shot2024.05 | — | 48.7 | — | — | |
| Trans-LoRASource Model=Llama-2-7b, Target Model=Gemma-7b, Discriminator Model=Llama-2-7b, Protocol=Zero-shot2024.05 | — | 48.7 | — | — | |
| Trans-LoRASource Model=Llama2-7B, Configuration=Source LoRA2026.03 | — | 27.2 | — | — | |
| Trans-LoRATarget Model=Llama2-13B, Configuration=Target (no LoRA)2026.03 | — | 37.1 | — | — | |
| Trans-LoRASource Model=Llama2-7B, Target Model=Llama2-13B, Configuration=Target LoRA2026.03 | — | 39.7 | — | — | |
| Trans-LoRASource Model=Gemma-2B, Configuration=Source LoRA2026.03 | — | 41.1 | — | — | |
| Trans-LoRATarget Model=Gemma-7B, Configuration=Target (no LoRA)2026.03 | — | 37.9 | — | — | |
| Trans-LoRASource Model=Gemma-2B, Target Model=Gemma-7B, Configuration=Target LoRA2026.03 | — | 50 | — | — | |
| TuneShift-KDSource Model=Llama2-7B, Configuration=Source LoRA (Reproduced)2026.03 | — | 28.2 | — | — | |
| TuneShift-KDTarget Model=Llama2-13B, Configuration=Target (no LoRA) (Reproduced)2026.03 | — | 36.9 | — | — | |
| TuneShift-KDSource Model=Llama2-7B, Target Model=Llama2-13B, Configuration=Target LoRA2026.03 | — | 40.2 | — | — | |
| TuneShift-KDSource Model=Gemma-2B, Configuration=Source LoRA (Reproduced)2026.03 | — | 40.9 | — | — | |
| TuneShift-KDTarget Model=Gemma-7B, Configuration=Target (no LoRA) (Reproduced)2026.03 | — | 37.8 | — | — | |
| TuneShift-KDSource Model=Gemma-2B, Target Model=Gemma-7B, Configuration=Target LoRA2026.03 | — | 51.3 | — | — |