Function-level Code Generation on MBPP+ augmented (test)
79.6Pass@1LLaDA2.0-flash
Evaluation results
| Method | Links | |
|---|---|---|
| LLaDA2.0-flashType=DLLM, Size=6B/100B2026.01 | 79.6 | |
| Seed-Diffusion-Preview(0705)Type=DLLM2026.01 | 79.4 | |
| Mercury CoderType=DLLM2026.01 | 77.1 | |
| Gemini DiffusionType=DLLM2026.01 | 76 | |
| DeepSeek-Coder-V2-InstructType=AR, Size=21B/236B2026.01 | 75.1 | |
| Qwen2.5-Coder-32B-InstructType=AR, Size=32B2026.01 | 75.1 | |
| LLaDA2.0-miniType=DLLM, Size=1B/16B2026.01 | 74.1 | |
| Stable-DiffCoder-8B-InstructType=DLLM, Size=8B2026.01 | 72.8 | |
| Qwen2.5-Coder-14B-InstructType=AR, Size=14B2026.01 | 72.8 | |
| Qwen2.5-Coder-7B-InstructType=AR, Size=7B2026.01 | 71.7 | |
| Seed-Coder-8B-InstructType=AR, Size=8B2026.01 | 71.2 | |
| GPT-4-TurboInstruction data=Not disclosed, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 70.7 | |
| DeepSeek-Coder-V2-Lite-InstructType=AR, Size=2.4B/16B2026.01 | 70.4 | |
| DeepSeek-Coder-33B-InstructType=AR, Size=33B2026.01 | 70.1 | |
| Yi-Coder-9B-ChatType=AR, Size=9B2026.01 | 69 | |
| OpenCoder-8B-InstructType=AR, Size=8B2026.01 | 69 | |
| OpenCodeInterpreter-DS-33BInstruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 67.7 | |
| CodeQwen1.5-7B-ChatInstruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 67.2 | |
| CodeQwen1.5-7B-ChatType=AR, Size=7B2026.01 | 67.2 | |
| Qwen3-8BType=AR, Size=8B2026.01 | 67.2 | |
| WizardCoder-33B-v1.1Instruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 66.9 | |
| DeepSeekCoder-33B-InstructInstruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 66.7 | |
| Llama3-70B-InstructInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 66.4 | |
| DeepSeek-Coder-6.7B-InstructType=AR, Size=6.7B2026.01 | 65.6 | |
| Magicoder-S-DS-6.7BInstruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 65.4 | |
| SelfCodeAlign-CQ-7BInstruction data=Self-generated, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 65.2 | |
| StarCoder2-15B-InstructType=AR, Size=15B2026.01 | 65.1 | |
| CodeLlama-70B-InstructType=AR, Size=70B2026.01 | 64.6 | |
| Snowflake Arctic (480B)Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 64.3 | |
| CodeQwen1.5-7B-OctoPackInstruction data=Publicly available, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 63.2 | |
| Mixtral-8x22B-Instruct-v0.1Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 62.9 | |
| Codestral-22BType=AR, Size=22B2026.01 | 62.2 | |
| DiffuCoder-7B-InstructType=DLLM, Size=7B2026.01 | 61.9 | |
| CodeLlama-70B-InstructInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 61.7 | |
| RMI 50% (G)%=502026.03 | 60.58 | |
| CodeQwen1.5-7B-BaseTransparent=false, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 60.2 | |
| Full Data%=1002026.03 | 59.52 | |
| Llama-3.1-8B-InstructType=AR, Size=8B2026.01 | 59 | |
| RDS+%=252026.03 | 58.99 | |
| Command-R+ (104B)Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 58.6 | |
| RMI QW 50%%=502026.03 | 58.47 | |
| RMI 50-75%%=252026.03 | 58.47 | |
| RMI Top 50%%=502026.03 | 58.2 | |
| Gemini ProInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 57.9 | |
| Random%=252026.03 | 57.67 | |
| SCAR%=252026.03 | 57.67 | |
| RMI Top-25%%=252026.03 | 57.14 | |
| SDLM-32BType=DLLM, Size=32B2026.01 | 57.1 | |
| Mistral LargeInstruction data=Not disclosed, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 56.6 | |
| RMI Bot-25%%=252026.03 | 54.76 | |
| IFD%=252026.03 | 54.76 | |
| StarCoder2-15BTransparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 53.1 | |
| Mixtral-8x7B-Instruct-v0.1Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 49.7 | |
| OctoCoder-16BInstruction data=Publicly available, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 49.1 | |
| CodeLlama-7B-InstructType=AR, Size=7B2026.01 | 44.4 | |
| Fast-dLLMv2Type=DLLM, Size=7B2026.01 | 41.3 |