Code Generation on HumanEval+ augmented (test)
90Pass@1Mercury Coder
Evaluation results
| Method | Links | |
|---|---|---|
| Mercury CoderType=DLLM2026.01 | 90 | |
| Gemini DiffusionType=DLLM2026.01 | 89.6 | |
| LLaDA2.0-flashType=DLLM, Size=6B/100B2026.01 | 87.8 | |
| Qwen2.5-Coder-14B-InstructType=AR, Size=14B2026.01 | 87.2 | |
| Qwen2.5-Coder-32B-InstructType=AR, Size=32B2026.01 | 87.2 | |
| Qwen2.5-Coder-7B-InstructType=AR, Size=7B2026.01 | 84.1 | |
| Seed-Diffusion-Preview(0705)Type=DLLM2026.01 | 82.8 | |
| Stable-DiffCoder-8B-InstructType=DLLM, Size=8B2026.01 | 82.3 | |
| DeepSeek-Coder-V2-InstructType=AR, Size=21B/236B2026.01 | 82.3 | |
| GPT-4-TurboInstruction data=Not disclosed, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 81.7 | |
| Qwen3-8BType=AR, Size=8B2026.01 | 80.5 | |
| LLaDA2.0-miniType=DLLM, Size=1B/16B2026.01 | 79.9 | |
| CodeQwen1.5-7B-ChatType=AR, Size=7B2026.01 | 78.7 | |
| OpenCoder-8B-InstructType=AR, Size=8B2026.01 | 78.7 | |
| Seed-Coder-8B-InstructType=AR, Size=8B2026.01 | 78.7 | |
| CodeQwen1.5-7B-ChatInstruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 77.7 | |
| DeepSeek-Coder-V2-Lite-InstructType=AR, Size=2.4B/16B2026.01 | 75.6 | |
| DeepSeekCoder-33B-InstructInstruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 75 | |
| DeepSeek-Coder-33B-InstructType=AR, Size=33B2026.01 | 75 | |
| Yi-Coder-9B-ChatType=AR, Size=9B2026.01 | 74.4 | |
| OpenCodeInterpreter-DS-33BInstruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 73.8 | |
| WeDLM-8B-ChatType=DLLM, Size=8B2026.01 | 73.8 | |
| SDLM-32BType=DLLM, Size=32B2026.01 | 73.8 | |
| WizardCoder-33B-v1.1Instruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 73.2 | |
| Codestral-22BType=AR, Size=22B2026.01 | 73.2 | |
| RMI Top 50%%=502026.03 | 73.17 | |
| Full Data%=1002026.03 | 72.56 | |
| RMI 50-75%%=252026.03 | 72.56 | |
| RDS+%=252026.03 | 71.34 | |
| DeepSeek-Coder-6.7B-InstructType=AR, Size=6.7B2026.01 | 71.3 | |
| SCAR%=252026.03 | 70.73 | |
| Llama3-70B-InstructInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 70.7 | |
| Magicoder-S-DS-6.7BInstruction data=GPT distillation, Transparent=true, Non-proprietary=true, Non-distilled=false, Decoding=greedy2024.10 | 70.7 | |
| RMI 50% (G)%=502026.03 | 70.12 | |
| RMI Top-25%%=252026.03 | 70.12 | |
| Mixtral-8x22B-Instruct-v0.1Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 70.1 | |
| Random%=252026.03 | 69.51 | |
| RMI QW 50%%=502026.03 | 68.9 | |
| RMI Bot-25%%=252026.03 | 67.68 | |
| SelfCodeAlign-CQ-7BInstruction data=Self-generated, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 67.1 | |
| IFD%=252026.03 | 66.46 | |
| CodeLlama-70B-InstructType=AR, Size=70B2026.01 | 65.9 | |
| CodeLlama-70B-InstructInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 65.2 | |
| DiffuCoder-7B-InstructType=DLLM, Size=7B2026.01 | 65.2 | |
| Snowflake Arctic (480B)Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 64.3 | |
| Mistral LargeInstruction data=Not disclosed, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 62.8 | |
| StarCoder2-15B-InstructType=AR, Size=15B2026.01 | 60.4 | |
| Llama-3.1-8B-InstructType=AR, Size=8B2026.01 | 59.8 | |
| Command-R+ (104B)Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 56.7 | |
| Gemini ProInstruction data=Proprietary, Transparent=false, Non-proprietary=false, Non-distilled=true, Decoding=greedy2024.10 | 55.5 | |
| LoRABase Model=Llama-3.1-8B2025.09 | 53.7 | |
| BoHABase Model=Llama-3.1-8B2025.09 | 52.4 | |
| HiRABase Model=Llama-3.1-8B2025.09 | 51.8 | |
| CodeQwen1.5-7B-OctoPackInstruction data=Publicly available, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 50.6 | |
| ABBABase Model=Llama-3.1-8B2025.09 | 50 | |
| CodeQwen1.5-7B-BaseTransparent=false, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 45.7 | |
| Fast-dLLMv2Type=DLLM, Size=7B2026.01 | 40.2 | |
| Mixtral-8x7B-Instruct-v0.1Instruction data=Not disclosed, Transparent=false, Non-proprietary=true, Decoding=greedy2024.10 | 39.6 | |
| StarCoder2-15BTransparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 37.8 | |
| CodeLlama-7B-InstructType=AR, Size=7B2026.01 | 33.5 | |
| OctoCoder-16BInstruction data=Publicly available, Transparent=true, Non-proprietary=true, Non-distilled=true, Decoding=greedy2024.10 | 32.9 | |
| BoHABase Model=Llama-3.2-1B2025.09 | 24.4 | |
| ABBABase Model=Llama-3.2-1B2025.09 | 22.6 | |
| HiRABase Model=Llama-3.2-1B2025.09 | 22.6 | |
| LoRABase Model=Llama-3.2-1B2025.09 | 22 |