Code Generation on BigCodeBench-Instruct Hard
28.4Pass@1o1
Evaluation Results
| Method | Links | |
|---|---|---|
| o12026.03 | 28.4 | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 27 | |
| DeepSeek-V2.5-238B2026.03 | 27 | |
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 25.7 | |
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 25 | |
| EVOLVECODER-4B (r3)Refinement round=r3, Backbone=Qwen3-4B (Thinking)2026.03 | 25 | |
| DS-Coder-V2-InstructSize Category=32B+ LLMs2025.12 | 24.3 | |
| EVOLVECODER-4B (r2)Refinement round=r2, Backbone=Qwen3-4B (Thinking)2026.03 | 24.3 | |
| Critique-Coder-4B2026.03 | 23 | |
| EVOLVECODER-4B (r0)Refinement round=r0, Backbone=Qwen3-4B (Thinking)2026.03 | 23 | |
| EVOLVECODER-4B (r1)Refinement round=r1, Backbone=Qwen3-4B (Thinking)2026.03 | 23 | |
| Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.12 | 22.2 | |
| DeepSeek-R1-Distill-14B2026.03 | 20.9 | |
| Qwen3-4B (Thinking)Thinking-mode sampling=true2026.03 | 20.9 | |
| AceCoder-7B2026.03 | 19.6 | |
| OpenCoder-8B-InstructSize Category=6B+ LLMs2025.12 | 18.2 | |
| Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.12 | 18.2 | |
| DeepCoder-14B2026.03 | 18.2 | |
| DS-Coder-33B-InstructSize Category=32B+ LLMs2025.12 | 17.6 | |
| UCoder-32BSize Category=32B+ LLMs2025.12 | 17.6 | |
| DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.12 | 16.2 | |
| UCoder-14BSize Category=13B+ LLMs2025.12 | 16.2 | |
| NonDPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=None2025.12 | 16.2 | |
| UCoder-7BSize Category=6B+ LLMs2025.12 | 15.5 | |
| Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.12 | 11.5 | |
| NonDPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=None2025.12 | 10.8 | |
| PrivCodeBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 10.8 | |
| DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.12 | 10.1 | |
| CodeLlama-13B-InstructSize Category=13B+ LLMs2025.12 | 9.5 | |
| NonDPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=None2025.12 | 9.5 | |
| PrivCodeBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 9.5 | |
| DP-AdapterBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 9.5 | |
| JFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 9.2 | |
| CodeLlama-34B-InstructSize Category=32B+ LLMs2025.12 | 8.8 | |
| DPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 8.8 | |
| NonDPFTBackbone=CodeGemma-7B, Privacy Epsilon=None2025.12 | 8.1 | |
| JFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 7.9 | |
| PrivCodeBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 7.4 | |
| DPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 5.4 | |
| PrivCodeBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 5.4 | |
| JFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 5.3 | |
| DPFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 5.1 | |
| DP-AdapterBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 4.8 | |
| JFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 4.8 | |
| DPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 4.1 | |
| DP-AdapterBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 3.9 | |
| DP-AdapterBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 3.8 | |
| CodeLlama-7B-InstructSize Category=6B+ LLMs2025.12 | 3.4 |