Code Generation on BigCodeBench-Instruct (Full)
0.504Pass@1o1
Evaluation results
| Method | Links | |
|---|---|---|
| o12026.03 | 0.504 | |
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 0.501 | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 0.496 | |
| DeepSeek-V2.5-238B2026.03 | 0.489 | |
| Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.12 | 0.484 | |
| DS-Coder-V2-InstructSize Category=32B+ LLMs2025.12 | 0.482 | |
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 0.468 | |
| UCoder-32BSize Category=32B+ LLMs2025.12 | 0.457 | |
| EVOLVECODER-4B (r3)Refinement round=r3, Backbone=Qwen3-4B (Thinking)2026.03 | 0.434 | |
| AceCoder-7B2026.03 | 0.433 | |
| OpenCoder-8B-InstructSize Category=6B+ LLMs2025.12 | 0.432 | |
| Critique-Coder-4B2026.03 | 0.431 | |
| EVOLVECODER-4B (r2)Refinement round=r2, Backbone=Qwen3-4B (Thinking)2026.03 | 0.428 | |
| EVOLVECODER-4B (r1)Refinement round=r1, Backbone=Qwen3-4B (Thinking)2026.03 | 0.424 | |
| DS-Coder-33B-InstructSize Category=32B+ LLMs2025.12 | 0.42 | |
| Qwen3-4B (Thinking)Thinking-mode sampling=true2026.03 | 0.42 | |
| EVOLVECODER-4B (r0)Refinement round=r0, Backbone=Qwen3-4B (Thinking)2026.03 | 0.412 | |
| UCoder-7BSize Category=6B+ LLMs2025.12 | 0.411 | |
| Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.12 | 0.41 | |
| UCoder-14BSize Category=13B+ LLMs2025.12 | 0.409 | |
| NonDPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=None2025.12 | 0.39 | |
| DeepCoder-14B2026.03 | 0.382 | |
| DeepSeek-R1-Distill-14B2026.03 | 0.381 | |
| Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.12 | 0.372 | |
| DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.12 | 0.368 | |
| DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.12 | 0.355 | |
| NonDPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=None2025.12 | 0.345 | |
| NonDPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=None2025.12 | 0.337 | |
| PrivCodeBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 0.296 | |
| PrivCodeBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 0.291 | |
| CodeLlama-34B-InstructSize Category=32B+ LLMs2025.12 | 0.29 | |
| JFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 0.288 | |
| CodeLlama-13B-InstructSize Category=13B+ LLMs2025.12 | 0.285 | |
| DPFTBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 0.284 | |
| DP-AdapterBackbone=DS-Coder-6.7B, Privacy Epsilon=42025.12 | 0.279 | |
| DP-AdapterBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 0.276 | |
| NonDPFTBackbone=CodeGemma-7B, Privacy Epsilon=None2025.12 | 0.274 | |
| DPFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 0.272 | |
| JFTBackbone=CodeQwen1.5-7B, Privacy Epsilon=42025.12 | 0.269 | |
| JFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 0.249 | |
| PrivCodeBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 0.229 | |
| PrivCodeBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 0.229 | |
| DPFTBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 0.226 | |
| DPFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 0.221 | |
| CodeLlama-7B-InstructSize Category=6B+ LLMs2025.12 | 0.219 | |
| JFTBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 0.219 | |
| DP-AdapterBackbone=CodeGemma-7B, Privacy Epsilon=42025.12 | 0.203 | |
| DP-AdapterBackbone=Qwen2.5-Coder-7B, Privacy Epsilon=42025.12 | 0.196 |