Code Generation on BigCodeBench-Completion Hard
36.5pass@1GPT-4o-2024-08-06
Evaluation results
| Method | Links | |
|---|---|---|
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 36.5 | |
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 35.1 | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 33.8 | |
| DeepSeek-Coder-V2-InstructSize=21B/236B2026.01 | 33.1 | |
| Stable-DiffCoder-8B-InstructSize=8B2026.01 | 31.8 | |
| Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.12 | 29.7 | |
| DS-Coder-V2-InstructSize Category=32B+ LLMs2025.12 | 29.7 | |
| UCoder-32BSize Category=32B+ LLMs2025.12 | 27.7 | |
| Seed-Coder-8B-InstructSize=8B2026.01 | 26.4 | |
| Qwen2.5-Coder-32BSize Category=32B+ LLMs2025.12 | 26.4 | |
| Codestral-22BSize=22B2026.01 | 24.3 | |
| UCoder-14BSize Category=13B+ LLMs2025.12 | 24.3 | |
| Qwen3-8BSize=8B2026.01 | 23 | |
| UCoder-7BSize Category=6B+ LLMs2025.12 | 22.3 | |
| Qwen2.5-Coder-14BSize Category=13B+ LLMs2025.12 | 22.3 | |
| DeepSeek-Coder-33B-InstructSize=33B2026.01 | 20.9 | |
| Qwen2.5-Coder-32B-InstructSize=32B2026.01 | 20.9 | |
| DS-Coder-33B-InstructSize Category=32B+ LLMs2025.12 | 20.9 | |
| Qwen2.5-Coder-7B-InstructSize=7B2026.01 | 20.3 | |
| Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.12 | 20.3 | |
| OpenCoder-8B-InstructSize=8B2026.01 | 18.9 | |
| OpenCoder-8B-InstructSize Category=6B+ LLMs2025.12 | 18.9 | |
| DeepSeek-Coder-V2-Lite-InstructSize=2.4B/16B2026.01 | 18.2 | |
| DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.12 | 18.2 | |
| Yi-Coder-9B-ChatSize=9B2026.01 | 17.6 | |
| Qwen2.5-Coder-14B-InstructSize=14B2026.01 | 16.2 | |
| Qwen2.5-Coder-7BSize Category=6B+ LLMs2025.12 | 16.2 | |
| DeepSeek-Coder-6.7B-InstructSize=6.7B2026.01 | 15.5 | |
| CodeQwen1.5-7B-ChatSize=7B2026.01 | 15.5 | |
| CodeLlama-70B-InstructSize=70B2026.01 | 15.5 | |
| DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.12 | 15.5 | |
| StarCoder2-15B-InstructSize=15B2026.01 | 14.9 | |
| Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.12 | 14.9 | |
| Llama-3.1-8B-InstructSize=8B2026.01 | 13.5 | |
| CodeLlama-34B-InstructSize Category=32B+ LLMs2025.12 | 10.8 | |
| CodeLlama-13B-InstructSize Category=13B+ LLMs2025.12 | 6.8 | |
| CodeLlama-7B-InstructSize=7B2026.01 | 4.1 | |
| CodeLlama-7B-InstructSize Category=6B+ LLMs2025.12 | 4.1 |