Code Generation on MBPP (Pass@1 (%))
91Pass@1Claude-3.5-Sonnet-20241022
Evaluation results
| Method | Links | |
|---|---|---|
| Claude-3.5-Sonnet-20241022Size Category=Closed-APIs2025.12 | 91 | |
| Qwen2.5-Coder-32B-InstructSize Category=32B+ LLMs2025.12 | 90.2 | |
| UCoder-32BSize Category=32B+ LLMs2025.12 | 89.7 | |
| DS-Coder-V2-InstructSize Category=32B+ LLMs2025.12 | 89.4 | |
| GPT-4o-2024-08-06Size Category=Closed-APIs2025.12 | 86.8 | |
| UCoder-14BSize Category=13B+ LLMs2025.12 | 86.5 | |
| Qwen2.5-Coder-14B-InstructSize Category=13B+ LLMs2025.12 | 86.2 | |
| UCoder-7BSize Category=6B+ LLMs2025.12 | 85.2 | |
| Qwen2.5-Coder-7B-InstructSize Category=6B+ LLMs2025.12 | 83.5 | |
| Qwen2.5-Coder-32BSize Category=32B+ LLMs2025.12 | 83 | |
| DS-Coder-V2-Lite-InstructSize Category=13B+ LLMs2025.12 | 82.8 | |
| Qwen2.5-Coder-14BSize Category=13B+ LLMs2025.12 | 81 | |
| DS-Coder-33B-InstructSize Category=32B+ LLMs2025.12 | 80.4 | |
| OpenCoder-8B-InstructSize Category=6B+ LLMs2025.12 | 79.1 | |
| Starcoder2-15B-Instruct-v0.1Size Category=13B+ LLMs2025.12 | 78 | |
| Qwen2.5-Coder-7BSize Category=6B+ LLMs2025.12 | 76.9 | |
| DS-Coder-6.7B-InstructSize Category=6B+ LLMs2025.12 | 74.9 | |
| CodeLlama-34B-InstructSize Category=32B+ LLMs2025.12 | 61.1 | |
| CodeLlama-13B-InstructSize Category=13B+ LLMs2025.12 | 60.3 | |
| CodeLlama-7B-InstructSize Category=6B+ LLMs2025.12 | 39.9 |