Code Bug Fixing on HumanEvalFix
48.6Python Fix RateStarCoder2-15B
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| StarCoder2-15BPrompt=Issue2024.02 | 48.6 | 41.6 | 48.4 | 48.5 | 20.7 | 24.2 | 38.7 | — | |
| DeepSeekCoder-33B-InstructPrompt=Instruct2024.02 | 47.5 | 47.6 | 46.5 | 52 | 48 | 10.2 | 42.1 | — | |
| DeepSeekCoder-6.7B-InstructPrompt=Instruct2024.02 | 44.9 | 55.3 | 52.2 | 42.9 | 37.9 | 19.5 | 42.1 | — | |
| CodeLlama-34B-InstructPrompt=Instruct2024.02 | 36.5 | 28.1 | 36.4 | 25.7 | 25.2 | 18.5 | 28.4 | — | |
| OctoCoder-15BPrompt=Instruct2024.02 | 30.4 | 28.4 | 30.6 | 30.2 | 26.1 | 16.5 | 27 | — | |
| StarCoderBase-15BPrompt=Commit2024.02 | 25.6 | 29.4 | 28.8 | 28.7 | 28.2 | 19.7 | 26.7 | — | |
| CodeLlama-13B-InstructPrompt=Instruct2024.02 | 19.4 | 18.9 | 24.1 | 21.6 | 10.1 | 0.4 | 15.8 | — | |
| StarCoderBase-15BPrompt=Instruct2024.02 | 12.6 | 16.8 | 18.9 | 12.5 | 11.2 | 0.6 | 12.1 | — | |
| StarCoder2-15BPrompt=Instruct2024.02 | 9.7 | 20.7 | 24.1 | 36.3 | 25.6 | 15.4 | 22 | — | |
| SWE-agentBackbone=Claude 4 Sonnet2025.12 | — | — | — | — | — | — | — | 87.7 |