Instructional Code Editing on CanItEdit Descriptive Instructions (Pass@1)
53.06Pass@1DeepSeekCoder-Instruct-33B
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeekCoder-Instruct-33BFormat=Instruct2024.02 | 53.06 | |
| StarCoder2-15BFormat=Issue2024.02 | 43.08 | |
| StarCoderBase-7BFormat=Commit2024.02 | 40.64 | |
| StarCoderBase-15BFormat=Commit2024.02 | 38.24 | |
| StarCoder2-7BFormat=Issue2024.02 | 35.23 | |
| CodeLlama-Instruct-34BFormat=Instruct2024.02 | 35 | |
| CodeLlama-Instruct-7BFormat=Instruct2024.02 | 33.89 | |
| DeepSeekCoder-Instruct-6.7BFormat=Instruct2024.02 | 33.89 | |
| OctoCoder-15BFormat=Instruct2024.02 | 31.46 | |
| CodeLlama-Instruct-13BFormat=Instruct2024.02 | 28.33 | |
| DeepSeekCoder-Instruct-1.3BFormat=Instruct2024.02 | 25.83 | |
| StarCoder2-3BFormat=Issue2024.02 | 21.68 | |
| StarCoderBase-3BFormat=Commit2024.02 | 19.62 |