Instructional code editing on CanItEdit Lazy Instructions
43.89Pass@1DeepSeekCoder-Instruct-33B
Evaluation Results
| Method | Links | |
|---|---|---|
| DeepSeekCoder-Instruct-33BFormat=Instruct2024.02 | 43.89 | |
| StarCoder2-15BFormat=Issue2024.02 | 38.45 | |
| DeepSeekCoder-Instruct-6.7BFormat=Instruct2024.02 | 33.61 | |
| CodeLlama-Instruct-7BFormat=Instruct2024.02 | 27.04 | |
| CodeLlama-Instruct-34BFormat=Instruct2024.02 | 26.76 | |
| StarCoderBase-15BFormat=Commit2024.02 | 26.38 | |
| StarCoderBase-7BFormat=Commit2024.02 | 25.83 | |
| OctoCoder-15BFormat=Instruct2024.02 | 25.69 | |
| CodeLlama-Instruct-13BFormat=Instruct2024.02 | 20.19 | |
| StarCoder2-7BFormat=Issue2024.02 | 18.55 | |
| DeepSeekCoder-Instruct-1.3BFormat=Instruct2024.02 | 18.33 | |
| StarCoder2-3BFormat=Issue2024.02 | 15.91 | |
| StarCoderBase-3BFormat=Commit2024.02 | 12.78 |