Instruction Following on HumanEvalFixDocs
49.79pass@1IRCoder
Evaluation results
| Method | Links | ||
|---|---|---|---|
| IRCoderParameters=5.7B2024.03 | 49.79 | 66.35 | |
| DeepSeekCoderParameters=5.7B2024.03 | 48.21 | 61.05 | |
| IRCoderParameters=6.7B2024.03 | 46.59 | 58.74 | |
| CodeLlamaParameters=6.7B2024.03 | 44.5 | 56.79 | |
| IRCoderParameters=7.3B2024.03 | 44.07 | 57.38 | |
| StarCoderBaseParameters=7.3B2024.03 | 40.74 | 55.27 | |
| IRCoderParameters=3.1B2024.03 | 28.99 | 42.76 | |
| StarCoderBaseParameters=3.1B2024.03 | 28.44 | 42.91 | |
| IRCoderParameters=1.3B2024.03 | 27.07 | 37.95 | |
| DeepSeekCoderParameters=1.3B2024.03 | 25.48 | 36.22 | |
| IRCoderParameters=1.1B2024.03 | 13.01 | 18 | |
| StarCoderBaseParameters=1.1B2024.03 | 12.23 | 17.7 |