Multilingual Code Completion on Multipl-E (Pass@k)
31.14Pass@1IRCoder
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| IRCoderParameters=5.7B2024.03 | 31.14 | 45 | 51.29 | |
| DeepSeekCoderParameters=5.7B2024.03 | 30.47 | 41.38 | 48.04 | |
| IRCoderParameters=6.7B2024.03 | 24.06 | 39.38 | 47.03 | |
| CodeLlamaParameters=6.7B2024.03 | 21.83 | 34.78 | 42.5 | |
| IRCoderParameters=1.3B2024.03 | 20.51 | 31.14 | 37.9 | |
| IRCoderParameters=7.3B2024.03 | 18.46 | 30.43 | 38.04 | |
| DeepSeekCoderParameters=1.3B2024.03 | 18.34 | 26.12 | 31.36 | |
| StarCoderBaseParameters=7.3B2024.03 | 17.94 | 27.38 | 34.12 | |
| IRCoderParameters=3.1B2024.03 | 14.36 | 22.58 | 28.02 | |
| StarCoderBaseParameters=3.1B2024.03 | 12.78 | 19.09 | 22.62 | |
| IRCoderParameters=1.1B2024.03 | 8.76 | 14.51 | 19.32 | |
| StarCoderBaseParameters=1.1B2024.03 | 8.35 | 13.43 | 16.43 |