Single line code infilling on MultiPL-E
74.5Python SPM Exact MatchCODE LLAMA
Evaluation results
| Method | Links | ||||||
|---|---|---|---|---|---|---|---|
| CODE LLAMASize=13B, decoding=greedy2023.08 | 74.5 | 68.3 | 77.6 | 80 | 80.7 | 85 | |
| CODE LLAMASize=7B, decoding=greedy2023.08 | 72.7 | 67.6 | 74.3 | 77.6 | 80.2 | 82.6 | |
| StarCoderSize=15.5B, decoding=greedy2023.08 | 62 | — | — | 73 | — | 74 | |
| SantaCoderSize=1.1B, decoding=greedy2023.08 | 44 | — | — | 62 | — | 60 | |
| InCoderSize=6B, decoding=greedy2023.08 | 31 | — | — | 49 | — | 51 |