Fill-in-the-middle on FIM-task Python v1.0 (test)
0.745Exact MatchCodeLlama-13B
Evaluation results
| Method | Links | |
|---|---|---|
| CodeLlama-13BModel Size=13B2024.02 | 0.745 | |
| StarCoder Base-15BModel Size=15B2024.02 | 0.62 | |
| StarCoder2-7BModel Size=7B2024.02 | 0.611 | |
| StableCode-3BModel Size=3B2024.02 | 0.591 | |
| StarCoder2-3BModel Size=3B2024.02 | 0.591 | |
| StarCoder2-15BModel Size=15B, Note=Implementation bug during training affected FIM performance2024.02 | 0.484 |