Repository-level code completion on CrossCodeEval
35.9Python EMStarCoder (Teacher)
Evaluation Results
| Method | Links | ||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|
| StarCoder (Teacher)Model Size=15.5B2024.07 | 35.9 | 66.1 | 41.5 | 72.9 | 38.7 | 73.7 | 56.3 | 79.3 | 43.1 | 73 | |
| DDKModel Size=3B, Distillation Strategy=DDK2024.07 | 31.7 | 62.2 | 34.6 | 69.8 | 33.2 | 69.3 | 50.9 | 76.2 | 37.6 | 69.4 | |
| StarCoder + KDModel Size=3B, Distillation Strategy=KD2024.07 | 26.5 | 53.2 | 32.4 | 61.1 | 31.6 | 64.5 | 48 | 69.8 | 34.6 | 61.2 | |
| StarCoder + CPTModel Size=3B, Distillation Strategy=CPT2024.07 | 24.8 | 49.3 | 31.6 | 61.5 | 30.5 | 63.7 | 47.1 | 68.4 | 33.5 | 60.7 | |
| StarCoder (Student)Model Size=3B2024.07 | 20.8 | 41.5 | 25.3 | 51.4 | 25.7 | 56.2 | 40.5 | 60.5 | 28.1 | 52.4 |