Code Completion on DS-1000 (test)
56.1Matplotlib Success RateDeepSeek-Coder-Base
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| DeepSeek-Coder-BaseSize=33B2024.01 | 56.1 | 49.6 | 25.8 | 36.8 | 36.8 | 40 | 46.7 | 40.2 | |
| CodeLlama-BaseSize=34B2024.01 | 50.3 | 42.7 | 23 | 25 | 28.3 | 33.9 | 40 | 34.3 | |
| DeepSeek-Coder-BaseSize=6.7B2024.01 | 48.4 | 35.5 | 20.6 | 19.1 | 22.6 | 38.3 | 24.4 | 30.5 | |
| CodeLlama-BaseSize=13B2024.01 | 46.5 | 28.6 | 18.2 | 19.1 | 18.9 | 27.8 | 33.3 | 26.8 | |
| StarCoder-BaseSize=16B2024.01 | 43.2 | 29.1 | 11 | 20.6 | 23.6 | 32.2 | 15.6 | 24.6 | |
| CodeLlama-BaseSize=7B2024.01 | 41.9 | 24.6 | 14.8 | 16.2 | 18.9 | 17.4 | 17.8 | 22.1 | |
| CodeGeeX2Size=6B2024.01 | 38.7 | 26.8 | 14.4 | 11.8 | 19.8 | 27 | 17.8 | 22.9 | |
| DeepSeek-Coder-BaseSize=1.3B2024.01 | 32.3 | 21.4 | 9.3 | 8.8 | 8.5 | 16.5 | 8.9 | 16.2 |