Data science code generation on DS-1000
60.3Matplotlib ScoreStarCoder2-15B
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| StarCoder2-15BFormat=Completion2024.02 | 60.3 | 43.3 | 23.2 | 11 | 26.4 | 26 | 36 | 33.8 | |
| DeepSeekCoder-33BFormat=Completion2024.02 | 56.1 | 49.6 | 25.8 | 36.8 | 36.8 | 40 | 46.7 | 40.2 | |
| StarCoder2-7BFormat=Completion2024.02 | 53.6 | 33.3 | 16.9 | 16.2 | 20.6 | 22.2 | 31.9 | 27.8 | |
| DeepSeekCoder-6.7BFormat=Completion2024.02 | 52.4 | 33 | 20 | 13.9 | 19.8 | 29.7 | 27.4 | 28.9 | |
| CodeLlama-34BFormat=Completion2024.02 | 50.3 | 42.7 | 23 | 25 | 28.3 | 33.9 | 40 | 34.3 | |
| CodeLlama-13BFormat=Completion2024.02 | 49 | 27.2 | 17.4 | 12.9 | 15.6 | 24 | 24.8 | 25.1 | |
| StarCoderBase-15BFormat=Completion2024.02 | 47 | 27.1 | 10.1 | 19.5 | 21.7 | 27 | 20.5 | 23.8 | |
| CodeLlama-7BFormat=Completion2024.02 | 46.3 | 21.6 | 13.9 | 12.2 | 17.5 | 16.7 | 20.6 | 21.5 | |
| StarCoder2-3BFormat=Completion2024.02 | 45.5 | 27.7 | 16.2 | 12.9 | 15.8 | 30.8 | 22.8 | 25 | |
| StableCode-3BFormat=Completion2024.02 | 42.5 | 24.5 | 16.2 | 15.4 | 13.5 | 20.2 | 27.7 | 22.7 | |
| StarCoderBase-7BFormat=Completion2024.02 | 38 | 23 | 8.2 | 13.1 | 13.7 | 24.5 | 14.6 | 19.1 | |
| DeepSeekCoder-1.3BFormat=Completion2024.02 | 36.2 | 18.8 | 9.1 | 10.7 | 7.9 | 13.9 | 13.3 | 16.2 | |
| StarCoderBase-3BFormat=Completion2024.02 | 32.1 | 16.8 | 5.3 | 9.2 | 13.2 | 10.5 | 17.2 | 14.2 |