Data Science Code Completion on DS-1000
32Pandas (Pass@1)AlchemistCoder-DS
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| AlchemistCoder-DSParameters=6.7B, Base Model=DeepSeek-Coder2024.05 | 32 | 51.7 | 44.5 | 33.1 | 38.4 | 33.8 | 49.8 | 40.5 | |
| AlchemistCoder-CLParameters=7B, Base Model=CodeLlama2024.05 | 30.9 | 43.6 | 46.7 | 30.2 | 37.4 | 41.2 | 52.3 | 40.3 | |
| MagicoderS-DSParameters=6.7B, Base Model=DeepSeek-Coder2024.05 | 30.6 | 46.8 | 44.2 | 30.2 | 33 | 29.7 | 45.2 | 37.1 | |
| MagicoderS-CLParameters=7B, Base Model=CodeLlama2024.05 | 25.1 | 40.9 | 35.6 | 29.3 | 36.5 | 38.2 | 51 | 36.7 | |
| DeepSeek-Coder-BaseParameters=6.7B2024.05 | 21.3 | 35 | 26.7 | 23.6 | 34.8 | 25 | 34.8 | 28.7 | |
| CodeLlama-PythonParameters=7B2024.05 | 16.2 | 16.4 | 15.6 | 17.9 | 20 | 22.1 | 38.7 | 21 | |
| AlchemistCoder-LParameters=7B, Base Model=Llama 22024.05 | 13.4 | 22.7 | 31.1 | 11.3 | 25.2 | 8.8 | 29 | 20.2 | |
| CodeLlamaParameters=7B2024.05 | 12 | 27.7 | 17.8 | 13.2 | 12.2 | 20.6 | 15.5 | 17 | |
| Llama 2Parameters=7B2024.05 | 2.4 | 7.3 | 6.7 | 6.6 | 2.6 | 1.5 | 7.7 | 5 |