Code Generation on MBPP subset of 500 (test)
64.6Pass@1Magicoder-S-DS
Evaluation Results
| Method | Links | |
|---|---|---|
| Magicoder-S-DSParams=6.7B, Base Model=DeepseekCoder, InsT Data=185K, Decoding Strategy=greedy2023.12 | 64.6 | |
| WaveCoder-Ultra-6.7BParams=6.7B, Base Model=DeepseekCoder, InsT Data=130K, Decoding Strategy=greedy2023.12 | 64.4 | |
| WaveCoder-Pro-6.7BParams=6.7B, Base Model=DeepseekCoder, InsT Data=20K, Decoding Strategy=greedy2023.12 | 63.6 | |
| WaveCoder-DS-6.7BParams=6.7B, Base Model=DeepseekCoder, InsT Data=20K, Decoding Strategy=greedy2023.12 | 62.8 | |
| DeepseekCoder-instruct*Params=6.7B, Base Model=DeepseekCoder, Decoding Strategy=greedy2023.12 | 62.8 | |
| DeepseekCoderParams=6.7B, InsT Data=None, Decoding Strategy=greedy2023.12 | 60.6 | |
| Magicoder-DSParams=6.7B, Base Model=DeepseekCoder, InsT Data=75K, Decoding Strategy=greedy2023.12 | 60.4 | |
| ChatGPTDecoding Strategy=n=2002023.12 | 52.2 | |
| WizardCoderParams=15B, Base Model=StarCoder, InsT Data=78K, Decoding Strategy=n=2002023.12 | 51.8 | |
| WaveCoder-SC-15BParams=15B, Base Model=StarCoder, InsT Data=20K, Decoding Strategy=n=2002023.12 | 51 | |
| WaveCoder-CL-13BParams=13B, Base Model=CodeLLaMa, InsT Data=20K, Decoding Strategy=n=2002023.12 | 49.6 | |
| CodeLLaMa-instructParams=13B, Base Model=CodeLLaMa, InsT Data=14K, Decoding Strategy=n=2002023.12 | 49.4 | |
| WaveCoder-CL-7BParams=7B, Base Model=CodeLLaMa, InsT Data=20K, Decoding Strategy=n=2002023.12 | 47.2 | |
| CodeLLaMaParams=13B, InsT Data=None, Decoding Strategy=n=2002023.12 | 47 | |
| CodeLLaMa-instructParams=7B, Base Model=CodeLLaMa, InsT Data=14K, Decoding Strategy=n=2002023.12 | 44.4 | |
| OctoCoderParams=15B, Base Model=StarCoder, InsT Data=13K, Decoding Strategy=n=2002023.12 | 43.5 | |
| StarCoderParams=15B, InsT Data=None, Decoding Strategy=n=2002023.12 | 43.3 | |
| CodeLLaMaParams=7B, InsT Data=None, Decoding Strategy=n=2002023.12 | 41.4 |