Code Completion on DS-1000 1.0 (test)
55.2Success Rate (Matplotlib)WizardCoder
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| WizardCoderFormat=Completion, temperature=0.2, top_p=0.5, max_length=1024, n=40, Parameters=15B2023.06 | 55.2 | 33.6 | 16.7 | 26.2 | 24.2 | 24.9 | 26.7 | 29.2 | |
| StarCoderFormat=Completion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 51.7 | 29.7 | 11.4 | 21.4 | 20.2 | 29.5 | 24.5 | 26 | |
| Code-Cushman-001Format=Completion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 40.7 | 21.8 | 7.9 | 12.4 | 11.3 | 18 | 12.2 | 18.1 | |
| CodeGen-monoFormat=Completion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 31.7 | 10.9 | 3.4 | 7 | 9 | 10.8 | 15.2 | 11.7 | |
| InCoder-6BFormat=Completion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 28.3 | 4.4 | 3.1 | 4.4 | 2.8 | 2.8 | 3.8 | 7.4 |