Code Insertion on DS-1000 1.0 (test)
55.2Success Rate (Matplotlib)WizardCoder
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| WizardCoderFormat=Insertion, temperature=0.2, top_p=0.5, max_length=1024, n=40, Parameters=15B2023.06 | 55.2 | 35.1 | 20.4 | 30.4 | 28.9 | 32.3 | 37.8 | 32.8 | |
| StarCoderFormat=Insertion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 51.7 | 30.8 | 10.3 | 21 | 20.2 | 27.4 | 20 | 25.4 | |
| InCoder-6BFormat=Insertion, temperature=0.2, top_p=0.5, max_length=1024, n=402023.06 | 28.3 | 4.6 | 2.9 | 4.4 | 2.8 | 3.1 | 7.8 | 7.5 |