Code Generation on MultiPL-E 2022 (test)
44.9JavaWizardCoder
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| WizardCoderParams=34B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 44.9 | 55.3 | 47.2 | 47.2 | 39.8 | 41.5 | 44.3 | 46.2 | |
| CodeLlama-InstructParams=34B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 41.5 | 45.9 | 41.5 | 37 | 24.3 | 32.7 | 37.6 | 39.3 | |
| CodeLlamaParams=34B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 40.2 | 41.7 | 41.4 | 40.4 | 22.7 | 31.4 | 35.3 | 38.7 | |
| CodeLlama-PythonParams=34B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 39.5 | 44.7 | 39.1 | 39.8 | 22.4 | 31.4 | 34.3 | 39.7 | |
| WizardCoderParams=15B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 35.8 | 41.9 | 39 | 39.3 | 33.5 | 34 | 33.7 | 27.1 | |
| Code-Cushman-001temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 31.9 | 31.3 | 30.6 | 29 | 11 | 1.5 | 22.1 | 25.2 | |
| StarCoderParams=15B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 30.2 | 30.8 | 31.6 | 26.1 | 15.5 | 23 | 22.7 | 21.8 | |
| StarCoderBaseParams=15B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 28.5 | 31.7 | 30.6 | 26.8 | 10.2 | 21.1 | 16.7 | 24.5 | |
| CodeGen-MultiParams=16B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 22.2 | 19.2 | 21 | 8.4 | 6.5 | 0 | 1.3 | 4.2 | |
| CodeGeeXParams=13B, temperature=0.2, top_p=0.95, max_length=512, n=502023.06 | 19.1 | 16.9 | 16.9 | 13.5 | 3.9 | 0.3 | 7.3 | 7.9 |