Code Generation on MBPP problems 11-510 (test)
50.4Pass@1Code-davinci-002
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Code-davinci-002Prompt Setting=3-shot2023.03 | 50.4 | — | 84.4 | |
| PaLMCoder-540BPrompt Setting=3-shot2023.03 | 47 | — | 80.8 | |
| PaLM-540BPrompt Setting=3-shot2023.03 | 36.8 | — | 75 | |
| CodeGeeX-13BPrompt Setting=1-shot2023.03 | 24.37 | 47.95 | 68.5 | |
| CodeGeeX-13BPrompt Setting=3-shot2023.03 | 22.44 | 43.24 | 63.52 | |
| InCoder-6.7BPrompt Setting=1-shot2023.03 | 19.4 | — | — | |
| LaMDA-137BPrompt Setting=3-shot2023.03 | 14.8 | — | 62.4 |