Code Generation on MBPP-S (test)
69.25Pass@1SRank
Evaluation Results
| Method | Links | |
|---|---|---|
| SRankModel=Codex002, Setting=zero-shot2023.10 | 69.25 | |
| CodeTModel=Codex002, Setting=zero-shot2023.10 | 67.7 | |
| Coder-ReviewerModel=Codex002, Setting=zero-shot2023.10 | 64.7 | |
| SRankModel=WizardCoder34B, Setting=zero-shot2023.10 | 64.14 | |
| CodeTModel=WizardCoder34B, Setting=zero-shot2023.10 | 63.39 | |
| GreedyModel=WizardCoder34B, Setting=zero-shot2023.10 | 60.42 | |
| SRankModel=WizardCoder15B, Setting=zero-shot2023.10 | 59.01 | |
| SRankModel=StarCoder, Setting=zero-shot2023.10 | 58.38 | |
| CodeTModel=WizardCoder15B, Setting=zero-shot2023.10 | 58.18 | |
| GreedyModel=Codex002, Setting=zero-shot2023.10 | 58.1 | |
| CodeTModel=StarCoder, Setting=zero-shot2023.10 | 58.05 | |
| SRankModel=CodeGen2.5-Instruct, Setting=zero-shot2023.10 | 57.02 | |
| CodeTModel=CodeGen2.5-Instruct, Setting=zero-shot2023.10 | 55.02 | |
| RandomModel=WizardCoder34B, Setting=zero-shot2023.10 | 54.37 | |
| Coder-ReviewerModel=CodeGen2.5-Instruct, Setting=zero-shot2023.10 | 52.74 | |
| Coder-ReviewerModel=WizardCoder15B, Setting=zero-shot2023.10 | 52.52 | |
| GreedyModel=WizardCoder15B, Setting=zero-shot2023.10 | 51.29 | |
| SRankModel=CodeGen16B, Setting=zero-shot2023.10 | 51.03 | |
| Coder-ReviewerModel=CodeGen16B, Setting=zero-shot2023.10 | 50.3 | |
| CodeTModel=CodeGen16B, Setting=zero-shot2023.10 | 49.5 | |
| Coder-ReviewerModel=StarCoder, Setting=zero-shot2023.10 | 49.48 | |
| RandomModel=Codex002, Setting=zero-shot2023.10 | 47.5 | |
| GreedyModel=StarCoder, Setting=zero-shot2023.10 | 45.9 | |
| RandomModel=WizardCoder15B, Setting=zero-shot2023.10 | 45.72 | |
| GreedyModel=CodeGen2.5-Instruct, Setting=zero-shot2023.10 | 42.86 | |
| GreedyModel=CodeGen16B, Setting=zero-shot2023.10 | 42.4 | |
| RandomModel=StarCoder, Setting=zero-shot2023.10 | 39.26 | |
| RandomModel=CodeGen2.5-Instruct, Setting=zero-shot2023.10 | 34.6 | |
| RandomModel=CodeGen16B, Setting=zero-shot2023.10 | 31.54 |