Code Generation on BabelCode
26.09C# Pass RatePaLM-Coder-540B
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| PaLM-Coder-540Bsampling=greedily, metric=pass@12023.05 | 26.09 | 24.22 | 21.12 | 1.86 | 25.47 | 29.81 | 4.35 | 24.84 | 25.47 | 26.71 | 22.98 | 30.43 | |
| PaLM 2-S*sampling=greedily, metric=pass@12023.05 | 24.22 | 34.16 | 19.25 | 8.7 | 31.06 | 32.3 | 16.77 | 26.09 | 26.09 | 34.16 | 28.57 | 32.3 | |
| PaLM 540Bsampling=greedily, metric=pass@12023.05 | 20.5 | 21.74 | 13.66 | 1.86 | 20.5 | 23.6 | 2.48 | 19.25 | 18.63 | 17.39 | 16.15 | 17.39 |