Program Synthesis on APPS
25.61Pass@5 (Introductory)CodeRL+CodeT5
Evaluation Results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| CodeRL+CodeT5Size=770M, k=10002022.07 | 25.61 | 17.17 | 6.78 | 4.88 | 8.48 | 9.53 | 8.91 | 12.62 | |
| CodexSize=12B, k=10002022.07 | 24.52 | 22.78 | 2.64 | 3.04 | 6.75 | 3.23 | 3.08 | 7.46 | |
| AlphaCodeSize=1B, k=500002022.07 | 20.36 | — | — | — | — | 9.66 | 7.75 | 11.42 | |
| AlphaCodeSize=1B, k=100002022.07 | 18.18 | — | — | — | — | 8.21 | 6.65 | 9.89 | |
| AlphaCodeSize=1B, k=10002022.07 | 14.36 | — | — | — | — | 5.63 | 4.58 | 7.17 |