Program Synthesis on APPS 1.0 (test)
25.61Pass@5 (Introductory)CodeRL+CodeT5
Evaluation Results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CodeRL+CodeT5Size=770M, k=10002022.07 | 25.61 | 17.17 | 6.78 | 4.88 | 8.48 | 9.53 | 8.91 | 12.62 | — | — | — | — | |
| CodexSize=12B, k=1000, Protocol=few-shot2022.07 | 24.52 | 22.78 | 2.64 | 3.04 | 6.75 | 3.23 | 3.08 | 7.46 | — | — | — | — | |
| AlphaCodeSize=1B, k=500002022.07 | 20.36 | — | — | — | — | 9.66 | 7.75 | 11.42 | — | — | — | — | |
| AlphaCodeSize=1B, k=100002022.07 | 18.18 | — | — | — | — | 8.21 | 6.65 | 9.89 | — | — | — | — | |
| CodeRL+CodeT5Size=770M2022.07 | 16.37 | 7.08 | 1.86 | 0.75 | 2.69 | 4.95 | 2.84 | 6.81 | 40 | 15.67 | 17.9 | 20.98 | |
| AlphaCodeSize=1B, k=10002022.07 | 14.36 | — | — | — | — | 5.63 | 4.58 | 7.17 | — | — | — | — | |
| CodexSize=12B, Protocol=few-shot2022.07 | 9.65 | 4.14 | 0.14 | 0.02 | 0.92 | 0.51 | 0.09 | 2.25 | 25.02 | 3.7 | 3.23 | 7.87 | |
| GPT-JSize=6B2022.07 | 9.2 | 5.6 | 1 | 0.5 | 1.82 | 1.73 | 1 | 3.08 | 35.2 | 13.15 | 13.51 | 17.63 | |
| GPT-NeoSize=2.7B2022.07 | 5.5 | 3.9 | 0.57 | 0 | 1.12 | 0.8 | 0 | 1.58 | 27.9 | 9.83 | 11.4 | 13.76 | |
| GPT2Size=1.5B2022.07 | 3.6 | 1.3 | 0.7 | 0 | 0.68 | 1.03 | 0 | 1.34 | 25 | 9.27 | 8.8 | 12.32 | |
| GPT2Size=0.1B2022.07 | 2.7 | 1 | 0.33 | 0 | 0.4 | 0.73 | 0 | 1.02 | — | — | — | — | |
| AlphaCodeSize=1B2022.07 | — | — | — | — | — | — | — | — | 17.67 | 5.24 | 7.06 | 8.09 | |
| GPT3Size=175B2022.07 | — | 0.2 | 0.03 | 0 | 0.06 | — | — | — | — | — | — | — |