Code Generation on ARCADE New Tasks
36Pass@5CODEX-davinci-002
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CODEX-davinci-0022022.12 | 36 | 43.8 | 50.8 | 54.8 | |
| PACHINCOModel Size=62B, Fine-tuned on=Notebooks2022.12 | 30.6 | 38 | 45 | 48.6 | |
| PACHINCONumber of Parameters=62B2022.12 | 30.6 | 38 | 45 | 48.6 | |
| PACHINCO (Ablation)Model Size=62B, Constraint=Without Schema Description2022.12 | 22.7 | 28.5 | 34.1 | 37.2 | |
| PALM + Python CodeModel Size=62B, Fine-tuned on=Python Code2022.12 | 21.4 | 28.4 | 35.6 | 39.8 | |
| CODEX-cushman-001Number of Parameters=12B2022.12 | 14.2 | 20.1 | 27 | 31.3 | |
| PALMModel Size=62B, Configuration=Base (1.3T Tokens)2022.12 | 12.5 | 17.4 | 22.8 | 26 | |
| CODEGENModel Size=16B, Training Mode=mono2022.12 | 12 | 17.1 | 22.7 | 26.3 | |
| CODEGENModel Size=6B, Training Mode=mono2022.12 | 8.9 | 13.1 | 18.3 | 21.6 | |
| INCODERModel Size=6B2022.12 | 7 | 10 | 13.5 | 15.8 | |
| CODEGENModel Size=2B, Training Mode=mono2022.12 | 6.5 | 9.8 | 13.8 | 16.3 | |
| CODEGENModel Size=16B, Training Mode=multi2022.12 | 4.6 | 6.8 | 9.5 | 11.3 | |
| INCODERModel Size=1B2022.12 | 3.8 | 5.8 | 8.4 | 10 | |
| CODEGENModel Size=6B, Training Mode=multi2022.12 | 3 | 4.6 | 6.7 | 8 | |
| CODEGENModel Size=2B, Training Mode=multi2022.12 | 2.7 | 4 | 5.5 | 6.5 | |
| CODEGENModel Size=350M, Training Mode=mono2022.12 | 1.9 | 2.8 | 4 | 4.9 | |
| CODEGENModel Size=350M, Training Mode=multi2022.12 | 1 | 1.6 | 2.5 | 3.1 |