Code Generation on ARCADE Existing Tasks
66.1pass@5CODEX-davinci-002
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| CODEX-davinci-0022022.12 | 66.1 | 72.6 | 78.3 | 81.2 | |
| PACHINCOModel Size=62B, Fine-tuned on=Notebooks2022.12 | 64.6 | 71 | 76 | 78.3 | |
| PACHINCONumber of Parameters=62B2022.12 | 64.6 | 71 | 76 | 78.3 | |
| PACHINCO (Ablation)Model Size=62B, Constraint=Without Schema Description2022.12 | 60.5 | 67.1 | 72.8 | 75.5 | |
| PALM + Python CodeModel Size=62B, Fine-tuned on=Python Code2022.12 | 58.8 | 65.7 | 71.7 | 74.7 | |
| CODEX-cushman-001Number of Parameters=12B2022.12 | 51 | 59 | 65.9 | 69.4 | |
| PALMModel Size=62B, Configuration=Base (1.3T Tokens)2022.12 | 49.7 | 57.5 | 64.4 | 67.8 | |
| CODEGENModel Size=16B, Training Mode=mono2022.12 | 46.7 | 54.5 | 60.9 | 64 | |
| CODEGENModel Size=6B, Training Mode=mono2022.12 | 42.1 | 49.4 | 56.3 | 59.7 | |
| INCODERModel Size=6B2022.12 | 41.3 | 48 | 54 | 57.2 | |
| CODEGENModel Size=2B, Training Mode=mono2022.12 | 35.8 | 43.3 | 50.7 | 54.9 | |
| CODEGENModel Size=16B, Training Mode=multi2022.12 | 31.2 | 37.6 | 43.8 | 47.2 | |
| INCODERModel Size=1B2022.12 | 30.1 | 36.4 | 42.6 | 46 | |
| CODEGENModel Size=6B, Training Mode=multi2022.12 | 28 | 34 | 39.9 | 43.3 | |
| CODEGENModel Size=2B, Training Mode=multi2022.12 | 25 | 29.9 | 34.8 | 37.7 | |
| CODEGENModel Size=350M, Training Mode=mono2022.12 | 18.9 | 24.3 | 30.1 | 33.4 | |
| CODEGENModel Size=350M, Training Mode=multi2022.12 | 13.3 | 16 | 18.8 | 20.4 |