Code Generation on ODEX
46.5English Overall Pass@1StarCoderBase
Evaluation Results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| StarCoderBase2023.05 | 46.5 | 40.7 | 53 | 30.1 | 25.4 | 35.5 | 41.2 | 37.6 | 49.2 | 46.1 | 34 | 56.1 | |
| StarCoder2023.05 | 44.7 | 37 | 53.1 | 37.6 | 32.9 | 42.9 | 44.2 | 39.6 | 54.5 | 50.4 | 33.8 | 64.1 | |
| SantaCoder2023.05 | 37.7 | 30.9 | 45.1 | 32.1 | 26 | 39.1 | 28.1 | 23 | 39.4 | 36.9 | 23 | 48.3 | |
| CodeGen-16B-Mono2023.05 | 33.7 | 25.2 | 43.1 | 30 | 25 | 36.1 | 37.8 | 26.6 | 62.8 | 46.8 | 30.4 | 60.1 | |
| code-davinci-0012023.05 | 33.6 | 26.9 | 41 | 36.9 | 31.7 | 42.9 | 31 | 23.7 | 47.3 | 43.2 | 28.9 | 55.1 | |
| code-cushman-0012023.05 | 31.9 | 24.4 | 40.2 | 31.9 | 27.7 | 36.7 | 25.7 | 21.2 | 35.5 | 40 | 26 | 51.6 |