Natural Language Reasoning on HELM
54Synth. Reason. (AS)code-davinci-002
Evaluation Results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| code-davinci-002Size=175B, Open Access=false2023.05 | 54 | 68.4 | 68.6 | 80.5 | 56.8 | 41 | 43.3 | — | — | |
| text-davinci-003Size=175B, Open Access=false2023.05 | 50.2 | 73.4 | 65.3 | 75.1 | 50.6 | 39 | 44.9 | 23.3 | 62.2 | |
| StarCoderBaseSize=15.5B, Open Access=true2023.05 | 44 | 21 | 50.4 | 85.4 | 8.4 | 15.1 | 7 | 19 | 53.2 | |
| code-cushman-001Size=12B, Open Access=false2023.05 | 34.1 | 16.4 | 48.1 | 45.1 | 4.9 | 9.9 | 7.2 | — | — | |
| Luminous SupremeSize=70B, Open Access=false2023.05 | 31.2 | — | 50.4 | 72.9 | 11.2 | 14.9 | 5.7 | 21.2 | 53 | |
| BLOOMSize=176B, Open Access=true2023.05 | 30.4 | 19.7 | 44.7 | 54.5 | 9.5 | 4.3 | 5.5 | 20.9 | 54.3 | |
| J1-Grande v2 betaSize=17B, Open Access=false2023.05 | 28.6 | 13.9 | 47 | 61.7 | 9.6 | 12.7 | 6.8 | 19.1 | 56.2 | |
| J1-Jumbo v1Size=178B, Open Access=false2023.05 | 26.3 | 17.4 | 54.3 | 44.5 | 5.4 | 8.9 | 3.3 | 23.2 | 48.4 | |
| GLMSize=130B, Open Access=true2023.05 | 25.2 | 25.4 | 44.3 | 54.9 | 6.1 | 0 | 5.9 | 19.3 | 45.1 | |
| Cohere Command BetaSize=52.4B, Open Access=false2023.05 | 24.3 | 24.5 | 47.3 | 42.1 | 13.8 | 13.3 | 7.5 | 22.9 | 60.6 | |
| OPTSize=175B, Open Access=true2023.05 | 22.5 | 24.8 | 50.7 | 49.4 | 4 | 6.5 | 2.6 | 22 | 53.2 | |
| UL2Size=20B, Open Access=true2023.05 | 20.5 | 21.7 | 50.1 | 14 | 2.4 | 0 | 0 | 20.7 | 50.6 | |
| GPT-NeoXSize=20B, Open Access=true2023.05 | 20.4 | 16.7 | 46.8 | 74.7 | 5.3 | 14.1 | 7.1 | 19.1 | 51.5 | |
| T5Size=11B, Open Access=true2023.05 | 19.6 | 10.1 | 41.2 | 34.7 | 2.3 | 0 | 0 | 15.9 | 55.8 | |
| OPTSize=66B, Open Access=true2023.05 | 19.3 | 21.3 | 40.8 | 47.1 | 1.8 | 4.8 | 2.9 | 17.5 | 52.7 | |
| YaLMSize=100B, Open Access=true2023.05 | 5.6 | 6.1 | 34.6 | 63.3 | 0 | 0 | 0 | 2.3 | 48.4 |