Reasoning and Problem Solving on STREET English (test)
99.1ARC ScoreGPT-3
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-3Prompt Format=SIM code, Number of shots=52024.03 | 99.1 | 45.2 | 37.8 | 32 | |
| GPT-3Prompt Format=direct, Number of shots=52024.03 | 97.6 | 4.4 | 21.7 | 22 | |
| GPT-3Prompt Format=linearized, Number of shots=52024.03 | 93.2 | 28.9 | 25.6 | 18 | |
| falcon-40b-instructPrompt Format=SIM code, Number of shots=52024.03 | 81.5 | 19.6 | 24.4 | 34 | |
| falcon-40b-instructPrompt Format=linearized, Number of shots=52024.03 | 76.5 | 28.1 | 18.9 | 24 | |
| BLOOMZ-TCCPrompt Format=SIM code, Number of shots=52024.03 | 76.2 | 1.9 | 21.7 | 28 | |
| GPT-3Prompt Format=linearized, Number of shots=5, source=Ribeiro et al. (2022)2024.03 | 72.9 | 34.8 | 40.2 | 19 | |
| BLOOMZ-TCCPrompt Format=direct, Number of shots=52024.03 | 70.3 | 3.3 | 20.9 | 24 | |
| BLOOMZPrompt Format=SIM code, Number of shots=52024.03 | 61.5 | 0.7 | 23.6 | 26 | |
| BLOOMZPrompt Format=direct, Number of shots=52024.03 | 54.7 | 2.6 | 24.8 | 20 | |
| BLOOMZ-TCCPrompt Format=linearized, Number of shots=52024.03 | 40.9 | 1.5 | 18.5 | 20 | |
| BLOOMZPrompt Format=linearized, Number of shots=52024.03 | 35.3 | 1.1 | 24 | 22 | |
| falcon-40b-instructPrompt Format=direct, Number of shots=52024.03 | 34.7 | 8.5 | 24.8 | 26 |