Reasoning and Problem Solving on xSTREET average 5 languages (test)
94.2ARC ScoreGPT-3
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| GPT-3Prompt Format=SIM code, Number of shots=52024.03 | 94.2 | 26.9 | 32.8 | 24.8 | |
| GPT-3Prompt Format=direct, Number of shots=52024.03 | 90.7 | 1.6 | 25.6 | 24.4 | |
| GPT-3Prompt Format=linearized, Number of shots=52024.03 | 72.7 | 24.4 | 27.7 | 23.2 | |
| BLOOMZ-TCCPrompt Format=SIM code, Number of shots=52024.03 | 61.1 | 1.3 | 25.4 | 24.8 | |
| falcon-40b-instructPrompt Format=linearized, Number of shots=52024.03 | 54.6 | 10.6 | 23.4 | 24.4 | |
| BLOOMZPrompt Format=SIM code, Number of shots=52024.03 | 52.6 | 1.5 | 23.3 | 24 | |
| BLOOMZ-TCCPrompt Format=direct, Number of shots=52024.03 | 49 | 1.8 | 21.9 | 23.2 | |
| falcon-40b-instructPrompt Format=SIM code, Number of shots=52024.03 | 43.9 | 4.6 | 23.5 | 26.8 | |
| BLOOMZPrompt Format=direct, Number of shots=52024.03 | 41.7 | 1.7 | 21.1 | 20.8 | |
| BLOOMZ-TCCPrompt Format=linearized, Number of shots=52024.03 | 36.9 | 1 | 22.5 | 28.5 | |
| BLOOMZPrompt Format=linearized, Number of shots=52024.03 | 33.5 | 0.9 | 24.4 | 24 | |
| falcon-40b-instructPrompt Format=direct, Number of shots=52024.03 | 32 | 6.1 | 24.6 | 24 |