Reasoning on Python Puzzles
57.3AccuracyRoT
Evaluation Results
| Method | Links | |
|---|---|---|
| RoTBackbone=GPT-4, Shots=2-shot2024.10 | 57.3 | |
| RoTBackbone=GPT-4, Shots=1-shot2024.10 | 54.6 | |
| BOTBackbone=GPT-4, Shots=2-shot2024.10 | 52.8 | |
| BOTBackbone=GPT-4, Shots=1-shot2024.10 | 51.8 | |
| RoTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 50.4 | |
| RoTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 50.2 | |
| BOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 48.2 | |
| BOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 48 | |
| Meta-promptingBackbone=GPT-4, Shots=2-shot2024.10 | 45.6 | |
| TOTBackbone=GPT-4, Shots=2-shot2024.10 | 45.2 | |
| Meta-promptingBackbone=GPT-4, Shots=1-shot2024.10 | 45 | |
| TOTBackbone=GPT-4, Shots=1-shot2024.10 | 43.8 | |
| GOTBackbone=GPT-4, Shots=2-shot2024.10 | 43.5 | |
| GOTBackbone=GPT-4, Shots=1-shot2024.10 | 42.6 | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 41.1 | |
| Meta-promptingBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 40.2 | |
| TOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 39.5 | |
| TOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 38.2 | |
| GOTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 36.7 | |
| COTBackbone=GPT-4, Shots=2-shot2024.10 | 35.9 | |
| GOTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 35.3 | |
| COTBackbone=GPT-4, Shots=1-shot2024.10 | 35.3 | |
| COTBackbone=GPT-3.5-turbo, Shots=2-shot2024.10 | 30.6 | |
| COTBackbone=GPT-3.5-turbo, Shots=1-shot2024.10 | 30.1 |