Family relationship reasoning on Alice Problem
89.5AccuracyDID
Evaluation results
| Method | Links | |
|---|---|---|
| DIDBase Model=Claude 3.5 Sonnet2024.10 | 89.5 | |
| ToTBase Model=Claude 3.5 Sonnet2024.10 | 87.1 | |
| CoTBase Model=Claude 3.5 Sonnet2024.10 | 83.7 | |
| IOBase Model=Claude 3.5 Sonnet2024.10 | 74.8 | |
| DIDBase Model=GPT-4o2024.10 | 70.3 | |
| ToTBase Model=GPT-4o2024.10 | 62.2 | |
| CoTBase Model=GPT-4o2024.10 | 55.9 | |
| IOBase Model=GPT-4o2024.10 | 43.4 | |
| DIDBase Model=GPT-3.5 Turbo2024.10 | 13.3 | |
| CoTBase Model=GPT-3.5 Turbo2024.10 | 8.6 | |
| ToTBase Model=GPT-3.5 Turbo2024.10 | 7.2 | |
| IOBase Model=GPT-3.5 Turbo2024.10 | 6.7 |