Spatial Reasoning on SPARTUN
83.1AccuracyPoT-LLM
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| PoT-LLMLLM=GPT-4o, Category=Prompting-based2024.12 | 83.1 | — | — | |
| Zero-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 81.6 | — | — | |
| CoTLLM=GPT-4o, Category=Prompting-based2024.12 | 81.4 | — | — | |
| Few-shotLLM=GPT-4o, Category=Prompting-based2024.12 | 80.5 | — | — | |
| CoTLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 79.7 | — | — | |
| CoT-SCLLM=GPT-4o, Category=Prompting-based2024.12 | 78.9 | — | — | |
| CoT-SCLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 78.1 | — | — | |
| Qwen3-32BSetting=PG (Pruned-grid)2026.06 | 77.07 | 82.4 | 70.4 | |
| Few-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 76.9 | — | — | |
| PoT-LLMLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 75.5 | — | — | |
| Qwen3-32BSetting=FG (Full-grid)2026.06 | 74.79 | 78.8 | 69.8 | |
| CoTLLM=Llama3-70B, Category=Prompting-based2024.12 | 73.8 | — | — | |
| Zero-shotLLM=GPT-4-turbo, Category=Prompting-based2024.12 | 72.8 | — | — | |
| PoT-LLMLLM=Llama3-70B, Category=Prompting-based2024.12 | 72.8 | — | — | |
| CoT-SCLLM=Llama3-70B, Category=Prompting-based2024.12 | 72.6 | — | — | |
| GPT-5.1Setting=PG (Pruned-grid)2026.06 | 71.68 | 81.37 | 60.23 | |
| GPT-5.1Setting=CoS2026.06 | 71.19 | 81.14 | 59.43 | |
| Qwen3-32BSetting=Rel. Trip.2026.06 | 70.67 | 77.4 | 62.4 | |
| Few-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 70 | — | — | |
| GPT-5.1Setting=Text2026.06 | 70 | 75.35 | 63.68 | |
| Qwen3-32BSetting=Text2026.06 | 68.85 | 85.3 | 48.8 | |
| GPT-5.1Setting=Rel. Trip.2026.06 | 68.14 | 79.08 | 55.21 | |
| GPT-5.1Setting=FG (Full-grid)2026.06 | 67.61 | 77.45 | 55.98 | |
| Zero-shotLLM=Llama3-70B, Category=Prompting-based2024.12 | 65 | — | — | |
| Qwen3-32BSetting=CoS2026.06 | 62.8 | 78.6 | 43.9 | |
| LLaMA3.1-70BSetting=FG (Full-grid)2026.06 | 60.21 | 76.27 | 40.98 | |
| LLaMA3.1-70BSetting=PG (Pruned-grid)2026.06 | 60.12 | 76.27 | 40.78 | |
| LLaMA3.1-70BSetting=CoS2026.06 | 59.59 | 75.41 | 40.31 | |
| LLaMA3.1-70BSetting=Text2026.06 | 55.76 | 75.45 | 32.5 | |
| LLaMA3.1-70BSetting=Rel. Trip.2026.06 | 55.32 | 74.8 | 32.3 | |
| PoT-LLMLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 52.7 | — | — | |
| CoT-SCLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 47.6 | — | — | |
| Zero-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 44.3 | — | — | |
| CoTLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 44.2 | — | — | |
| Few-shotLLM=GPT-3.5-turbo, Category=Prompting-based2024.12 | 35.1 | — | — |