Spatial Reasoning on OmniSpatial 8.4K (test)
55.05Average AccuracyGemini-2.5-Pro
Evaluation Results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Gemini-2.5-ProRank=1, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 55.05 | 66.67 | 68.34 | 69.1 | 78.85 | 71.15 | 36.17 | 50.17 | 50.65 | 39.57 | 37.89 | |
| o3-04-16Rank=2, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 54.52 | 71.05 | 63.64 | 70.61 | 69.05 | 61.7 | 42.55 | 47.15 | 53.35 | 30.98 | 46.28 | |
| Gemini-2.5-FlashRank=3, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 51.8 | 54.84 | 70.43 | 65.33 | 66 | 68.75 | 39.53 | 39.46 | 52.7 | 33.49 | 36.84 | |
| GPT-4.1Rank=4, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 48.75 | 61.18 | 66.67 | 66.6 | 69.52 | 60.91 | 30.93 | 35.58 | 47.06 | 27.26 | 33.06 | |
| GPT-4oRank=5, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 44.47 | 60 | 57.97 | 62.38 | 55.24 | 59.09 | 29.9 | 31.01 | 42.06 | 24.34 | 28.05 | |
| Qwen-VL-2.5-32BRank=6, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 43.97 | 58.82 | 52.42 | 52.97 | 68.57 | 51.82 | 27.84 | 29.51 | 51.7 | 26.82 | 35.98 | |
| Gemini-2.0-FlashRank=7, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 42.41 | 56.47 | 56.76 | 55.95 | 63.81 | 57.27 | 15.46 | 30.04 | 43.2 | 24.56 | 28.05 | |
| Qwen-VL-2.5-3BRank=8, Evaluation Setting=Manual CoT + LLM evaluation2025.06 | 39.76 | 57.65 | 47.34 | 49.07 | 49.52 | 58.18 | 29.9 | 26.97 | 40.62 | 30.47 | 29.88 |