Spatial Reasoning on SURDS (test)
0YawSpatialBot
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| SpatialBotsetting=zero-shot2025.12 | 0 | 0 | 12 | 0 | 0 | 0 | 2 | |
| SpatialRGPTsetting=zero-shot2025.12 | 1.3 | 0.55 | 10.59 | 1.95 | 0.86 | 7.35 | 3.77 | |
| LLaVA-OV-Qwen2-72B-sisetting=zero-shot2025.12 | 1.95 | 3.03 | 23.57 | 3.78 | 9.73 | 8.65 | 8.45 | |
| GPT-4o-minisetting=zero-shot2025.12 | 3.24 | 0.28 | 0.22 | 4.22 | 21.51 | 2.05 | 5.25 | |
| Randomsetting=zero-shot2025.12 | 5.73 | 1.12 | 34.27 | 8.76 | 11.57 | 11.89 | 12.22 | |
| Qwen2.5-VL-3Bsetting=zero-shot2025.12 | 6.27 | 3.81 | 27.68 | 17.84 | 14.81 | 10.49 | 13.48 | |
| Qwen2.5-VL-3B-InstructModel Scale=3B, Textual CoT=true2026.03 | 6.27 | 3.81 | 27.68 | 17.84 | 14.81 | 10.49 | 13.48 | |
| Qwen2.5VL-7Bsetting=zero-shot2025.12 | 7.57 | 3.46 | 25.95 | 11.46 | 17.95 | 9.3 | 12.61 | |
| Qwen2.5-VL-7B-InstructModel Scale=7B, Textual CoT=true2026.03 | 7.57 | 3.46 | 25.95 | 11.46 | 17.95 | 9.3 | 12.61 | |
| Gemini-2.0-flashsetting=zero-shot2025.12 | 9.3 | 5.41 | 32.97 | 69.3 | 77.3 | 20 | 35.71 | |
| OmniDrive-R1setting=zero-shot2025.12 | 9.35 | 39.46 | 36.72 | 46.25 | 46.51 | 13.42 | 31.95 | |
| Qwen2.5VL-72Bsetting=zero-shot2025.12 | 11.57 | 6.13 | 44 | 58.05 | 66.16 | 14.92 | 33.47 | |
| Qwen2.5-VL-72B-InstructModel Scale=72B, Textual CoT=true2026.03 | 11.57 | 6.13 | 44 | 58.05 | 66.16 | 14.92 | 33.47 | |
| GPT-4osetting=zero-shot2025.12 | 13.08 | 1.62 | 2.49 | 11.57 | 47.89 | 3.14 | 13.3 | |
| Gemini-1.5-prosetting=zero-shot2025.12 | 19.14 | 4.41 | 22.7 | 61.95 | 66.38 | 22.05 | 32.77 | |
| SURDS-3B*Training Protocol=SFT+RL, Model Scale=3B, Textual CoT=true2026.03 | 20.97 | 44.81 | 69.84 | 49.3 | 51.35 | 8.54 | 40.8 | |
| InternVL3-8B†Training Protocol=Only SFT, Model Scale=8B, Textual CoT=true2026.03 | 45.19 | 61.83 | 100 | 82.05 | 86.16 | 84.43 | 76.61 | |
| LaST-VLA-8B†Training Protocol=Only SFT, Model Scale=8B, Textual CoT=false2026.03 | 70.16 | 71.28 | 100 | 86.05 | 90.27 | 88 | 84.29 |