Spatial Reasoning on SpatialRGPT-Bench qualitative 1.0 (val test)
99.17Below/Above AccuracySpatialRGPT-7B
Evaluation results
| Method | Links | |||||||
|---|---|---|---|---|---|---|---|---|
| SpatialRGPT-7BInput Modality=RGB2024.06 | 99.17 | 99.04 | 79.24 | 89.28 | 83.65 | 87.27 | 89.8 | |
| SpatialRGPT-7BInput Modality=3D2024.06 | 99.17 | 99.04 | 80.19 | 91.96 | 87.5 | 91.81 | 91.78 | |
| SpatialRGPT-VILA-1.5-3B2024.06 | 99.17 | 100 | 81.13 | 88.39 | 85.57 | 93.63 | 91.47 | |
| SpatialRGPT-VILA-1.5-8B2024.06 | 99.17 | 100 | 84.9 | 89.28 | 91.34 | 90.9 | 92.69 | |
| GPT-4VModel Category=VLM, Visual Prompting=Set-of-Mark (SoM)2024.06 | 75 | 55.23 | 42.45 | 54.46 | 49.03 | 47.27 | 54.33 | |
| GPT-4Model Category=Blind LLM2024.06 | 64.16 | 42.85 | 42.85 | 61.6 | 61.6 | 49.09 | 57.83 | |
| GPT-4VModel Category=VLM2024.06 | 63.34 | 46.67 | 64.15 | 60.71 | 68.26 | 45.45 | 58.14 | |
| LLaVA-v1.6-34BModel Category=VLM2024.06 | 44.16 | 45.71 | 36.79 | 53.57 | 37.5 | 45.45 | 43.98 | |
| LLaVA-v1.6-34BModel Category=VLM, Visual Prompting=Set-of-Mark (SoM)2024.06 | 44.16 | 40.01 | 33.96 | 47.32 | 41.34 | 46.36 | 42.31 | |
| RegionVILA-7BModel Category=Region-aware VLM2024.06 | 30.83 | 47.61 | 35.84 | 44.64 | 35.57 | 49.09 | 40.48 | |
| KOSMOS-2Model Category=Region-aware VLM2024.06 | 28.33 | 15.23 | 4.71 | 26.78 | 12.5 | 12.72 | 17.04 |