Spatio-Temporal Reasoning on V-STaR
27.5Chain1 (When) m tIoUDEViL
Evaluation results
| Method | Links | |||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| DEViL2025.12 | 27.5 | 34.3 | 19.7 | 40.3 | 37.2 | 44.5 | 41.7 | 16.7 | 25 | 33.2 | 40.7 | |
| Open-o3-Video-32BModel size=32B2025.10 | 26.4 | 64.6 | — | — | 33.6 | — | 11.9 | — | 26 | 37.9 | 53.7 | |
| Ours (4B)Frames=642026.03 | 25.2 | — | — | — | 24.8 | — | 7 | — | 23.4 | — | — | |
| Open-o3-Video-4BModel size=4B2025.10 | 25.1 | 59.7 | — | — | 31.2 | — | 8.1 | — | 23.8 | 34.6 | 47.3 | |
| Open-o3-Video-8BModel size=8B2025.10 | 24.9 | 61.1 | — | — | 33.2 | — | 8.8 | — | 23.5 | 35.5 | 49 | |
| Gemini-2-Flash2025.12 | 24.5 | 53 | 15.9 | 0.9 | 4.6 | 0.6 | 2.2 | 15.2 | 23.8 | 26.9 | 35.6 | |
| Gemini-2-Flash2026.03 | 24.5 | — | — | — | 4.6 | — | 2.2 | — | 23.8 | — | — | |
| Open-o3 Video2026.03 | 24.5 | — | — | — | 25.4 | — | 6 | — | 24 | — | — | |
| Gemini-2-Flash2025.10 | 24.5 | 53 | — | — | 4.6 | — | 2.2 | — | 23.8 | 26.9 | 35.6 | |
| Open-o3-Video-7BNumber of sampled frames=162025.10 | 24.5 | 61 | — | — | 25.4 | — | 6 | — | 24 | 33.7 | 46.6 | |
| Qwen3-VL-32BModel size=32B2025.10 | 24.5 | 47.5 | — | — | 30.9 | — | 7.5 | — | 25.7 | 33.9 | 45.6 | |
| Qwen3-VL-8BModel size=8B2025.10 | 23.5 | 42.9 | — | — | 27.3 | — | 7.1 | — | 24.4 | 28 | 34.4 | |
| Video-LLaMA32025.12 | 23 | 41.9 | 19.8 | 0.1 | 0.9 | 0 | 0.2 | 20.4 | 23.1 | 21.7 | 27 | |
| Video-LLaMA32026.03 | 23 | — | — | — | 0.9 | — | 0.2 | — | 23.1 | — | — | |
| Video-LLaMA32025.10 | 23 | 41.9 | — | — | 0.9 | — | 0.2 | — | 23.1 | 21.7 | 27 | |
| Qwen3-VL-4BFrames=642026.03 | 21.3 | — | — | — | 22.3 | — | 5 | — | 18.5 | — | — | |
| LLaVA-ST2025.12 | 19.9 | 55 | 17.4 | 20.1 | 22.2 | 8.2 | 9.5 | 15.7 | 23.4 | 30.8 | 40.6 | |
| TRACE2025.12 | 19.1 | 17.6 | 14.1 | — | — | — | — | 12 | 17.1 | 12 | 13.3 | |
| TRACE2026.03 | 19.1 | — | — | — | 0 | — | 0 | — | 17.1 | — | — | |
| TRACE2025.10 | 19.1 | 17.6 | — | — | 0 | — | 0 | — | 17.1 | 12 | 13.3 | |
| Qwen3-VL-4BModel size=4B2025.10 | 18.6 | 59.6 | — | — | 26.8 | — | 7.7 | — | 19.2 | 31.9 | 43.7 | |
| VTimeLLMParameters=7B2025.12 | 17.1 | 41.5 | 10.9 | 0 | 0.21 | 0 | 0 | 2.1 | 5.96 | 17.7 | 22 | |
| GPT-4o2025.12 | 16.7 | 60.8 | 10.4 | 2.8 | 6.5 | 1.2 | 3 | 10 | 12.8 | 26.8 | 38.2 | |
| GPT-4o2026.03 | 16.7 | — | — | — | 6.5 | — | 3 | — | 12.8 | — | — | |
| GPT-4o2025.10 | 16.7 | 60.8 | — | — | 6.5 | — | 3 | — | 12.8 | 26.8 | 38.2 | |
| Qwen2.5-VL-7B2026.03 | 15.4 | — | — | — | 17 | — | 2.5 | — | 13.8 | — | — | |
| Qwen2.5-VL-7B (base)Evaluation framework=vLLM, Number of sampled frames=162025.10 | 15.4 | 33.5 | — | — | 17 | — | 2.5 | — | 13.8 | 19.3 | 22.4 | |
| VideoChat22025.12 | 13.7 | 36.2 | 13.1 | 0.1 | 2.5 | 0.3 | 1 | 12.1 | 12.5 | 17 | 20.3 | |
| VideoChat22026.03 | 13.7 | — | — | — | 2.5 | — | 1 | — | 12.5 | — | — | |
| VideoChat22025.10 | 13.7 | 36.2 | — | — | 2.5 | — | 1 | — | 12.5 | 17 | 20.3 | |
| Oryx-1.5Parameters=7B2025.12 | 13.5 | 20.5 | 4.5 | 2.2 | 10.1 | 1 | 3.5 | 5.6 | 14.8 | 15.1 | 13.8 | |
| Oryx-1.5-7B2026.03 | 13.5 | — | — | — | 10.1 | — | 3.5 | — | 14.8 | — | — | |
| Oryx-1.5-7B2025.10 | 13.5 | 20.5 | — | — | 10.1 | — | 3.5 | — | 14.8 | 15.1 | 13.8 | |
| TimeChatParameters=7B2025.12 | 12.01 | 26.4 | 8.7 | — | — | — | — | 8.5 | 13.6 | 13.1 | 14.8 | |
| Qwen2.5-VLParameters=7B2025.12 | 11.5 | 54.5 | 8.9 | 8.4 | 13.6 | 1.4 | 2 | 5.4 | 7.6 | 24 | 32.4 | |
| LLaVA-Video2025.12 | 10.5 | 49.5 | 6.3 | 0.2 | 1.9 | 0.3 | 1.3 | 5.5 | 12.2 | 20.8 | 27.3 | |
| LLaVA-Video2026.03 | 10.5 | — | — | — | 1.9 | — | 1.3 | — | 12.2 | — | — | |
| LLaVA-Video2025.10 | 10.5 | 49.5 | — | — | 1.9 | — | 1.3 | — | 12.2 | 20.8 | 27.3 | |
| InternVL-2.5Parameters=8B2025.12 | 8.7 | 44.2 | 4.9 | 0 | 0.7 | 0 | 0.1 | 3.8 | 7.8 | 17.6 | 24.9 | |
| InternVL-2.5-8B2026.03 | 8.7 | — | — | — | 0.7 | — | 0.1 | — | 7.8 | — | — | |
| InternVL-2.5-8B2025.10 | 8.7 | 44.2 | — | — | 0.7 | — | 0.1 | — | 7.8 | 17.6 | 24.9 | |
| Sa2VAParameters=8B2025.12 | 0.1 | 16.4 | 0 | 34.18 | 32.3 | 40.4 | 37.5 | 0 | 0 | 17.1 | 20.3 | |
| Sa2VA-8B2026.03 | 0.1 | — | — | — | 32.3 | — | 37.5 | — | 0 | — | — | |
| Sa2VA-8B2025.10 | 0.1 | 16.4 | — | — | 32.3 | — | 37.5 | — | 0 | 17.1 | 20.3 | |
| GPT-4oRelease Date=May 242026.04 | — | — | — | — | — | — | — | — | — | 26.8 | 38.2 | |
| Omni-o32026.04 | — | — | — | — | — | — | — | — | — | 27.1 | 35 | |
| Open-o3-videoRelease Date=Oct 252026.04 | — | — | — | — | — | — | — | — | — | 33.6 | 46.6 | |
| Qwen2.5-OmniRelease Date=Mar 252026.04 | — | — | — | — | — | — | — | — | — | 17.3 | 20.5 | |
| Qwen2.5-VLRelease Date=Feb 252026.04 | — | — | — | — | — | — | — | — | — | 19.3 | 22.4 | |
| Qwen3-OmniRelease Date=Sep 252026.04 | — | — | — | — | — | — | — | — | — | 30.3 | 42.3 | |
| Qwen3-VLRelease Date=Oct 252026.04 | — | — | — | — | — | — | — | — | — | 18.8 | 21.1 | |
| Time-R1Release Date=Jun 252026.04 | — | — | — | — | — | — | — | — | — | 27.8 | 36 | |
| VideoRFTRelease Date=May 252026.04 | — | — | — | — | — | — | — | — | — | 16.9 | 19.8 |