Motion Direction Understanding on MODIRECT-SYNBENCH
100Top-1 Accuracy (P-Syn)Human
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| Human2026.05 | 100 | 100 | 100 | 100 | 100 | |
| LLaVA-Video-7B w/ DeltaDirectmode=LoRA post-tuning2026.05 | 99.7 | 84.9 | 85.2 | 71.7 | 85.4 | |
| Full-FT-0.5B w/ DeltaDirectmode=Full fine-tuning, backbone=Qwen2-0.5B + SigLIP2026.05 | 99.7 | 98.7 | 91 | 80.1 | 92.4 | |
| LLaVA-Video-7B w/ DeltaDirectFine-tuning protocol=Instruction tuning, Supervision method=DeltaDirect, Backbone=7B2026.05 | 99.7 | 84.9 | 85.2 | 71.7 | 85.4 | |
| Qwen2-0.5B Full-FT w/ DeltaDirectFine-tuning protocol=Full fine-tuning, Supervision method=DeltaDirect, Backbone=Qwen2-0.5B2026.05 | 99.7 | 98.7 | 91 | 80.1 | 92.4 | |
| LLaVA-Video-7B w/ MODIRECT-INSTmode=LoRA post-tuning2026.05 | 99.5 | 80.7 | 74.7 | 60.5 | 78.9 | |
| Full-FT-0.5B w/ VideoChat2 + MODIRECT-INSTmode=Full fine-tuning, backbone=Qwen2-0.5B + SigLIP2026.05 | 99.5 | 97.3 | 62.5 | 51.5 | 77.7 | |
| LLaVA-Video-7B w/ MODIRECT-INSTFine-tuning protocol=Instruction tuning, Supervision method=MODIRECT-INST, Backbone=7B2026.05 | 99.5 | 80.7 | 74.7 | 60.5 | 78.9 | |
| Qwen2-0.5B Full-FT w/ MODIRECT-INSTFine-tuning protocol=Full fine-tuning, Supervision method=MODIRECT-INST, Backbone=Qwen2-0.5B2026.05 | 99.5 | 97.3 | 62.5 | 51.5 | 77.7 | |
| Qwen3-VL-4B2026.05 | 66.6 | 50.2 | 40.8 | 41.1 | 49.7 | |
| Qwen3-VL-4BBackbone=4B2026.05 | 66.6 | 50.2 | 40.8 | 41.1 | 49.7 | |
| Gemini 2.5 Flash2026.05 | 58.1 | 43.4 | 61.7 | 50.6 | 53.5 | |
| Gemini 2.5 Flash2026.05 | 58.1 | 43.4 | 61.7 | 50.6 | 53.5 | |
| VideoLLaMA3-7B2026.05 | 56.4 | 46.6 | 50 | 48.3 | 50.3 | |
| VideoLLaMA3-7BBackbone=7B2026.05 | 56.4 | 46.6 | 50 | 48.3 | 50.3 | |
| GPT-4o2026.05 | 46.6 | 37.8 | 48 | 40.7 | 43.3 | |
| GPT-4o2026.05 | 46.6 | 37.8 | 48 | 40.7 | 43.3 | |
| Qwen2.5-VL-7B2026.05 | 45.1 | 32.5 | 30.8 | 30.2 | 34.7 | |
| Qwen2.5-VL-7BBackbone=7B2026.05 | 45.1 | 32.5 | 30.8 | 30.2 | 34.7 | |
| InternVL-2.5-4B2026.05 | 31.9 | 31.8 | 30.4 | 31.4 | 31.4 | |
| InternVL-2.5-4BBackbone=4B2026.05 | 31.9 | 31.8 | 30.4 | 31.4 | 31.4 | |
| VideoLLaMA3-2B2026.05 | 31.2 | 27.5 | 28.1 | 28.3 | 28.8 | |
| GPT-4o-mini2026.05 | 29.5 | 27.6 | 28.6 | 27.1 | 28.2 | |
| LLaVA-OneVision-7B2026.05 | 28.3 | 23.4 | 29.9 | 29.3 | 27.7 | |
| LLaVA-OneVision-7BBackbone=7B2026.05 | 28.3 | 23.4 | 29.9 | 29.3 | 27.7 | |
| Video-LLaVA-7B2026.05 | 28.1 | 27.8 | 25.5 | 27.2 | 27.2 | |
| Video-LLaVA-7BBackbone=7B2026.05 | 28.1 | 27.8 | 25.5 | 27.2 | 27.2 | |
| LLaVA-Video-7B2026.05 | 27.6 | 23.4 | 26.9 | 25.8 | 25.9 | |
| LLaVA-Video-7BBackbone=7B2026.05 | 27.6 | 23.4 | 26.9 | 25.8 | 25.9 | |
| VideoChat2-HD-7B2026.05 | 26.1 | 23.5 | 24.6 | 24.1 | 24.6 | |
| Full-FT-0.5B w/ VideoChat2mode=Full fine-tuning, backbone=Qwen2-0.5B + SigLIP2026.05 | 26.1 | 23.5 | 24.6 | 24.1 | 24.6 | |
| VideoChat2-HD-7BBackbone=7B2026.05 | 26.1 | 23.5 | 24.6 | 24.1 | 24.6 | |
| LLaVA-NeXT-Video-7B2026.05 | 25.8 | 24.8 | 25 | 25.2 | 25.2 | |
| LLaVA-NeXT-Video-7BBackbone=7B2026.05 | 25.8 | 24.8 | 25 | 25.2 | 25.2 | |
| mPLUG-Owl3-7B2026.05 | 25.6 | 24.2 | 25.9 | 25.8 | 25.4 | |
| LLaMA-VID-7B2026.05 | 25.3 | 25.6 | 24.7 | 25 | 25.2 | |
| LLaMA-VID-7BBackbone=7B2026.05 | 25.3 | 25.6 | 24.7 | 25 | 25.2 | |
| LLaVA-Video-7B w/ FlashVID2026.05 | 25.2 | 25.1 | 24.3 | 24.8 | 24.9 | |
| LLaVA-Video-7B w/ FlashVIDSupervision method=FlashVID, Backbone=7B2026.05 | 25.2 | 25.1 | 24.3 | 24.8 | 24.9 | |
| Random Chance2026.05 | 25 | 25 | 25 | 25 | 25 | |
| Random Chance2026.05 | 25 | 25 | 25 | 25 | 25 | |
| LLaVA-OneVision-SI-7B2026.05 | 24.4 | 23.4 | 26.2 | 26.6 | 25.2 |