Binary Question Answering on ACaM Synthetic Videos (test)
99Accuracy (Static)Human Performance
Evaluation results
| Method | Links | ||||||||
|---|---|---|---|---|---|---|---|---|---|
| Human Performance2026.07 | 99 | 99.64 | 98.58 | 98.59 | 100 | 100 | 99.15 | 99.15 | |
| Our SFT Qwen3-VL-4BParameters=4B2026.07 | 88.42 | 82.6 | 74.44 | 81.42 | 70.23 | 73.95 | 78.28 | 78.62 | |
| Mega-SaM2026.07 | 86 | 80.2 | 67.1 | — | — | — | — | — | |
| GPT-52026.07 | 78.65 | 70.75 | 64.05 | 79.66 | 69.88 | 82.35 | 70.22 | 70.2 | |
| Qwen3-VL-4BParameters=4B2026.07 | 76.03 | 72.26 | 59.76 | 65.38 | 59.18 | 72.73 | 66.58 | 65.44 | |
| ShotVL-7BParameters=7B2026.07 | 75.23 | 76.82 | 68.08 | 72.57 | 56.25 | 83.19 | 72.29 | 72.73 | |
| Qwen3-VL-32BParameters=32B2026.07 | 75 | 81.3 | 65.34 | 70.83 | 72.16 | 78.57 | 73.31 | 72.82 | |
| Gemini-3-Pro2026.07 | 75 | 63.4 | 66.67 | 61.18 | 80 | 86.96 | 68.26 | 65.42 | |
| CamReasoner-7BParameters=7B2026.07 | 74.07 | 78.64 | 70.15 | 76.8 | 77.19 | 94.23 | 75.98 | 76.24 | |
| InternVL3.5-8BParameters=8B2026.07 | 72.92 | 59.68 | 45.09 | 64 | 25.4 | 69.14 | 54.79 | 52.17 | |
| Our SFT Qwen3-VL-8BParameters=8B2026.07 | 70.89 | 87.18 | 78.73 | 79.63 | 68.75 | 79.25 | 80.56 | 80.69 | |
| Gemini-3.1-Pro2026.07 | 70.83 | 66.97 | 62.96 | 59.09 | 71.91 | 87.91 | 67.09 | 64.76 | |
| ViPE2026.07 | 66.7 | 58.9 | 76.6 | — | — | — | — | — | |
| Qwen3-VL-8BParameters=8B2026.07 | 57.53 | 77.48 | 62.99 | 72.55 | 61.39 | 84.78 | 69.88 | 69.65 | |
| CameraModel-7BParameters=7B2026.07 | 57.43 | 55.21 | 56.99 | 52.73 | 70.97 | 47.06 | 56.53 | 56.21 | |
| Random Guess2026.07 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | 50 | |
| LLaVa-OV-7BParameters=7B2026.07 | 35.56 | 49.67 | 31.3 | 27.27 | 59.18 | 72.16 | 43.81 | 39.71 | |
| InternVL3.5-14BParameters=14B2026.07 | 24.56 | 58.58 | 60.31 | 69.16 | 70.09 | 77.55 | 61.01 | 57.41 | |
| Spatial-MLLM2026.07 | 23.88 | 65.63 | 55.09 | 64.97 | 64.38 | 70.42 | 60.8 | 57.06 | |
| VLM-3R-7BParameters=7B2026.07 | 21.21 | 35.23 | 36.03 | 26.37 | 19.35 | 70.73 | 35.73 | 31.5 | |
| InternVideo2.5-8BParameters=8B2026.07 | 18.18 | 67.5 | 53.22 | 68.22 | 67.65 | 83.93 | 61.68 | 58.06 | |
| G2VLM-2BParameters=2B2026.07 | 11.11 | 8.58 | 9.55 | 0 | 7.41 | 30 | 9.76 | 8.99 | |
| Qwen2.5-VL-7BParameters=7B2026.07 | 10.91 | 69.96 | 66.46 | 67.23 | 61.95 | 82.57 | 66.58 | 65.29 |