Video Counterfactual Reasoning on CounterVQA 1.0 (test)
77.4Accuracy (Human to Human Level 1)Qwen-3-VL 8B + SFT + RL
Evaluation results
| Method | Links | ||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen-3-VL 8B + SFT + RLModel Type=Open-source, Training Protocol=SFT + RL2025.11 | 77.4 | 57.1 | 63.2 | 68.8 | 70.6 | 73.9 | 77.4 | 73.8 | 70.1 | 71.6 | 76 | 72.6 | |
| Qwen-3-VL 8B + SFTModel Type=Open-source, Training Protocol=SFT2025.11 | 76.3 | 54.5 | 73.7 | 72 | 65.9 | 71.4 | 70.6 | 68.8 | 68.9 | 67.9 | 71 | 69.3 | |
| Gemini-2.5-ProModel Type=Closed-source, Training Protocol=API2025.11 | 57.1 | 25 | 40 | 43.8 | 39.3 | 42.4 | 42.6 | 41.3 | 41.1 | 40.5 | 42.4 | 41.5 | |
| Qwen-3-VL 8BModel Type=Open-source, Training Protocol=Vanilla2025.11 | 45.1 | 54.5 | 52.6 | 49.1 | 56.7 | 63.7 | 66.4 | 61.9 | 54.6 | 62.1 | 65.3 | 60.1 | |
| Qwen-2.5-VL 32BModel Type=Open-source, Training Protocol=Vanilla2025.11 | 28.6 | 21.4 | 64.3 | 36.7 | 33.5 | 51.5 | 41.9 | 39.7 | 33 | 46.2 | 44 | 39.4 | |
| Qwen-2.5-VL 7BModel Type=Open-source, Training Protocol=Vanilla2025.11 | 13.3 | 11.5 | 3.1 | 9.1 | 12.1 | 9.9 | 2.5 | 8.1 | 12.3 | 10.3 | 2.7 | 8.3 | |
| ChatGPT-4oModel Type=Closed-source, Training Protocol=API2025.11 | 12.5 | 15.8 | 8.3 | 12.7 | 9 | 6.5 | 0.7 | 3.9 | 9.6 | 8 | 1.3 | 6.2 | |
| Qwen-2.5-VL 7B + SFTModel Type=Open-source, Training Protocol=SFT2025.11 | 12.5 | 15.4 | 3.6 | 10.7 | 12.8 | 9.9 | 3.3 | 8.7 | 12.7 | 11.2 | 3.4 | 9.1 | |
| Qwen-2.5-VL 7B + SFT + RLModel Type=Open-source, Training Protocol=SFT + RL2025.11 | 12.1 | 15.4 | 3.5 | 10.3 | 12.8 | 9.9 | 4.2 | 9 | 12.7 | 11.2 | 4.1 | 9.3 |