# Cinematography language understanding on ShotBench 1.0 (test) Source: [Cinematography language understanding on ShotBench 1.0 (test)]() Collection index: [llms.txt]() Data updated: 2026-03-31T06:59:54.516Z Evaluation of model performance on the ShotBench 1.0 test set, specifically assessing understanding of various cinematography elements such as shot size, framing, camera angle, lighting, and composition. A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. | Scope | Recorded value | | --- | --- | | Task | Cinematography language understanding | | Dataset (including any stated split) | ShotBench 1.0 (test) | | Metric | Shot Size Accuracy (SS) | | Unit | PERCENT | | Direction | Higher is better | | Results loaded | 25 of 25 | | Coverage | All results currently returned for this benchmark | Evaluation settings may differ between results. The original papers and recorded configurations are needed to establish comparability. A shared metric name alone does not establish an identical protocol. ## Metrics - [Shot Size Accuracy (SS)](): Higher is better; unit: PERCENT. - [Shot Framing Accuracy (SF)](): Higher is better; unit: PERCENT. - [Camera Angle Accuracy (CA)](): Higher is better; unit: PERCENT. - [Lens Size Accuracy (LS)](): Higher is better; unit: PERCENT. - [Lighting Type Accuracy (LT)](): Higher is better; unit: PERCENT. - [Lighting Conditions Accuracy (LC)](): Higher is better; unit: PERCENT. - [Shot Composition Accuracy (SC)](): Higher is better; unit: PERCENT. - [Camera Movement Accuracy (CM)](): Higher is better; unit: PERCENT. - [Average Accuracy](): Higher is better; unit: PERCENT. ## Recorded results | Method | Shot Size Accuracy (SS) | Result date | Report paper | | --- | --- | --- | --- | | ShotVL (3B) | 77.9 | Not recorded | 68fa79b31e209131cea7ef72 | | Qwen2.5-VL-72B-Instruct | 75.1 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL3-78B | 69.7 | Not recorded | 68fa79b31e209131cea7ef72 | | GPT-4o | 69.3 | Not recorded | 68fa79b31e209131cea7ef72 | | Qwen2.5-VL-7B-Instruct | 69.1 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL3-38B | 68 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL2.5-38B | 67.8 | Not recorded | 68fa79b31e209131cea7ef72 | | Qwen2.5-VL-32B-Instruct | 62.3 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL3-8B | 62.1 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL3-14B | 59.6 | Not recorded | 68fa79b31e209131cea7ef72 | | LLaVA-Onevision-Qwen2-7B-Ov-Chat | 58.4 | Not recorded | 68fa79b31e209131cea7ef72 | | Gemini-2.5-flash-preview-04-17 | 57.7 | Not recorded | 68fa79b31e209131cea7ef72 | | LLaVA-Video-7B-Qwen2 | 56.9 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL2.5-8B | 56.3 | Not recorded | 68fa79b31e209131cea7ef72 | | InternVL3-2B | 56.3 | Not recorded | 68fa79b31e209131cea7ef72 | | Qwen2.5-VL-3B-Instruct | 54.6 | Not recorded | 68fa79b31e209131cea7ef72 | | Internlm-xcomposer2d5-7B | 51.1 | Not recorded | 68fa79b31e209131cea7ef72 | | Gemini-2.0-flash | 48.9 | Not recorded | 68fa79b31e209131cea7ef72 | | VILA1.5-8B | 40.6 | Not recorded | 68fa79b31e209131cea7ef72 | | VILA1.5-13B | 36.7 | Not recorded | 68fa79b31e209131cea7ef72 | | LLaVA-NeXT-Video-7B | 35.9 | Not recorded | 68fa79b31e209131cea7ef72 | | Ovis2-8B | 35.9 | Not recorded | 68fa79b31e209131cea7ef72 | | VILA1.5-3B | 33.4 | Not recorded | 68fa79b31e209131cea7ef72 | | Instructblip-vicuna-7B | 27 | Not recorded | 68fa79b31e209131cea7ef72 | | Instructblip-vicuna-13B | 26.8 | Not recorded | 68fa79b31e209131cea7ef72 | ### ShotVL (3B) Shot Size Accuracy (SS): 77.9. Result ID: 69cb6fef47f34ce924b33504. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Ours"}. This record does not assert independent reproduction. ### Qwen2.5-VL-72B-Instruct Shot Size Accuracy (SS): 75.1. Result ID: 69cb6fef47f34ce924b334ff. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL3-78B Shot Size Accuracy (SS): 69.7. Result ID: 69cb6fef47f34ce924b33500. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### GPT-4o Shot Size Accuracy (SS): 69.3. Result ID: 69cb6fef47f34ce924b33503. [Reporting paper]() Recorded evaluation configuration: {"Access":"Proprietary"}. This record does not assert independent reproduction. ### Qwen2.5-VL-7B-Instruct Shot Size Accuracy (SS): 69.1. Result ID: 69cb6fef47f34ce924b334ed. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL3-38B Shot Size Accuracy (SS): 68. Result ID: 69cb6fef47f34ce924b334fd. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL2.5-38B Shot Size Accuracy (SS): 67.8. Result ID: 69cb6fef47f34ce924b334fc. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Qwen2.5-VL-32B-Instruct Shot Size Accuracy (SS): 62.3. Result ID: 69cb6fef47f34ce924b334fe. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL3-8B Shot Size Accuracy (SS): 62.1. Result ID: 69cb6fef47f34ce924b334f3. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL3-14B Shot Size Accuracy (SS): 59.6. Result ID: 69cb6fef47f34ce924b334f4. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### LLaVA-Onevision-Qwen2-7B-Ov-Chat Shot Size Accuracy (SS): 58.4. Result ID: 69cb6fef47f34ce924b334f0. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Gemini-2.5-flash-preview-04-17 Shot Size Accuracy (SS): 57.7. Result ID: 69cb6fef47f34ce924b33502. [Reporting paper]() Recorded evaluation configuration: {"Access":"Proprietary"}. This record does not assert independent reproduction. ### LLaVA-Video-7B-Qwen2 Shot Size Accuracy (SS): 56.9. Result ID: 69cb6fef47f34ce924b334ef. [Reporting paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL2.5-8B Shot Size Accuracy (SS): 56.3. Result ID: 69cb6fef47f34ce924b334f1. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### InternVL3-2B Shot Size Accuracy (SS): 56.3. Result ID: 69cb6fef47f34ce924b334f2. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Qwen2.5-VL-3B-Instruct Shot Size Accuracy (SS): 54.6. Result ID: 69cb6fef47f34ce924b334ec. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Internlm-xcomposer2d5-7B Shot Size Accuracy (SS): 51.1. Result ID: 69cb6fef47f34ce924b334f5. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Gemini-2.0-flash Shot Size Accuracy (SS): 48.9. Result ID: 69cb6fef47f34ce924b33501. [Reporting paper]() Recorded evaluation configuration: {"Access":"Proprietary"}. This record does not assert independent reproduction. ### VILA1.5-8B Shot Size Accuracy (SS): 40.6. Result ID: 69cb6fef47f34ce924b334f8. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### VILA1.5-13B Shot Size Accuracy (SS): 36.7. Result ID: 69cb6fef47f34ce924b334f9. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### LLaVA-NeXT-Video-7B Shot Size Accuracy (SS): 35.9. Result ID: 69cb6fef47f34ce924b334ee. [Reporting paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Ovis2-8B Shot Size Accuracy (SS): 35.9. Result ID: 69cb6fef47f34ce924b334f6. [Reporting paper]() [Method origin paper]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### VILA1.5-3B Shot Size Accuracy (SS): 33.4. Result ID: 69cb6fef47f34ce924b334f7. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Instructblip-vicuna-7B Shot Size Accuracy (SS): 27. Result ID: 69cb6fef47f34ce924b334fa. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. ### Instructblip-vicuna-13B Shot Size Accuracy (SS): 26.8. Result ID: 69cb6fef47f34ce924b334fb. [Reporting paper]() [Method origin paper]() [Implementation]() Recorded evaluation configuration: {"Access":"Open-Sourced"}. This record does not assert independent reproduction. Page 1 of 1. This is an index page, not the full collection.