Cinematography language understanding on ShotBench 1.0 (test)

77.9Shot Size Accuracy (SS)ShotVL (3B)

Evaluation Results

Method
ShotVL (3B)2025.06	77.9	85.6	68.8	59.3	65.7	53.1	57.4	51.7	65.1
Qwen2.5-VL-72B-Instruct2025.06	75.1	82.9	56.7	46.8	59	49.4	54.1	48.9	59.1
InternVL3-78B2025.06	69.7	80	54.5	44	65.5	47.4	51.8	44.4	57.2
GPT-4o2025.06	69.3	83.1	58.2	48.9	63.2	48	55.2	48.3	59.3
Qwen2.5-VL-7B-Instruct2025.06	69.1	73.5	53.2	47	60.5	47.4	49.9	30.2	53.8
InternVL3-38B2025.06	68	84	51.9	43.6	64.4	46.9	54.7	44.6	57.3
InternVL2.5-38B2025.06	67.8	85.4	55.4	41.7	61.7	48.9	52.4	44	57.2
Qwen2.5-VL-32B-Instruct2025.06	62.3	76.6	51	48.3	61.7	44	52.2	43.8	55
InternVL3-8B2025.06	62.1	65.8	46.8	42.9	58	44.3	46.8	44.2	51.4
InternVL3-14B2025.06	59.6	82.2	55.4	40.7	61.7	44.6	51.1	38.2	54.2
LLaVA-Onevision-Qwen2-7B-Ov-Chat2025.06	58.4	71	52.3	38.7	59.5	44.9	50.9	39.7	51.9
Gemini-2.5-flash-preview-04-172025.06	57.7	82.9	51.4	43.8	65.2	45.7	45.9	43.5	54.5
LLaVA-Video-7B-Qwen22025.06	56.9	65.4	45.1	36	63.5	45.4	37.4	35.3	48.1
InternVL2.5-8B2025.06	56.3	70.3	50.8	41.1	60.2	45.1	50.1	33.6	50.9
InternVL3-2B2025.06	56.3	56	44.4	34.6	56.8	44.6	43	38.1	46.7
Qwen2.5-VL-3B-Instruct2025.06	54.6	56.6	43.1	36.6	59.3	45.1	41.5	31.9	46.1
Internlm-xcomposer2d5-7B2025.06	51.1	71	39.8	32.7	59.3	35.7	35.7	38.8	45.5
Gemini-2.0-flash2025.06	48.9	75.5	44.6	31.9	62.2	48.9	52.4	47.4	51.5
VILA1.5-8B2025.06	40.6	44.5	39.1	29.7	48.9	32.9	34.4	36.9	38.4
VILA1.5-13B2025.06	36.7	54.6	40.7	34.8	52.8	35.4	34.2	31.3	40.1
LLaVA-NeXT-Video-7B2025.06	35.9	37.1	32.5	27.8	50.9	31.7	28	31.3	34.4
Ovis2-8B2025.06	35.9	37.1	32.5	27.8	50.9	31.7	28	35.3	34.9
VILA1.5-3B2025.06	33.4	44.9	32.1	28.6	50.6	35.7	28.4	21.5	34.4
Instructblip-vicuna-7B2025.06	27	27.9	34.5	29.4	44.4	29.7	27.1	25	30.6
Instructblip-vicuna-13B2025.06	26.8	29.2	27.9	28	39	24	27.1	22	28