Multimodal idiomatic understanding on Varnika (test) benchmark leaderboard · SOTA2 Research

Multimodal idiomatic understanding on Varnika (test)

0.54ROUGE-1 ScoreQwen2.5-VL-7B-Instruct

Evaluation Results

Method	Links
Qwen2.5-VL-7B-Instruct2026.06		0.54	0.32	0.41	0.43	56.32	28.92	21.48	24.19	0.95	0.82	0.47
Qwen2.5-VL-7B-Instruct2026.06		0.48	0.22	0.36	0.37	51.06	24.81	16.51	20.68	0.93	0.75	0.46
SmolVLM-Instruct2026.06		0.48	0.21	0.36	0.36	50.46	24.16	16.59	20.92	0.92	0.66	0.35
Gemma3-12b-pt2026.06		0.46	0.19	0.35	0.35	38.81	17.82	11.93	14.58	0.92	0.79	0.55
Paligemma2-10B2026.06		0.45	0.2	0.34	0.35	35.97	17.25	14.2	15.94	0.89	0.59	0.32
SmolVLM-Instruct2026.06		0.43	0.18	0.33	0.33	47.48	21.82	14.06	17.7	0.92	0.59	0.35
Gemma3-12b-pt2026.06		0.4	0.15	0.29	0.29	32.86	13.36	7.9	11.64	0.9	0.74	0.54
Llava-1.5-7B2026.06		0.4	0.17	0.32	0.34	32.52	16.64	12.95	14.25	0.91	0.61	0.33
Paligemma2-10B2026.06		0.39	0.15	0.27	0.29	32.62	15.73	8.95	13.73	0.88	0.53	0.32
Llava-1.5-7B2026.06		0.35	0.15	0.26	0.26	27.46	12.74	8.2	11.4	0.9	0.57	0.31
Blip2-7B2026.06		0.3	0.13	0.23	0.28	23.76	12.93	13.29	15.64	0.89	0.55	0.33
Blip2-7B2026.06		0.26	0.07	0.17	0.18	18.64	8.62	6.28	9.17	0.88	0.52	0.32