Multi-modal Reasoning on MUIRBENCH

92.94Difference Reasoning AccuracyHuman

Evaluation results

Method
Human2026.02	92.94	93.15	94.87	97.56	85.71	94.83	87.5	94.62	82.05	98.99	98	87.76	86.3
Qwen3VL-4B-Thinking2026.02	68.24	73.81	81.62	57.32	53.57	85.78	34.38	86.02	51.28	92.96	45	62.76	68.15
ReMoT-CoT-4B2026.02	62.65	72.81	82.48	55.49	50	84.7	40.62	84.95	52.56	92.46	48	64.8	67.47
GPT-4-Turbo2026.02	60.59	—	—	—	—	—	—	—	52.56	79.15	57	50.51	64.04
GPT-4o2026.02	60.29	68	49.15	44.51	36.9	86.85	23.44	71.51	51.28	88.69	56	56.12	80.14
Gemini Pro2026.02	45.29	49.35	28.63	35.98	28.57	66.59	12.5	59.14	47.44	64.82	48	41.33	43.84
Idefics-9B-Instruct2026.02	34.41	35.43	29.91	28.05	13.1	35.99	12.5	27.41	48.72	46.98	35	32.65	43.49
LLaVA-v1.5-7B-xtuner2026.02	33.53	33.23	26.92	25.61	23.81	22.84	4.69	39.78	29.49	44.72	26	38.78	47.6
Emu2-Chat (37B)2026.02	32.65	33.62	31.2	27.44	26.19	37.28	15.63	48.39	43.59	37.69	34	31.63	23.97
Mantis-8B-Idefics22026.02	28.82	44.5	38.46	33.54	26.19	53.88	18.75	56.99	38.46	67.59	26	48.47	35.62
Idefics2-8B2026.02	27.65	26.08	21.79	26.22	26.19	24.78	15.62	56.45	39.74	25.38	21	17.86	17.12
VILA1.5-13B2026.02	24.71	33.12	19.66	28.66	25	40.95	10.94	56.45	30.77	42.71	31	24.49	30.14
Random Choice2026.02	23.18	23.99	20.98	23.41	25	24.12	22.81	25	25	29.56	25	20	21.3
LLaVA-NeXT-34B2026.02	22.06	33.31	36.32	26.22	33.33	37.93	21.88	54.3	41.03	38.19	12	38.27	25
OpenFlamingo-v2-9B2026.02	21.76	23.73	21.79	26.83	30.95	24.14	21.88	22.58	25.64	31.91	25	18.88	15.41
LLaVA-v1.5-13B2026.02	20	24.38	25.21	29.27	14.29	20.26	20.31	36.56	25.64	31.66	20	22.96	20.89
MiniGPT-4-v22026.02	20	17.35	11.97	14.02	25	17.03	18.75	14.52	21.79	21.61	13	17.35	14.73
CogVLM2026.02	19.71	20.85	14.1	26.22	16.67	21.34	12.5	41.4	41.03	19.6	13	16.33	15.75
LLaVA-internLM2-7B2026.02	19.12	28.15	34.19	26.22	32.14	25.65	7.81	42.47	39.74	35.43	12	23.98	28.42