Multi-hop Multiple-choice Question Answering on T2S-Bench-MR 1.0 (test)

81.4Overall EMGemini-2.5-Pro

Evaluation results

Method
Gemini-2.5-Pro2026.03	81.4	91.56	80	91.78	90.65	93.46	74.51	90.62	89.86	91.01
Claude-sonnet-4-5-202509292026.03	76.8	86.85	84.17	92.14	88.79	92.49	65.2	82.71	79.71	81.16
DeepSeek-reasoner(R1)2026.03	76.6	87.2	83.33	91.58	84.11	91.37	65.69	82.62	85.51	86.67
Claude-4-Sonnet-202505142026.03	75.6	86.63	84.17	92.47	89.72	94.02	61.27	80.68	81.16	82.61
Gemini-2.5-Flash2026.03	72.2	83.67	77.5	87.33	78.5	84.33	64.22	82.76	76.81	78.94
GPT-5.22026.03	71.8	84.32	77.5	90.47	84.11	89.69	62.75	82.24	69.57	71.45
Qwen3-32B2026.03	69.4	83.41	78.33	89.72	89.72	93.6	53.43	78.24	69.57	71.88
GPT-5.12026.03	67.8	83.46	75.83	89.75	83.18	89.5	51.47	77.95	78.26	79.42
Claude-haiku-4-5-202510012026.03	67.4	80.87	78.33	90.33	85.98	89.07	49.51	73.35	72.46	73.91
Kimi-K2-Instruct-09052026.03	67	81	75.83	88.58	84.11	88.72	50.98	74.74	72.46	74.35
Qwen3-235B-A22B-Instruct-25072026.03	62.8	80.25	61.67	82.72	78.5	87.54	50	74.61	78.26	81.3
GPT-4o2026.03	61.8	79.24	73.33	89.31	79.44	86.64	44.12	72.8	66.67	69.32
Gemini-2.0-flash2026.03	61.2	79.63	72.5	87.03	75.7	86.76	41.67	70.99	76.81	81.26
Qwen3-235B-A22B-Thinking-25072026.03	60.8	79.89	60	82.19	74.77	85.42	50	76.95	72.46	76.04
DeepSeek-V3-03242026.03	60.6	78.75	65.83	82.31	78.5	87.85	43.63	72.71	73.91	76.33
DeepSeek-V3.12026.03	60.4	78.59	66.67	82.72	78.5	87.85	42.65	72.07	73.91	76.33
DeepSeek-chat2026.03	60.2	78.37	70	83.86	78.5	88.03	40.69	71.26	72.46	74.88
DeepSeek-V3.22026.03	60	78.31	66.67	82.72	78.5	87.85	42.16	71.73	72.46	75.31
GPT-4.1-mini2026.03	59	76.83	66.67	84.19	68.22	82.55	48.04	72.37	63.77	68.31
DeepSeek-R1-05282026.03	57	63.76	58.33	61.08	65.42	68.5	46.57	59.11	72.46	74.78
Mistral-Small-3.2-24B-Instruct-25062026.03	56.8	75.48	65	82.56	80.37	87.07	36.76	67.66	65.22	68.31
Mistral-Large-Instruct-24112026.03	56.6	74.75	65	83.47	70.09	77.1	42.65	71.7	62.32	64.98
Llama-3.1-70B-Instruct2026.03	56.6	74.9	65.83	81.56	77.57	84.89	37.75	68.15	63.77	67.78
MiniMax-Text-012026.03	55.8	74.69	66.67	80.36	72.9	81.87	35.78	68.5	69.57	71.98
Ministral-3-14B-Instruct-25122026.03	55.8	75.68	59.17	81.47	77.57	85.14	39.22	69.1	65.22	70.39
Llama-3.3-70B-Instruct2026.03	54	73.1	59.17	78.81	72.9	82.55	40.2	68.8	56.52	61.26
Gemini-2.0-flash-lite2026.03	53.4	72.41	65	81.08	79.44	83.74	29.9	63.35	62.32	66.57
Ministral-3-8B-Instruct-25122026.03	52.4	71.37	59.17	75.11	71.96	78.26	34.31	67.18	63.77	66.62
Llama-3.1-405B-Instruct2026.03	50.4	60.85	58.33	66.81	53.27	57.23	41.18	59.08	59.42	61.3
Qwen3-8B2026.03	47.6	66.47	51.67	69.33	59.81	71.84	38.24	67.07	49.28	51.4
Qwen3-30B-A3B-Instruct-25072026.03	47.4	72.15	46.67	75.17	65.42	81.84	36.27	68.58	53.62	62.46
Qwen3-14B2026.03	47.4	70.38	51.67	76.92	60.75	78.38	34.8	66.43	56.52	58.26
Qwen3-Next-80B-A3B-Instruct2026.03	46.6	57	57.5	67.64	59.81	66.6	28.92	44.24	59.42	61.3
Claude-3-haiku-202403072026.03	44.8	68.77	39.17	71.47	59.81	71.5	37.25	69.11	53.62	58.84
Qwen3-30B-A3B-Thinking-25072026.03	43.2	64.82	45	64.17	57.01	75.32	32.84	63.22	49.28	54.44
Ministral-3-3B-Instruct-25122026.03	39.2	62.29	43.33	67.36	51.4	66.88	28.43	60.69	44.93	51.06
GLM-4.52026.03	37	43.92	27.5	33.42	43.93	50.78	28.43	37.36	68.12	70.97
GLM-4.62026.03	28.8	33.36	20	22.33	25.23	27.73	26.96	35.07	55.07	56.23
GPT-3.5-turbo2026.03	25.4	48.63	29.17	47.33	31.78	49	14.71	50.07	40.58	46.09
Llama-3.2-3B-Instruct2026.03	24.6	52.34	20.83	51.58	24.3	46.85	24.51	59.37	31.88	41.35
Qwen3-Next-80B-A3B-Thinking2026.03	24.4	36.21	25.83	36.67	28.97	37.23	20.59	37.65	26.09	29.57
Llama-3.1-8B-Instruct2026.03	24.2	54.22	19.17	53.36	24.3	55.79	24.02	58.97	33.33	39.23
GPT-4o-mini2026.03	20.4	54.19	15.83	58.58	20.56	51.96	23.04	61.09	20.29	29.61
Ministral-8B-Instruct-24102026.03	17.4	45.09	14.17	49.31	17.76	40.31	22.55	55.38	7.25	14.78
MiniMax-M22026.03	4.2	4.83	3.33	4.17	6.54	7.17	1.96	2.68	8.7	8.7