Tool-augmented agent execution on MCP-Bench

53.5Task FulfillmentGround Truth Server

Evaluation Results

Method
Ground Truth Server2026.06	53.5	72.6	74.6	74.4	62.3	55.4	—	—	—	—	—	—	73.8
SING2026.06	50.5	67.1	67.4	69	60.8	35.3	—	—	—	—	—	—	67.5
SING2026.06	50.1	66.8	71.4	68.3	60.1	33.4	—	—	—	—	—	—	68.4
MCP-Zero2026.06	49.5	62.9	68.9	68.3	58.7	34.9	—	—	—	—	—	—	67.6
MCP-Zero2026.06	48.2	61.1	68	68	57.7	33	—	—	—	—	—	—	66.7
ReAct2026.05	46.8	36.6	41.5	31.2	40.5	19.8	77.7	0	41.7	36.4	30.1	38.9	36.8
ReAct-Plan-Exec2026.05	38.5	38.7	42.5	47.8	36.9	20.6	100	75.4	38.6	45.2	28.8	87.7	50
SPIRAL-Exec2026.05	34.5	47.5	55.3	58.4	33.3	27.5	100	79.9	41	56.9	30.4	89.9	54.6
ReAct-Plan-Exec2026.05	33.7	42.6	45.2	40.4	29.6	26.8	100	85.8	38.2	42.8	28.2	92.9	50.5
SPIRAL-Exec2026.05	32.8	51	59.6	61	33.6	30.9	100	82.8	41.9	60.3	32.2	91.4	56.5
SPIRAL-Exec2026.05	31.6	26.6	33.6	48.4	20.5	20.2	100	100	29.1	41	20.3	100	47.6
ReAct-Plan-Exec2026.05	31.4	47.2	53.1	65.9	27.1	29.9	100	78.4	39.3	59.5	28.5	89.2	54.1
Qwen2.5-4B2026.06	2.4	3.58	3.42	3.78	2.48	1.45	—	—	2.99	3.6	1.97	—	2.85
Qwen2.5-4B2026.06	1.97	2.32	3.17	3.75	1.43	1.38	—	—	2.14	3.46	1.41	—	2.34
SmolLM2-3B2026.06	1.48	2.18	2.44	2.51	1.41	1.07	—	—	1.83	2.48	1.24	—	1.85
Llama-3.2-3B2026.06	1.32	2.43	2.62	2.85	1.17	1.1	—	—	1.88	2.73	1.13	—	1.91
Llama-3.2-3B2026.06	1.25	1.65	1.82	1.82	1.45	0.95	—	—	1.45	1.82	1.2	—	1.49
DeepSeek-R1-Distill-Qwen-1.5B2026.06	1.05	1.33	1.02	1.16	1	1	—	—	1.19	1.09	1	—	1
Qwen2.5-4B2026.06	0.946	1.53	2.01	2.95	1.05	0.929	—	—	1.24	2.48	0.987	—	1.57
Qwen2.5-4B2026.06	0.933	1.37	2.33	2.48	0.9	0.817	—	—	1.15	2.41	0.858	—	1.47
SmolLM2-3B2026.06	0.817	1.05	0.983	1.13	0.767	0.75	—	—	0.933	1.06	0.758	—	0.917
Llama-3.2-3B2026.06	0.767	1.45	1.2	0.8	0.8	0.717	—	—	1.11	1	0.758	—	0.956
Qwen2.5-4B2026.06	0.737	1.35	1.09	1.6	0.671	0.554	—	—	1.05	1.35	0.612	—	1
DeepSeek-R1-Distill-Qwen-1.5B2026.06	0.727	0.855	0.709	0.891	0.673	0.673	—	—	0.791	0.8	0.673	—	0.692
Llama-3.2-3B2026.06	0.717	1.02	1.28	1.23	0.733	0.75	—	—	0.867	1.26	0.742	—	0.956
SmolLM2-3B2026.06	0.675	0.771	0.938	1.7	0.638	0.638	—	—	0.723	1.32	0.638	—	0.892
SmolLM2-3B2026.06	0.65	0.867	0.617	0.633	0.583	0.567	—	—	0.758	0.625	0.575	—	0.653
DeepSeek-R1-Distill-Qwen-1.5B2026.06	0.6	0.733	0.567	0.567	0.567	0.567	—	—	0.667	0.567	0.567	—	0.6
SmolLM2-3B2026.06	0.567	0.55	0.55	0.55	0.55	0.55	—	—	0.558	0.55	0.55	—	0.553
Llama-3.2-3B2026.06	0.55	0.85	0.917	0.867	0.583	0.583	—	—	0.7	0.892	0.583	—	0.725
DeepSeek-R1-Distill-Qwen-1.5B2026.06	0.458	1.28	0.625	0.608	0.458	0.458	—	—	0.871	0.617	0.458	—	0.649
DeepSeek-R1-Distill-Qwen-1.5B2026.06	0.433	0.6	0.45	0.433	0.433	0.433	—	—	0.517	0.442	0.433	—	0.464