Tool-use agent security evaluation on SIREN benchmark leaderboard · SOTA2 Research

Tool-use agent security evaluation on SIREN

23.56Explicit Directive (UA)CaMeL

Evaluation Results

Method	Links
CaMeL2026.01		23.56	44.83	3.8	20.11	17.88	25.14	10.66	30.51	2	0	11.68	25.34	24.87	0	46.79
VIGIL2026.01		17.24	16.09	52.17	1.09	21.79	24.02	28.31	0	14.67	3.33	27.53	8.13	40.57	0.32	74.49
CaMeL2026.01		17.24	45.98	4.35	16.3	13.97	37.43	13.97	33.09	1.33	0	10.74	27.84	26.55	0	30.84
Vanilla ReAct2026.01		15.52	64.94	11.96	69.57	15.08	54.19	12.87	56.62	8.67	48.67	12.93	58.92	30.56	16.65	65.31
VIGIL2026.01		14.37	22.99	25.54	1.63	17.88	37.99	19.85	0	12.67	2.67	18.46	11.99	39.3	0.21	40.82
DRIFT2026.01		10.92	49.43	13.59	59.78	13.41	55.87	23.16	5.51	14	4	15.85	33.06	47.63	10.22	55.1
Spotlighting2026.01		10.34	62.07	10.87	69.02	15.64	48.04	11.4	32.72	9.33	52	11.57	50.89	21.29	9.8	73.47
MELON2026.01		8.62	63.22	21.74	54.35	12.29	47.49	15.44	3.31	9.33	6	13.87	32.64	24.66	0.42	43.88
DRIFT2026.01		8.05	62.07	25	28.26	10.06	64.8	17.28	6.25	10	13.33	14.6	32.64	59.75	14.12	76.53
DeBERTa2026.01		5.75	58.62	15.76	40.76	7.26	40.22	14.71	9.93	4.67	28.67	10.32	33.26	8.85	1.48	34.69
Tool-Filter2026.01		4.02	40.23	5.43	1.09	5.59	22.91	11.4	13.6	7.33	18.67	7.19	18.56	6.53	2.32	48.98
Vanilla ReAct2026.01		3.45	88.51	26.09	71.2	25.7	65.36	12.13	75.37	13.33	67.33	15.95	73.83	39.52	38.88	79.59
Tool-Filter2026.01		2.87	49.43	3.26	0.54	2.79	22.35	9.56	0.37	4.67	43.33	5.11	20.13	7.48	0.11	45.92
Spotlighting2026.01		2.3	87.93	52.72	40.22	29.05	74.3	11.03	58.82	8	60	20.33	63.61	43.94	39.83	77.55
DeBERTa2026.01		2.3	90.23	16.3	58.15	10.61	66.48	15.81	7.72	4	32.67	10.64	47.24	21.29	8.11	43.88
MELON2026.01		1.72	87.93	46.74	37.5	24.58	61.45	18.38	0.74	2.67	12	19.5	36.7	35.63	0.21	71.43