Hidden Objective Monitoring on Prompt-Manipulation A1 Monitor Notice benchmark leaderboard · SOTA2 Research

Hidden Objective Monitoring on Prompt-Manipulation A1 Monitor Notice

100Exit Detection RateGemini-1.5-Flash

Evaluation Results

Method	Links
Gemini-1.5-Flash2026.05		100	100	100	20.18	88.15	96.86	100	100	100	89.47
Gemini-1.5-Flash2026.05		100	100	100	0	62.75	98.7	99	100	100	84.49
Gemini-1.5-Flash2026.05		100	100	100	49.93	99.02	98.74	100	100	100	94.19
CoT-Guard 4B2026.05		92.89	84.62	98.44	1.63	15.85	99.12	76.54	63.45	98.22	70.08
CoT-Guard 4B2026.05		92.15	94.87	100	0	18.2	100	68.9	99.2	98.22	74.62
Gemini-1.5-Flash2026.05		88.82	100	100	0	1.96	96.86	78.76	93.06	98.93	73.15
Qwen2-1.5B2026.05		80.26	92.31	97.67	0.81	6.04	98.11	72.99	9.86	88.92	60.77
GPT-4o2026.05		74.2	100	100	0	0	83.8	11.7	88	97.33	61.67
Qwen2-1.5B2026.05		62.56	87.18	99.61	2.46	22.1	98.2	22.8	95.65	97.63	65.35
Qwen2-72B2026.05		50	97.44	89.51	0	1.17	84.28	1.63	16.97	14.84	39.54
Qwen2-1.5B2026.05		38.82	100	98.15	0	0.39	89.31	1.3	1.07	9.07	37.57
GPT-4o-mini2026.05		36.18	90	79.93	0	0.59	40.9	11.4	70.4	47.2	41.84
Qwen2-7B2026.05		32.89	97.44	90.12	0.82	1.96	86.16	1.62	0.27	15.2	36.28
Qwen2-72B2026.05		28.29	94.87	89.77	0	1.36	84.8	35.6	78.48	76.61	54.42
Llama3.1-8B-Instruct2026.05		19	38.3	36.1	7.7	17	37.5	20.6	11.9	32.5	24.51
Qwen2-7B2026.05		17.76	85.38	79.32	0.82	0.98	47.2	0.7	59.2	50.93	38.03
Qwen2-1.5B2026.05		12.77	79.49	54.92	1.73	2.1	63.8	1.7	55.92	42.06	34.94
Qwen2-1.5B2026.05		5.26	70	17.76	0	0	54.86	0.32	0	1.06	16.58
Qwen2-1.5B2026.05		3.29	70	25.93	0	0	8.8	0	24	14.6	16.29
GPT-4o-mini2026.05		1.97	85	26.54	0	0	75.47	0	0	0.27	21.03
Llama3.1-8B-Instruct2026.05		0.6	0	0	0.9	0.4	1.4	0.4	1.8	2.4	0.88
GPT-4o2026.05		0	100	87.1	0	0	83.33	0	0	0	30.05