Over-refusal Evaluation Suite (XSTest, WildJailbreak, WildGuard, OKTest, OR-Bench) benchmark leaderboard · SOTA2 Research

Over-refusal Evaluation Suite (XSTest, WildJailbreak, WildGuard, OKTest, OR-Bench)

3.2XSTest Refusal Rate (%)Pure

Evaluation Results

Method	Links
Pure2026.02		3.2	0	1.9	5.33	19.79	6.04
Pure2026.02		3.2	0.48	1.27	5.33	15.09	5.07
Pure2026.02		3.6	1.43	1.48	7.33	15.62	5.89
Short2026.02		4.4	1.43	2.22	12.67	41.93	12.53
SafeChain2026.02		4.4	2.38	3.92	9.33	10.61	6.13
Short2026.02		4.8	3.33	4.23	10.33	40.86	12.71
Short2026.02		5.2	3.81	4.02	12.33	46.85	14.44
SafeChain2026.02		6.4	0	3.17	7.67	9.25	5.3
IFPO2026.02		7.2	1.43	3.6	9.67	7.28	5.84
IFPO2026.02		7.6	2.38	4.34	9.67	5.99	6
IFPO2026.02		7.6	2.38	4.34	9.67	5.99	4.33
STAR-1-mix2026.02		8	1.9	4.13	12.67	48.29	15
SafeChain2026.02		8.8	5.24	4.97	9.33	12.81	8.23
Agent2026.02		10.4	6.67	6.88	14	79.15	23.42
STAR-1-mix2026.02		10.8	0.48	4.23	10.33	50.34	15.24
BC2026.02		11.2	4.76	5.71	14.67	48.6	16.99
Agent2026.02		11.6	7.62	7.3	15.33	71.72	22.71
BC2026.02		12.4	1.9	4.44	12.33	43.82	14.98
STAR-12026.02		12.8	2.86	6.14	17	55.12	18.78
BC2026.02		13.6	3.81	4.66	12.33	47.23	16.33
STAR-12026.02		14	3.81	6.67	15.67	54.51	18.93
Agent2026.02		14	19.05	11.75	16.67	80.59	28.41
STAR-1-mix2026.02		16	1.43	4.97	11.33	57.16	18.18
STAR-12026.02		19.2	1.9	7.41	18.33	63.31	22.03