Over-refusal Evaluation on XSTest and CoCoNot
2.6XSTest Refusal RateHARC
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| HARCBackbone=QWEN-2.5-7B2026.07 | 2.6 | 4.2 | |
| HARCBackbone=LLAMA-3.1-8B2026.07 | 3.5 | 8.1 | |
| Circuit BreakerBackbone=QWEN-2.5-7B2026.07 | 5.7 | 8.1 | |
| HARC + DPOBackbone=QWEN-2.5-7B2026.07 | 5.7 | 9.4 | |
| BaselineBackbone=QWEN-2.5-7B2026.07 | 9.1 | 9.4 | |
| HARC + DPOBackbone=LLAMA-3.1-8B2026.07 | 10 | 19.5 | |
| BaselineBackbone=LLAMA-3.1-8B2026.07 | 10.9 | 7.4 | |
| Circuit BreakerBackbone=LLAMA-3.1-8B2026.07 | 11.3 | 5.4 | |
| CASTBackbone=QWEN-2.5-7B2026.07 | 12.6 | 13.4 | |
| DPOBackbone=QWEN-2.5-7B2026.07 | 16.5 | 15.4 | |
| SFTBackbone=LLAMA-3.1-8B2026.07 | 23.9 | 27.5 | |
| CASTBackbone=LLAMA-3.1-8B2026.07 | 28.3 | 26.2 | |
| SFTBackbone=QWEN-2.5-7B2026.07 | 33.9 | 40.9 | |
| DPOBackbone=LLAMA-3.1-8B2026.07 | 48.8 | 22.1 | |
| RepBendBackbone=LLAMA-3.1-8B2026.07 | 87.8 | 80.5 | |
| RepBendBackbone=QWEN-2.5-7B2026.07 | 100 | 100 |