Over-refusal Accuracy on XSTest and OKTest
99.2Over-refusal Accuracy (XSTest)ReSA-RL (Ours)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| ReSA-RL (Ours)Defense Category=Answer-Then-Check2025.09 | 99.2 | 95.33 | 97.27 | |
| OpenAI-Deliberative Alignment*Defense Category=Fine-tuning defense2025.09 | 97.2 | 87.67 | 92.44 | |
| ReSA-SFT (Ours)Defense Category=Answer-Then-Check2025.09 | 97.2 | 88.67 | 92.94 | |
| deepseek-v3-20250324Defense Category=SOTA General LLM2025.09 | 94.8 | 91 | 92.9 | |
| gpt-4.1-20250414Defense Category=SOTA General LLM2025.09 | 94.4 | 89.33 | 91.87 | |
| GuardReasonerDefense Category=Post-hoc defense2025.09 | 93.2 | 84 | 88.6 | |
| gpt-4.1-20250414Defense Category=SOTA General LLM with goal priority defense2025.09 | 90.8 | 90.33 | 90.57 | |
| o4-mini-20250416Defense Category=SOTA Reasoning LLM with Safety Reflection2025.09 | 90 | 91 | 90.5 | |
| claude-sonnet-4-20250514Defense Category=SOTA General LLM2025.09 | 89.6 | 74.33 | 81.97 | |
| deepseek-v3-20250324Defense Category=SOTA General LLM with goal priority defense2025.09 | 81.2 | 80.33 | 80.77 | |
| deepseek-r1-20250528Defense Category=SOTA Reasoning LLM with Safety Reflection2025.09 | 80.8 | 66 | 73.4 | |
| Realsafe-r1Defense Category=Fine-tuning defense2025.09 | 51.6 | 59.67 | 55.65 |