Over-refusal Evaluation on XSTest
100Evaluation Score (avg@4)Safechain
Evaluation Results
| Method | Links | |
|---|---|---|
| SafechainBase Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 100 | |
| SafechainBase Model=Qwen3-8B2026.05 | 100 | |
| SafechainBase Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 99.6 | |
| SAGEModel=Llama3-8B, Defense=SAGE2026.06 | 99.6 | |
| BaseBase Model=Qwen3-8B2026.05 | 98.8 | |
| DAPOBase Model=Qwen3-8B2026.05 | 98.5 | |
| Self-ReSETBase Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 98.4 | |
| Self-ReSETBase Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 96.4 | |
| BaseBase Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 96.4 | |
| Self-ReSETBase Model=Qwen3-8B2026.05 | 96.4 | |
| RECAPBase Model=Qwen3-8B2026.05 | 96.2 | |
| DAPOBase Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 95.2 | |
| RECAPBase Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 94.4 | |
| RECAPBase Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 94 | |
| BaseBase Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 91.2 | |
| DAPOBase Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 91.2 | |
| STAR-1Base Model=Qwen3-8B2026.05 | 87.6 | |
| STAR-1Base Model=DeepSeek-R1-Distill-Llama-8B2026.05 | 85.2 | |
| STAR-1Base Model=DeepSeek-R1-Distill-Qwen-7B2026.05 | 73.6 | |
| SFT mix50/1000/augModel=Gemma-3-4B2026.05 | 66.8 | |
| SAGEModel=Qwen2.5-7B, Defense=SAGE2026.06 | 61.2 | |
| SecDecodingModel=Qwen3-32B (Draft: 1.7B)2026.06 | 42 | |
| SFT mix50/1000/augModel=Mistral-7B2026.05 | 30.4 | |
| Self-ReminderModel=Llama-3.1-8B2026.05 | 25.6 | |
| SafeDecodingModel=Qwen3-32B (Draft: 1.7B)2026.06 | 20 | |
| PROACTModel=Qwen2.5-7B, Defense=PROACT2026.06 | 19.7 | |
| THRDModel=Llama3-8B, Defense=Ours2026.06 | 18.8 | |
| PROACTModel=Llama3-8B, Defense=PROACT2026.06 | 16.8 | |
| No DefenseModel=Llama3-8B, Defense=No Defense2026.06 | 16 | |
| Specreason+RPOModel=Qwen3-32B (Draft: 1.7B)2026.06 | 16 | |
| SafeDecodingModel=DeepSeek-70B (Draft: 8B)2026.06 | 16 | |
| SFT mix50/1000/augModel=Qwen3-4B2026.05 | 13.6 | |
| THRDModel=Qwen2.5-7B, Defense=Ours2026.06 | 12.4 | |
| RPOModel=Qwen3-32B (Draft: 1.7B)2026.06 | 12 | |
| SafeSpecModel=DeepSeek-70B (Draft: 8B)2026.06 | 12 | |
| Llama-Guard-3Model=Mistral-7B2026.05 | 12 | |
| baselineModel=Mistral-7B2026.05 | 11.2 | |
| DPO mix50/1000/augModel=Mistral-7B2026.05 | 10.8 | |
| SafeSpecModel=Qwen3-32B (Draft: 1.7B)2026.06 | 10 | |
| No DefenseModel=Qwen2.5-7B, Defense=No Defense2026.06 | 9.6 | |
| SmoothLLMModel=Llama-3.1-8B, K=10, q=10% swap2026.05 | 9.6 | |
| SFT mix50/200/augModel=Llama-3.1-8B2026.05 | 8.4 | |
| SpecreasonModel=Qwen3-32B (Draft: 1.7B)2026.06 | 8 | |
| SRModel=Qwen3-32B (Draft: 1.7B)2026.06 | 8 | |
| Specreason+SRModel=DeepSeek-70B (Draft: 8B)2026.06 | 8 | |
| Llama-Guard-3Model=Llama-3.1-8B2026.05 | 8 | |
| baselineModel=Qwen3-4B2026.05 | 8 | |
| DPO mix50/1000/augModel=Qwen3-4B2026.05 | 8 | |
| baselineModel=Llama-3.1-8B2026.05 | 7.2 | |
| DPO mix50/1000/augModel=Llama-3.1-8B2026.05 | 7.2 | |
| SmoothLLMModel=Mistral-7B, K=10, q=10% swap2026.05 | 7.2 | |
| SFT mix0/200/cleanModel=Qwen2.5-7B2026.05 | 6.4 | |
| baselineModel=Gemma-3-4B2026.05 | 6 | |
| DPO mix50/1000/augModel=Gemma-3-4B2026.05 | 5.6 | |
| Spec-DecodingModel=DeepSeek-70B (Draft: 8B)2026.06 | 5 | |
| RPOModel=DeepSeek-70B (Draft: 8B)2026.06 | 5 | |
| SmoothLLMModel=Gemma-3-4B, K=10, q=10% swap2026.05 | 4.8 | |
| DPO mix50/2368/aug ext. pool⋄Model=Qwen2.5-7B2026.05 | 4.5 | |
| baselineModel=Qwen2.5-7B2026.05 | 4.4 | |
| Specreason+SRModel=Qwen3-32B (Draft: 1.7B)2026.06 | 4 | |
| SRModel=DeepSeek-70B (Draft: 8B)2026.06 | 4 | |
| Specreason+RPOModel=DeepSeek-70B (Draft: 8B)2026.06 | 4 | |
| SmoothLLMModel=Qwen3-4B, K=10, q=10% swap2026.05 | 4 | |
| DPO mix50/1000/cleanModel=Qwen2.5-7B2026.05 | 3.6 | |
| No DefenseModel=DeepSeek-70B (Draft: 8B)2026.06 | 3 | |
| SecDecodingModel=DeepSeek-70B (Draft: 8B)2026.06 | 3 | |
| SmoothLLMModel=Qwen2.5-7B, K=10, q=10% swap2026.05 | 2.4 | |
| No DefenseModel=Qwen3-32B (Draft: 1.7B)2026.06 | 1 | |
| Spec-DecodingModel=Qwen3-32B (Draft: 1.7B)2026.06 | 1 | |
| SpecreasonModel=DeepSeek-70B (Draft: 8B)2026.06 | 0 |