List your company
ResearchFormerly WizWand

Over-refusal Evaluation on XSTest

100Evaluation Score (avg@4)Safechain
-4235077May 1, 2026May 9, 2026May 17, 2026May 25, 2026Jun 2, 2026Jun 10, 2026Jun 18, 2026
Updated 4mo ago

Evaluation results

MethodLinks
SafechainBase Model=DeepSeek-R1-Distill-Llama-8B2026.05100
SafechainBase Model=Qwen3-8B2026.05100
SafechainBase Model=DeepSeek-R1-Distill-Qwen-7B2026.0599.6
SAGEModel=Llama3-8B, Defense=SAGE2026.0699.6
BaseBase Model=Qwen3-8B2026.0598.8
DAPOBase Model=Qwen3-8B2026.0598.5
Self-ReSETBase Model=DeepSeek-R1-Distill-Llama-8B2026.0598.4
Self-ReSETBase Model=DeepSeek-R1-Distill-Qwen-7B2026.0596.4
BaseBase Model=DeepSeek-R1-Distill-Llama-8B2026.0596.4
Self-ReSETBase Model=Qwen3-8B2026.0596.4
RECAPBase Model=Qwen3-8B2026.0596.2
DAPOBase Model=DeepSeek-R1-Distill-Qwen-7B2026.0595.2
RECAPBase Model=DeepSeek-R1-Distill-Llama-8B2026.0594.4
RECAPBase Model=DeepSeek-R1-Distill-Qwen-7B2026.0594
BaseBase Model=DeepSeek-R1-Distill-Qwen-7B2026.0591.2
DAPOBase Model=DeepSeek-R1-Distill-Llama-8B2026.0591.2
STAR-1Base Model=Qwen3-8B2026.0587.6
STAR-1Base Model=DeepSeek-R1-Distill-Llama-8B2026.0585.2
STAR-1Base Model=DeepSeek-R1-Distill-Qwen-7B2026.0573.6
SFT mix50/1000/augModel=Gemma-3-4B2026.0566.8
SAGEModel=Qwen2.5-7B, Defense=SAGE2026.0661.2
SecDecodingModel=Qwen3-32B (Draft: 1.7B)2026.0642
SFT mix50/1000/augModel=Mistral-7B2026.0530.4
Self-ReminderModel=Llama-3.1-8B2026.0525.6
SafeDecodingModel=Qwen3-32B (Draft: 1.7B)2026.0620
PROACTModel=Qwen2.5-7B, Defense=PROACT2026.0619.7
THRDModel=Llama3-8B, Defense=Ours2026.0618.8
PROACTModel=Llama3-8B, Defense=PROACT2026.0616.8
No DefenseModel=Llama3-8B, Defense=No Defense2026.0616
Specreason+RPOModel=Qwen3-32B (Draft: 1.7B)2026.0616
SafeDecodingModel=DeepSeek-70B (Draft: 8B)2026.0616
SFT mix50/1000/augModel=Qwen3-4B2026.0513.6
THRDModel=Qwen2.5-7B, Defense=Ours2026.0612.4
RPOModel=Qwen3-32B (Draft: 1.7B)2026.0612
SafeSpecModel=DeepSeek-70B (Draft: 8B)2026.0612
Llama-Guard-3Model=Mistral-7B2026.0512
baselineModel=Mistral-7B2026.0511.2
DPO mix50/1000/augModel=Mistral-7B2026.0510.8
SafeSpecModel=Qwen3-32B (Draft: 1.7B)2026.0610
No DefenseModel=Qwen2.5-7B, Defense=No Defense2026.069.6
SmoothLLMModel=Llama-3.1-8B, K=10, q=10% swap2026.059.6
SFT mix50/200/augModel=Llama-3.1-8B2026.058.4
SpecreasonModel=Qwen3-32B (Draft: 1.7B)2026.068
SRModel=Qwen3-32B (Draft: 1.7B)2026.068
Specreason+SRModel=DeepSeek-70B (Draft: 8B)2026.068
Llama-Guard-3Model=Llama-3.1-8B2026.058
baselineModel=Qwen3-4B2026.058
DPO mix50/1000/augModel=Qwen3-4B2026.058
baselineModel=Llama-3.1-8B2026.057.2
DPO mix50/1000/augModel=Llama-3.1-8B2026.057.2
SmoothLLMModel=Mistral-7B, K=10, q=10% swap2026.057.2
SFT mix0/200/cleanModel=Qwen2.5-7B2026.056.4
baselineModel=Gemma-3-4B2026.056
DPO mix50/1000/augModel=Gemma-3-4B2026.055.6
Spec-DecodingModel=DeepSeek-70B (Draft: 8B)2026.065
RPOModel=DeepSeek-70B (Draft: 8B)2026.065
SmoothLLMModel=Gemma-3-4B, K=10, q=10% swap2026.054.8
DPO mix50/2368/aug ext. pool⋄Model=Qwen2.5-7B2026.054.5
baselineModel=Qwen2.5-7B2026.054.4
Specreason+SRModel=Qwen3-32B (Draft: 1.7B)2026.064
SRModel=DeepSeek-70B (Draft: 8B)2026.064
Specreason+RPOModel=DeepSeek-70B (Draft: 8B)2026.064
SmoothLLMModel=Qwen3-4B, K=10, q=10% swap2026.054
DPO mix50/1000/cleanModel=Qwen2.5-7B2026.053.6
No DefenseModel=DeepSeek-70B (Draft: 8B)2026.063
SecDecodingModel=DeepSeek-70B (Draft: 8B)2026.063
SmoothLLMModel=Qwen2.5-7B, K=10, q=10% swap2026.052.4
No DefenseModel=Qwen3-32B (Draft: 1.7B)2026.061
Spec-DecodingModel=Qwen3-32B (Draft: 1.7B)2026.061
SpecreasonModel=DeepSeek-70B (Draft: 8B)2026.060