Over-refusal on Wildjailbreak (Benign)
1.43Wildjailbreak Benign Refusal RateCategorical Steering
Evaluation results
| Method | Links | ||
|---|---|---|---|
| Categorical SteeringBackbone=REFUSE-LLAMA, Method=Categorical Steering2026.03 | 1.43 | — | |
| Low-Rank CombinationBackbone=REFUSE-LLAMA, Method=Low-Rank Combination2026.03 | 3.33 | — | |
| REFUSE-LLAMABackbone=REFUSE-LLAMA2026.03 | 4.76 | — | |
| Low-Rank CombinationBackbone=LLAMA 3 8B INSTRUCT, Method=Low-Rank Combination2026.03 | 8.57 | — | |
| LLAMA 3 8B INSTRUCTBackbone=LLAMA 3 8B INSTRUCT2026.03 | 12.38 | — | |
| DPOBackbone=Llama3 8B Instruct2025.04 | 28.4 | — | |
| NPOModel=Llama3 8B Instruct2025.04 | 43.2 | 71.65 | |
| NPOBackbone=Llama3 8B Instruct2025.04 | 43.2 | — | |
| NPO (W/O safe set)Backbone=Mistral 7B Instruct-v0.22025.04 | 50 | — | |
| CBModel=Llama3 8B Instruct, safety-tuned=true2025.04 | 52.4 | 75.87 | |
| CB*Backbone=Llama3 8B Instruct, publicly_available_safety_tuned=true2025.04 | 52.4 | — | |
| Safety PromptingBackbone=Llama3 8B Instruct2025.04 | 65.6 | — | |
| Low-Rank CombinationBackbone=DEEPSEEK R1 DISTILL LLAMA, Method=Low-Rank Combination2026.03 | 66.67 | — | |
| NPOModel=Mistral 7B Instruct v0.22025.04 | 70 | 74.52 | |
| NPOBackbone=Mistral 7B Instruct-v0.22025.04 | 70 | — | |
| DEEPSEEK R1 DISTILL LLAMABackbone=DEEPSEEK R1 DISTILL LLAMA2026.03 | 72.38 | — | |
| RMUModel=Llama3 8B Instruct2025.04 | 72.4 | 72.9 | |
| RMUBackbone=Llama3 8B Instruct2025.04 | 72.4 | — | |
| NPO (W/O safe set)Backbone=Llama3 8B Instruct2025.04 | 76 | — | |
| CBModel=Mistral 7B Instruct v0.2, safety-tuned=true2025.04 | 82 | 73.62 | |
| CB*Backbone=Mistral 7B Instruct-v0.2, publicly_available_safety_tuned=true2025.04 | 82 | — | |
| WHPBackbone=Llama3 8B Instruct2025.04 | 83.6 | — | |
| TAModel=Llama3 8B Instruct2025.04 | 88.8 | 70.71 | |
| TABackbone=Llama3 8B Instruct2025.04 | 88.8 | — | |
| CBModel=Llama3 8B Instruct2025.04 | 89.2 | 81.61 | |
| REPBENDModel=Llama3 8B Instruct2025.04 | 89.2 | 83.14 | |
| CBBackbone=Llama3 8B Instruct2025.04 | 89.2 | — | |
| REPBENDBackbone=Llama3 8B Instruct2025.04 | 89.2 | — | |
| RMUModel=Mistral 7B Instruct v0.22025.04 | 90.4 | 71.85 | |
| RMUBackbone=Mistral 7B Instruct-v0.22025.04 | 90.4 | — | |
| SFTBackbone=Llama3 8B Instruct2025.04 | 90.8 | — | |
| Original Weight ModelModel=Llama3 8B Instruct2025.04 | 92 | 73.9 | |
| OriginalBackbone=Llama3 8B Instruct2025.04 | 92 | — | |
| WHP (W/O safe set)Backbone=Llama3 8B Instruct2025.04 | 92.4 | — | |
| REPBENDModel=Mistral 7B Instruct v0.22025.04 | 93.6 | 81.23 | |
| REPBENDBackbone=Mistral 7B Instruct-v0.22025.04 | 93.6 | — | |
| R2D2Model=Mistral 7B Instruct v0.2, safety-tuned=true2025.04 | 96.8 | 72.67 | |
| SFTBackbone=Mistral 7B Instruct-v0.22025.04 | 96.8 | — | |
| R2D2*Backbone=Mistral 7B Instruct-v0.2, publicly_available_safety_tuned=true2025.04 | 96.8 | — | |
| TAModel=Mistral 7B Instruct v0.22025.04 | 97.6 | 72.96 | |
| CBModel=Mistral 7B Instruct v0.22025.04 | 97.6 | 77.34 | |
| TABackbone=Mistral 7B Instruct-v0.22025.04 | 97.6 | — | |
| CBBackbone=Mistral 7B Instruct-v0.22025.04 | 97.6 | — | |
| WHPBackbone=Mistral 7B Instruct-v0.22025.04 | 98.4 | — | |
| Safety PromptingBackbone=Mistral 7B Instruct-v0.22025.04 | 98.8 | — | |
| DPOBackbone=Mistral 7B Instruct-v0.22025.04 | 99.6 | — | |
| WHP (W/O safe set)Backbone=Mistral 7B Instruct-v0.22025.04 | 99.6 | — | |
| Original Weight ModelModel=Mistral 7B Instruct v0.22025.04 | 100 | 63.81 | |
| OriginalBackbone=Mistral 7B Instruct-v0.22025.04 | 100 | — | |
| Original Weight ModelModel=Gemma2 2B Instruct2025.04 | — | 98.8 | |
| Original Weight ModelModel=Qwen2.5 14B Instruct2025.04 | — | 100 | |
| REPBENDModel=Gemma2 2B Instruct2025.04 | — | 82.8 | |
| REPBENDModel=Qwen2.5 14B Instruct2025.04 | — | 99.6 |