Refusal Evaluation on OR-Bench
99.6Toxic Refusal RateSFT (30/70)
Evaluation Results
| Method | Links | ||
|---|---|---|---|
| SFT (30/70)Base Model=OLMo-2-1124-7B-Instruct2025.09 | 99.6 | 91.8 | |
| SFT (30/70)Base Model=Qwen2.5-7B-Instruct2025.09 | 99.5 | 98.5 | |
| SFT (5/95)Base Model=Gemma-2-9B-it2025.09 | 99.3 | 89 | |
| ASGUARDBase Model=Gemma-2-9B-it2025.09 | 99 | 88 | |
| RepBendBase Model=Gemma-2-9B-it2025.09 | 98.9 | 84.7 | |
| SFT (30/70)Base Model=Gemma-2-9B-it2025.09 | 98.7 | 94.9 | |
| ASGUARDBase Model=Qwen2.5-7B-Instruct2025.09 | 98 | 70.5 | |
| ASGUARDBase Model=OLMo-2-1124-7B-Instruct2025.09 | 97.5 | 69.2 | |
| Head AblationBase Model=Gemma-2-9B-it2025.09 | 97.2 | 73.6 | |
| RepEBase Model=Gemma-2-9B-it2025.09 | 97.1 | 70.7 | |
| Only ScalingBase Model=Llama-3.1-8B-Instruct2025.09 | 96.9 | 66.2 | |
| CBBase Model=Gemma-2-9B-it2025.09 | 96.9 | 71.1 | |
| Gemma-2-9B-itBase Model=Gemma-2-9B-it2025.09 | 96.7 | 70.5 | |
| DPOBase Model=Gemma-2-9B-it2025.09 | 96.6 | 66.8 | |
| ASGUARDBase Model=Llama-3.1-8B-Instruct2025.09 | 96.4 | 66.8 | |
| RepBendBase Model=Llama-3.1-8B-Instruct2025.09 | 96.1 | 77.9 | |
| SFT (5/95)Base Model=OLMo-2-1124-7B-Instruct2025.09 | 94.8 | 57.9 | |
| Only ScalingBase Model=Qwen2.5-7B-Instruct2025.09 | 94.3 | 42.6 | |
| SFT (5/95)Base Model=Llama-3.1-8B-Instruct2025.09 | 94.1 | 50.8 | |
| DPOBase Model=OLMo-2-1124-7B-Instruct2025.09 | 93.2 | 48.3 | |
| Only ScalingBase Model=OLMo-2-1124-7B-Instruct2025.09 | 92.8 | 48.8 | |
| OLMo-2-1124-7B-InstructBase Model=OLMo-2-1124-7B-Instruct2025.09 | 92.5 | 43.5 | |
| Head AblationBase Model=OLMo-2-1124-7B-Instruct2025.09 | 92.5 | 45.1 | |
| CBBase Model=OLMo-2-1124-7B-Instruct2025.09 | 92.3 | 43.1 | |
| RepBendBase Model=OLMo-2-1124-7B-Instruct2025.09 | 92 | 42.8 | |
| SFT (30/70)Base Model=Llama-3.1-8B-Instruct2025.09 | 91.9 | 80.3 | |
| Only ScalingBase Model=Gemma-2-9B-it2025.09 | 91.9 | 72.4 | |
| RepEBase Model=OLMo-2-1124-7B-Instruct2025.09 | 91.4 | 43.2 | |
| SFT (5/95)Base Model=Qwen2.5-7B-Instruct2025.09 | 91.1 | 38.8 | |
| DPOBase Model=Llama-3.1-8B-Instruct2025.09 | 90.2 | 33.2 | |
| Llama-3.1-8B-InstructBase Model=Llama-3.1-8B-Instruct2025.09 | 88.5 | 28.9 | |
| RepEBase Model=Llama-3.1-8B-Instruct2025.09 | 87.9 | 29.7 | |
| Head AblationBase Model=Llama-3.1-8B-Instruct2025.09 | 86.7 | 34.7 | |
| CBBase Model=Llama-3.1-8B-Instruct2025.09 | 85.1 | 84.8 | |
| Head AblationBase Model=Qwen2.5-7B-Instruct2025.09 | 80.1 | 14.6 | |
| CBBase Model=Qwen2.5-7B-Instruct2025.09 | 79.8 | 13.1 | |
| Qwen2.5-7B-InstructBase Model=Qwen2.5-7B-Instruct2025.09 | 79.5 | 12.9 | |
| DPOBase Model=Qwen2.5-7B-Instruct2025.09 | 79.3 | 15 | |
| RepEBase Model=Qwen2.5-7B-Instruct2025.09 | 79.3 | 12.8 | |
| RepBendBase Model=Qwen2.5-7B-Instruct2025.09 | 75.4 | 12.2 |