Safety Refusal Evaluation on PolyRefuse 1.0 (sw)
96.7Harmful Refusal RateOurs (HRL + 32 LRL)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Ours (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=HRL + 32 LRL2026.05 | 96.7 | 3.6 | 93.1 | |
| Ours (HRL)Backbone=Gemma-2-9B, Steering Method=HRL2026.05 | 95.5 | 3.8 | 91.7 | |
| CAST (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=CAST (HRL + 32 LRL)2026.05 | 88.3 | 1 | 87.3 | |
| CAST (Original)Backbone=Gemma-2-9B, Steering Method=CAST (Original)2026.05 | 87.6 | 1 | 86.6 | |
| AdaSteer (Original)Backbone=Gemma-2-9B, Steering Method=AdaSteer (Original)2026.05 | 86.4 | 4 | 82.4 | |
| Gemma-2-9BBackbone=Gemma-2-9B, Steering Method=None2026.05 | 86.2 | 1 | 85.2 | |
| AdaSteer (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 83.4 | 79.2 | 4.2 | |
| AdaSteer (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 73.3 | 9.4 | 63.9 | |
| Ours (HRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL2026.05 | 56.6 | 10.6 | 46 | |
| Ours (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 56.6 | 10.4 | 46.2 | |
| Ours (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 53.7 | 1.4 | 52.3 | |
| CAST (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (Original)2026.05 | 50 | 37.6 | 12.4 | |
| AdaSteer (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 46.2 | 5 | 41.2 | |
| CAST (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 45.1 | 10.4 | 34.7 | |
| AdaSteer (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 43.5 | 25.8 | 17.7 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Steering Method=None2026.05 | 38.6 | 0.8 | 37.8 | |
| CAST (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (Original)2026.05 | 29.2 | 12.8 | 16.4 | |
| Ours (HRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL2026.05 | 29 | 1 | 28 | |
| AdaSteer (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 26.7 | 16.4 | 10.3 | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Steering Method=None2026.05 | 22.9 | 9 | 13.9 | |
| CAST (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 21 | 7 | 14 |