Safety Refusal Evaluation on PolyRefuse 1.0 (km)
90.9Harmful Refusal RateOurs (HRL + 32 LRL)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Ours (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=HRL + 32 LRL2026.05 | 90.9 | 8.2 | 82.7 | |
| AdaSteer (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 87.2 | 87.8 | -0.6 | |
| Ours (HRL)Backbone=Gemma-2-9B, Steering Method=HRL2026.05 | 86.2 | 6.4 | 79.8 | |
| CAST (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=CAST (HRL + 32 LRL)2026.05 | 81.1 | 6 | 75.1 | |
| AdaSteer (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 79.9 | 53 | 26.9 | |
| CAST (Original)Backbone=Gemma-2-9B, Steering Method=CAST (Original)2026.05 | 79.2 | 6 | 73.2 | |
| Ours (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 74.3 | 15.6 | 58.7 | |
| Gemma-2-9BBackbone=Gemma-2-9B, Steering Method=None2026.05 | 74.3 | 5.4 | 68.9 | |
| AdaSteer (Original)Backbone=Gemma-2-9B, Steering Method=AdaSteer (Original)2026.05 | 74 | 17 | 57 | |
| CAST (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (Original)2026.05 | 70.6 | 58 | 12.6 | |
| AdaSteer (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 63.3 | 49.6 | 13.7 | |
| Ours (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 58.2 | 4.8 | 53.4 | |
| AdaSteer (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 55.8 | 13.6 | 42.2 | |
| Ours (HRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL2026.05 | 55.6 | 8.6 | 47 | |
| CAST (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 53.1 | 23.6 | 29.5 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Steering Method=None2026.05 | 44.8 | 5.2 | 39.6 | |
| CAST (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (Original)2026.05 | 43.4 | 13 | 30.4 | |
| AdaSteer (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 40.7 | 13.2 | 27.5 | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Steering Method=None2026.05 | 31.1 | 7 | 24.1 | |
| CAST (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 29.2 | 5.8 | 23.4 | |
| Ours (HRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL2026.05 | 25.3 | 3.6 | 21.7 |