Safety Refusal Evaluation on PolyRefuse yo 1.0
17.7Harmful Refusal RateOurs (HRL)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| Ours (HRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL2026.05 | 17.7 | 8.6 | 9.1 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Steering Method=None2026.05 | 20.8 | 9.8 | 11 | |
| AdaSteer (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 28.3 | 15.4 | 12.9 | |
| Ours (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 32.7 | 9.2 | 23.5 | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Steering Method=None2026.05 | 43.5 | 39.8 | 3.7 | |
| AdaSteer (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 44.8 | 47.8 | -3 | |
| CAST (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (Original)2026.05 | 46.9 | 36.2 | 10.7 | |
| CAST (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 50 | 40.8 | 9.2 | |
| CAST (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 54.7 | 43.8 | 10.9 | |
| AdaSteer (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 62.4 | 72 | -9.6 | |
| Gemma-2-9BBackbone=Gemma-2-9B, Steering Method=None2026.05 | 66.6 | 30.4 | 36.2 | |
| AdaSteer (Original)Backbone=Gemma-2-9B, Steering Method=AdaSteer (Original)2026.05 | 70.5 | 39.4 | 31.1 | |
| CAST (Original)Backbone=Gemma-2-9B, Steering Method=CAST (Original)2026.05 | 73.3 | 33.6 | 39.7 | |
| CAST (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=CAST (HRL + 32 LRL)2026.05 | 73.6 | 31.4 | 42.2 | |
| Ours (HRL)Backbone=Gemma-2-9B, Steering Method=HRL2026.05 | 76.2 | 19.6 | 56.6 | |
| CAST (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (Original)2026.05 | 76.4 | 72.4 | 4 | |
| AdaSteer (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 79.2 | 77.2 | 2 | |
| Ours (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 87.2 | 48.6 | 38.6 | |
| Ours (HRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL2026.05 | 87.6 | 48.4 | 39.2 | |
| Ours (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=HRL + 32 LRL2026.05 | 90.4 | 25.6 | 64.8 | |
| AdaSteer (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 91.8 | 89.6 | 2.2 |