Safety Refusal Evaluation on PolyRefuse am 1.0
93.2Harmful Refusal RateAdaSteer (HRL + 32 LRL)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| AdaSteer (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 93.2 | 84.2 | 9 | |
| Ours (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=HRL + 32 LRL2026.05 | 91.6 | 16.4 | 75.2 | |
| Ours (HRL)Backbone=Gemma-2-9B, Steering Method=HRL2026.05 | 84.3 | 12.6 | 71.7 | |
| CAST (Original)Backbone=Gemma-2-9B, Steering Method=CAST (Original)2026.05 | 77.6 | 18 | 59.6 | |
| CAST (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=CAST (HRL + 32 LRL)2026.05 | 74.3 | 15.8 | 58.5 | |
| Gemma-2-9BBackbone=Gemma-2-9B, Steering Method=None2026.05 | 72.7 | 14.4 | 58.3 | |
| AdaSteer (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 71.9 | 81.6 | -9.7 | |
| AdaSteer (Original)Backbone=Gemma-2-9B, Steering Method=AdaSteer (Original)2026.05 | 70.5 | 29.6 | 40.9 | |
| CAST (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (Original)2026.05 | 58.9 | 53 | 5.9 | |
| AdaSteer (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 40 | 72.4 | -32.4 | |
| Ours (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 32.3 | 8.4 | 23.9 | |
| Ours (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 32.2 | 10.4 | 21.8 | |
| CAST (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 26.4 | 18.8 | 7.6 | |
| Ours (HRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL2026.05 | 23.3 | 6.2 | 17.1 | |
| AdaSteer (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 16.8 | 15 | 1.8 | |
| CAST (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (Original)2026.05 | 10.3 | 10.2 | 0.1 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Steering Method=None2026.05 | 9.8 | 7 | 2.8 | |
| AdaSteer (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 9.3 | 20.6 | -11.3 | |
| Ours (HRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL2026.05 | 8.9 | 6 | 2.9 | |
| CAST (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 6.1 | 6 | 0.1 | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Steering Method=None2026.05 | 4.4 | 5.4 | -1 |