Safety Refusal Evaluation on PolyRefuse 1.0 (si)
90.2Harmful Refusal RateOurs (HRL + 32 LRL)
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| Ours (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=HRL + 32 LRL2026.05 | 90.2 | 8 | 82.2 | |
| CAST (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (Original)2026.05 | 88.5 | 80.6 | 7.9 | |
| AdaSteer (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 86 | 92.2 | -6.2 | |
| Ours (HRL)Backbone=Gemma-2-9B, Steering Method=HRL2026.05 | 81.8 | 5.2 | 76.6 | |
| CAST (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 81.1 | 57.2 | 23.9 | |
| CAST (Original)Backbone=Gemma-2-9B, Steering Method=CAST (Original)2026.05 | 75.2 | 8.4 | 66.8 | |
| Ours (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 74.7 | 25.6 | 49.1 | |
| AdaSteer (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 73.3 | 53.8 | 19.5 | |
| CAST (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=CAST (HRL + 32 LRL)2026.05 | 72.2 | 8.6 | 63.6 | |
| Gemma-2-9BBackbone=Gemma-2-9B, Steering Method=None2026.05 | 67 | 5.8 | 61.2 | |
| Ours (HRL + 32 LRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL + 32 LRL2026.05 | 64.2 | 8 | 56.2 | |
| Ours (HRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=HRL2026.05 | 61.2 | 16.8 | 44.4 | |
| AdaSteer (Original)Backbone=Llama-3.1-8B-Instruct, Steering Method=AdaSteer (Original)2026.05 | 58.2 | 22.2 | 36 | |
| AdaSteer (Original)Backbone=Gemma-2-9B, Steering Method=AdaSteer (Original)2026.05 | 57.3 | 16.8 | 40.5 | |
| CAST (Original)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (Original)2026.05 | 48.4 | 17.4 | 31 | |
| AdaSteer (HRL + 32 LRL)Backbone=Gemma-2-9B, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 47 | 45.6 | 1.4 | |
| AdaSteer (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=AdaSteer (HRL + 32 LRL)2026.05 | 45.6 | 20.6 | 25 | |
| Qwen2.5-7B-InstructBackbone=Qwen2.5-7B-Instruct, Steering Method=None2026.05 | 41.4 | 11.6 | 29.8 | |
| Llama-3.1-8B-InstructBackbone=Llama-3.1-8B-Instruct, Steering Method=None2026.05 | 40.9 | 4.8 | 36.1 | |
| CAST (HRL + 32 LRL)Backbone=Qwen2.5-7B-Instruct, Steering Method=CAST (HRL + 32 LRL)2026.05 | 36.4 | 13.4 | 23 | |
| Ours (HRL)Backbone=Llama-3.1-8B-Instruct, Steering Method=HRL2026.05 | 31.3 | 4.6 | 26.7 |