Refusal mechanism ablation on JBB-Behaviors 100 harmful prompts (test)
100Refusal RateCAA
Evaluation results
| Method | Links | ||
|---|---|---|---|
| CAAModel=Qwen2.5-1.5B-Instruct, Steering Strength (alpha)=1.02026.05 | 100 | 0.888 | |
| CAAModel=Qwen2.5-72B-Instruct, Steering Strength (alpha)=1.02026.05 | 98 | 0.406 | |
| BaselineModel=Qwen2.5-1.5B-Instruct, Steering Strength (alpha)=0.02026.05 | 97 | — | |
| BaselineModel=Qwen2.5-3B-Instruct, Steering Strength (alpha)=0.02026.05 | 92.9 | — | |
| BaselineModel=Llama-3.1-8B-Instruct, Steering Strength (alpha)=0.02026.05 | 88.9 | — | |
| BaselineModel=Qwen2.5-7B-Instruct, Steering Strength (alpha)=0.02026.05 | 82.8 | — | |
| BaselineModel=Llama-3.1-70B-Instruct, Steering Strength (alpha)=0.02026.05 | 72.7 | — | |
| BaselineModel=Qwen2.5-72B-Instruct, Steering Strength (alpha)=0.02026.05 | 65.7 | — | |
| BaselineModel=Llama-3.2-3B-Instruct, Steering Strength (alpha)=0.02026.05 | 57.6 | — | |
| BaselineModel=Llama-3.2-1B-Instruct, Steering Strength (alpha)=0.02026.05 | 43.4 | — | |
| CAAModel=Llama-3.1-8B-Instruct, Steering Strength (alpha)=1.02026.05 | 38.4 | 0.493 | |
| CNAModel=Qwen2.5-3B-Instruct, Steering Strength (alpha)=1.02026.05 | 34.3 | 0.984 | |
| CNAModel=Llama-3.2-3B-Instruct, Steering Strength (alpha)=1.02026.05 | 26.3 | 0.977 | |
| CNAModel=Qwen2.5-1.5B-Instruct, Steering Strength (alpha)=1.02026.05 | 26.3 | 0.982 | |
| CNAModel=Llama-3.2-1B-Instruct, Steering Strength (alpha)=1.02026.05 | 20.2 | 0.975 | |
| CNAModel=Qwen2.5-7B-Instruct, Steering Strength (alpha)=1.02026.05 | 13.1 | 0.98 | |
| CNAModel=Llama-3.1-70B-Instruct, Steering Strength (alpha)=1.02026.05 | 12.1 | 0.981 | |
| CNAModel=Llama-3.1-8B-Instruct, Steering Strength (alpha)=1.02026.05 | 5.1 | 0.969 | |
| CAAModel=Qwen2.5-7B-Instruct, Steering Strength (alpha)=1.02026.05 | 5.1 | 0.414 | |
| CNAModel=Qwen2.5-72B-Instruct, Steering Strength (alpha)=1.02026.05 | 5.1 | 0.983 | |
| CAAModel=Llama-3.2-1B-Instruct, Steering Strength (alpha)=1.02026.05 | 0 | 0.554 | |
| CAAModel=Llama-3.2-3B-Instruct, Steering Strength (alpha)=1.02026.05 | 0 | 0.431 | |
| CAAModel=Llama-3.1-70B-Instruct, Steering Strength (alpha)=1.02026.05 | 0 | 0.569 | |
| CAAModel=Qwen2.5-3B-Instruct, Steering Strength (alpha)=1.02026.05 | 0 | 0.844 |