Safety Evaluation on AdvBench N=520
0.2Flip RateLLaMA-3.1-8B
Evaluation results
| Method | Links | ||
|---|---|---|---|
| LLaMA-3.1-8BBits=4, Mitigation Strategy=Group-642026.06 | 0.2 | 75.3 | |
| LLaMA-3.1-8BBits=4, Mitigation Strategy=Unprotected2026.06 | 0.8 | — | |
| LLaMA-3.1-8BBits=4, Mitigation Strategy=FP16 L0-12026.06 | 1.2 | -49 | |
| LLaMA-3.1-8BBits=4, Mitigation Strategy=Attn top-2, Selected Layers=L2,L182026.06 | 1.5 | -75 | |
| Qwen-2.5-7BBits=4, Mitigation Strategy=FP16 L0-12026.06 | 1.9 | 97.8 | |
| Gemma-2-9BBits=2, Mitigation Strategy=Group-642026.06 | 2.7 | 97 | |
| Mistral-7BBits=4, Mitigation Strategy=Group-642026.06 | 7.3 | 52 | |
| Mistral-7BBits=4, Mitigation Strategy=Attn top-2, Selected Layers=L1,L22026.06 | 9.8 | 36 | |
| Mistral-7BBits=4, Mitigation Strategy=FP16 L0-12026.06 | 10.4 | 32 | |
| Mistral-7BBits=4, Mitigation Strategy=Unprotected2026.06 | 15.2 | — | |
| Gemma-2-9BBits=2, Mitigation Strategy=FP16 L0-12026.06 | 79.2 | 13.9 | |
| Gemma-2-9BBits=2, Mitigation Strategy=Attn top-2, Selected Layers=L1,L22026.06 | 80.8 | 12.2 | |
| Qwen-2.5-7BBits=4, Mitigation Strategy=Group-642026.06 | 88.2 | 2.4 | |
| Qwen-2.5-7BBits=4, Mitigation Strategy=Attn top-2, Selected Layers=L5,L222026.06 | 88.9 | 1.5 | |
| Qwen-2.5-7BBits=4, Mitigation Strategy=Unprotected2026.06 | 90.3 | — | |
| Gemma-2-9BBits=2, Mitigation Strategy=Unprotected2026.06 | 92 | — |