Safety Evaluation on JailbreakBench (test)
1.12ASR (Llama-Guard-3-8B)MLP
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| MLPBackbone=Qwen2.5-72B2026.05 | 1.12 | 4.5 | 1 | |
| MLPBackbone=Qwen2.5-7B2026.05 | 1.62 | 3.62 | 1.25 | |
| MLPBackbone=Llama-3.1-70B2026.05 | 4.12 | 2.75 | 2.62 | |
| MoEBackbone=Gemma-2-9B2026.05 | 5 | 4.38 | 6.25 | |
| MoEBackbone=Gemma-2-27B2026.05 | 5.01 | 4.62 | 5.75 | |
| MoEBackbone=Llama-3.1-8B2026.05 | 5.38 | 4.38 | 6 | |
| MoEBackbone=Mistral-7B-v0.32026.05 | 10.38 | 7.38 | 10.12 | |
| FMBackbone=Gemma-2-27B2026.05 | 11.1 | 9.62 | 14.88 | |
| MLPBackbone=Gemma-2-27B2026.05 | 11.12 | 9.88 | 13.25 | |
| MoEBackbone=Qwen2.5-72B2026.05 | 13.12 | 13 | 14 | |
| FMBackbone=Gemma-2-9B2026.05 | 14 | 9.25 | 15.75 | |
| CVAEBackbone=Qwen2.5-72B2026.05 | 15 | 16 | 11.38 | |
| MoEBackbone=Llama-3.1-70B2026.05 | 16 | 16.38 | 14.75 | |
| FMBackbone=Llama-3.1-8B2026.05 | 18.12 | 15.75 | 25.25 | |
| MLPBackbone=Gemma-2-9B2026.05 | 18.38 | 17.38 | 16.38 | |
| MoEBackbone=Qwen2.5-7B2026.05 | 19.52 | 5.12 | 4.88 | |
| MLPBackbone=Mistral-7B-v0.32026.05 | 21.75 | 10 | 8.5 | |
| MLPBackbone=Llama-3.1-8B2026.05 | 27.38 | 15.75 | 17.38 | |
| FMBackbone=Qwen2.5-7B2026.05 | 30.16 | 20.25 | 26.12 | |
| Safe LoRABackbone=Qwen2.5-72B2026.05 | 30.29 | 39.25 | 28.25 | |
| CVAEBackbone=Gemma-2-9B2026.05 | 35.25 | 34.38 | 31.87 | |
| FMBackbone=Qwen2.5-72B2026.05 | 35.88 | 29 | 48 | |
| FMBackbone=Mistral-7B-v0.32026.05 | 37.5 | 25.12 | 40.12 | |
| AEBackbone=Gemma-2-27B2026.05 | 39.36 | 31.25 | 42.75 | |
| CVAEBackbone=Qwen2.5-7B2026.05 | 42.62 | 54.87 | 48.38 | |
| FMBackbone=Llama-3.1-70B2026.05 | 44.5 | 38.62 | 49.62 | |
| Safe LoRABackbone=Gemma-2-9B2026.05 | 45.12 | 45.25 | 48.38 | |
| AEBackbone=Gemma-2-9B2026.05 | 46.49 | 39 | 55 | |
| AEBackbone=Qwen2.5-72B2026.05 | 46.75 | 40.25 | 57.25 | |
| CVAEBackbone=Llama-3.1-8B2026.05 | 49.5 | 55.88 | 55.75 | |
| CVAEBackbone=Mistral-7B-v0.32026.05 | 49.62 | 45.25 | 53.12 | |
| Safe LoRABackbone=Llama-3.1-8B2026.05 | 50.25 | 49.38 | 64 | |
| Safe LoRABackbone=Mistral-7B-v0.32026.05 | 52.38 | 41.88 | 57 | |
| AEBackbone=Qwen2.5-7B2026.05 | 53.14 | 39.75 | 52.75 | |
| CVAEBackbone=Gemma-2-27B2026.05 | 55.25 | 42.38 | 52.25 | |
| AEBackbone=Llama-3.1-8B2026.05 | 56.82 | 54.37 | 67.88 | |
| Safe LoRABackbone=Qwen2.5-7B2026.05 | 59.57 | 65.88 | 58.5 | |
| Unsafe (Finetuned)Backbone=Qwen2.5-72B2026.05 | 60.25 | 50 | 80.5 | |
| SaLoRABackbone=Qwen2.5-72B2026.05 | 62.25 | 53.62 | 73.38 | |
| Unsafe (Finetuned)Backbone=Gemma-2-9B2026.05 | 63.04 | 51.88 | 89.88 | |
| CVAEBackbone=Llama-3.1-70B2026.05 | 63.62 | 52.5 | 48.25 | |
| AEBackbone=Mistral-7B-v0.32026.05 | 65.12 | 47.75 | 69 | |
| Unsafe (Finetuned)Backbone=Llama-3.1-70B2026.05 | 65.81 | 58.81 | 86.38 | |
| Unsafe (Finetuned)Backbone=Gemma-2-27B2026.05 | 66.37 | 53.44 | 86.94 | |
| Unsafe (Finetuned)Backbone=Llama-3.1-8B2026.05 | 66.85 | 60.56 | 90.44 | |
| AEBackbone=Llama-3.1-70B2026.05 | 67.38 | 57.38 | 70.75 | |
| Safe LoRABackbone=Llama-3.1-70B2026.05 | 68.12 | 63.62 | 61.12 | |
| Safe LoRABackbone=Gemma-2-27B2026.05 | 72 | 67.38 | 63.12 | |
| SaLoRABackbone=Gemma-2-9B2026.05 | 73.09 | 57 | 90.62 | |
| Unsafe (Finetuned)Backbone=Qwen2.5-7B2026.05 | 73.36 | 58.81 | 81.69 | |
| Unsafe (Finetuned)Backbone=Mistral-7B-v0.32026.05 | 74.25 | 53.87 | 92.06 | |
| SaLoRABackbone=Llama-3.1-8B2026.05 | 75.84 | 70.25 | 90.12 | |
| SaLoRABackbone=Gemma-2-27B2026.05 | 76.25 | 61.25 | 84 | |
| SaLoRABackbone=Llama-3.1-70B2026.05 | 78.38 | 67.62 | 85.5 | |
| SaLoRABackbone=Qwen2.5-7B2026.05 | 79.97 | 63.12 | 78.12 | |
| SaLoRABackbone=Mistral-7B-v0.32026.05 | 83.38 | 62 | 89.88 |