Jailbreak Attack Evaluation on AdvBench
86.3ASR Success RateAmnesia
Evaluation results
| Method | Links | ||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AmnesiaModel=Llama-3-8B-Instruct, Setting=Amnesia (Layers 13 & 14)2026.03 | 86.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline (No Attack)Model=Llama-3-8B-Instruct, Setting=Baseline (No Attack)2026.03 | 55.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Baseline (No Attack)Model=Llama-2-7B-Chat, Setting=Baseline (No Attack)2026.03 | 34.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Mixtral 8×7BModel=Mixtral 8×7B2026.01 | 27 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Grok 3 MiniModel=Grok 3 Mini2026.01 | 5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DeepSeek V3.1Model=DeepSeek V3.12026.01 | 3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GPT-4o-miniModel=GPT-4o-mini2026.01 | 2.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Qwen2.5 7BModel=Qwen2.5 7B2026.01 | 2.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini 2.5 FlashModel=Gemini 2.5 Flash2026.01 | 1.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ARCATrials=322024.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2 | 0 | — | 0 | 0 | 0 | — | — | — | |
| AutoDANTrials=32024.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | 73 | — | — | — | — | 66 | — | — | — | |
| CipherChat (SelfCipher)Trials=12024.03 | — | — | — | — | — | — | — | — | — | — | — | — | 9 | 5 | 39 | 19 | 0 | 4 | 0 | 0 | 4 | — | |
| CodeAttackTrials=1, input encoding=string2024.03 | — | — | — | — | — | — | — | — | — | — | — | — | 51 | 94 | 22 | 12 | 89 | 24 | 33 | 40 | 93 | — | |
| CodeAttackTrials=1, input encoding=queue2024.03 | — | — | — | — | — | — | — | — | — | — | — | — | 68 | 92 | 28 | 32 | 87 | 36 | 88 | 90 | 93 | — | |
| CodeAttackTrials=1, input encoding=stack2024.03 | — | — | — | — | — | — | — | — | — | — | — | — | 78 | 84 | 80 | 81 | 84 | 84 | 54 | 70 | 82 | — | |
| DPOModel=DPO2024.08 | — | 0 | 47 | 12 | 30 | 30 | 7 | 50 | 61 | 44 | 4 | 18.4 | 28.4 | — | — | — | — | — | — | — | — | — | |
| E-DPOModel=E-DPO2024.08 | — | 0 | 38 | 8 | 15 | 21 | 5.2 | 41 | 53 | 31 | 4 | 13.6 | 20.89 | — | — | — | — | — | — | — | — | — | |
| GCGTrials=32024.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | 86 | 0 | — | 0 | 4 | 16 | — | — | — | |
| PAIRTrials=32024.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | 54 | — | 4 | 4 | 30 | — | — | — | |
| POLARISTarget Model=Qwen3-8B, Evaluator=GPT-5-mini2026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 20 | |
| POLARISTarget Model=Qwen3-8B, Evaluator=DeepSeek-R12026.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 8 | |
| SFTModel=SFT2024.08 | — | 6 | 80 | 13 | 37 | 31 | 14.8 | 65 | 78 | 91 | 4 | 21.2 | 40.09 | — | — | — | — | — | — | — | — | — |