Jailbreak Attack on AdvBench (test)
7,827ASRR2J
Evaluation results
| Method | Links | |||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| R2JTarget Model=GPT-3.5-turbo-0125, Attacker Training Dataset (D_a)=AdvBench2025.02 | 7,827 | — | — | 403 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=GPT-3.5-turbo-0125, Attacker Training Dataset (D_a)=DangerousQA2025.02 | 7,423 | — | — | 390 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=GPT-3.5-turbo-0125, Attacker Training Dataset (D_a)=CatQA-en2025.02 | 7,327 | — | — | 388 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=Llama-2-7b-chat, Attacker Training Dataset (D_a)=AdvBench2025.02 | 6,827 | — | — | 361 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=Llama-2-7b-chat, Attacker Training Dataset (D_a)=CatQA-en2025.02 | 6,385 | — | — | 357 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=Llama-2-7b-chat, Attacker Training Dataset (D_a)=DangerousQA2025.02 | 5,923 | — | — | 336 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=GPT-3.5-turbo-0125, Attacker Training Dataset (D_a)=HEX-PHI2025.02 | 5,615 | — | — | 299 | — | — | — | — | — | — | — | — | — | — | — | |
| R2JTarget Model=Llama-2-7b-chat, Attacker Training Dataset (D_a)=HEX-PHI2025.02 | 4,288 | — | — | 224 | — | — | — | — | — | — | — | — | — | — | — | |
| DRAAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.71 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.33 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.29 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.15 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.27 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.33 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.06 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.12 | |
| CoL Multi-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.62 | |
| DRAAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 99 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.93 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 99 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.29 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 99 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.96 | |
| DRAAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.75 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.03 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 97 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.2 | |
| MACAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 95 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.47 | |
| DRAAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 95 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.77 | |
| DarkciteAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 95 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.72 | |
| DarkciteAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 94 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.88 | |
| AutoDANAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.62 | |
| GCGAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.04 | |
| DarkciteAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.61 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 93 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.56 | |
| DANAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 91 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.59 | |
| DRAAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 89 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.27 | |
| TAPAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.88 | |
| ZO-SPSAVictim Model=InstructBLIP, Training Time=10.3h, Test Time=0.91s2026.01 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDANAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.64 | |
| CoL Single-turnAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 83 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.66 | |
| TAPAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 82 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.95 | |
| DarkciteAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 82 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.86 | |
| GCGAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.47 | |
| DANAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 81 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.03 | |
| MACAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 80 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.98 | |
| DarkciteAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.64 | |
| DarkciteAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 76 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.35 | |
| TAPAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.76 | |
| BiPOSteering vector=+v*2024.05 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ZO-SPSAVictim Model=LLaVA, Training Time=19.4h, Test Time=6.35s2026.01 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DRAAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 73 | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.09 | |
| GCGAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 71 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.48 | |
| TAPAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 71 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.76 | |
| TAPAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 71 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.82 | |
| TAPAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.78 | |
| DRAAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 63 | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.58 | |
| TAPAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Mistral-7B2025.05 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.67 | |
| ZO-SPSAVictim Model=MiniGPT-4, Training Time=29.5h, Test Time=55.03s2026.01 | 60 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DarkciteAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.46 | |
| DANAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Vicuna-7B2025.05 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.39 | |
| AutoDANAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.76 | |
| MACAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 41 | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.4 | |
| No Jailbreak (Baseline)Target Model=GPT-3.5-turbo-0125, Attacker Training Dataset (D_a)=None2025.02 | 38 | — | — | 101 | — | — | — | — | — | — | — | — | — | — | — | |
| No Jailbreak (Baseline)Target Model=Llama-2-7b-chat, Attacker Training Dataset (D_a)=None2025.02 | 38 | — | — | 101 | — | — | — | — | — | — | — | — | — | — | — | |
| GCGAttack Category=White-box, Victim Category=Open-Source Victims, Victim Model=Llama-3-8B2025.05 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.06 | |
| v_freeformSteering vector=+v_freeform2024.05 | 16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Llama-2-7b-chat-hfSteering vector=Initial2024.05 | 16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| v_CAASteering vector=-v_CAA2024.05 | 16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DANAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Dobao-1.5-pro2025.05 | 10 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.05 | |
| v_freeformSteering vector=-v_freeform2024.05 | 3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DANAttack Category=Black-box, Victim Category=Open-Source Victims, Victim Model=Llama-2-7B2025.05 | 2 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.04 | |
| DANAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=GPT-3.5-Turbo2025.05 | 1 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | |
| Llama-2-7b-chat-hfSteering vector=Initial2024.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| v_CAASteering vector=+v_CAA2024.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| BiPOSteering vector=-v*2024.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DANAttack Category=Black-box, Victim Category=Closed-Source Victims, Victim Model=Qwen3-Turbo2025.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | 1 | |
| Claude-4-sonnetModel=Claude-4-sonnet2025.09 | — | — | — | — | — | — | — | — | 98 | 62 | 8 | 20 | 6 | 0 | — | |
| Crescendo2026.01 | — | — | — | — | 19 | 28.6 | 66.7 | 0 | — | — | — | — | — | — | — | |
| DAN2026.01 | — | — | — | — | 0 | 9.5 | 14.3 | 14.3 | — | — | — | — | — | — | — | |
| Deepseek-chatModel=Deepseek-chat2025.09 | — | — | — | — | — | — | — | — | 88 | 72 | 52 | 26 | 0 | 0 | — | |
| Deepseek-v3Model=Deepseek-v32025.09 | — | — | — | — | — | — | — | — | 84 | 80 | 52 | 24 | 0 | 0 | — | |
| DeepseekR1-8BModel=DeepseekR1-8B2025.09 | — | — | — | — | — | — | — | — | 90 | 78 | 42 | 42 | 20 | 60 | — | |
| Doubao-1.5-thinking-proModel=Doubao-1.5-thinking-pro2025.09 | — | — | — | — | — | — | — | — | 86 | 86 | 36 | 40 | 2 | 0 | — | |
| DSNModel=Llama-2-13B2024.04 | — | 38 | 64 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSNModel=Llama-3-8B2024.04 | — | 63 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSNModel=Llama-3.1-8B2024.04 | — | 69 | 61 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSNModel=Qwen2-7B2024.04 | — | 51 | 77 | — | — | — | — | — | — | — | — | — | — | — | — | |
| DSNModel=Gemma2-9B2024.04 | — | 78 | 71 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Echo Chamber2026.01 | — | — | — | — | 55 | 50 | 50 | 25 | — | — | — | — | — | — | — | |
| Ernie-4.0-turbo-8kModel=Ernie-4.0-turbo-8k2025.09 | — | — | — | — | — | — | — | — | 38 | 64 | 70 | 18 | 6 | 0 | — | |
| GCGModel=Llama-2-13B2024.04 | — | 24 | 53 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGModel=Llama-3-8B2024.04 | — | 53 | 59 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGModel=Llama-3.1-8B2024.04 | — | 56 | 40 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGModel=Qwen2-7B2024.04 | — | 45 | 65 | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGModel=Gemma2-9B2024.04 | — | 68 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | |
| Gemini-2.0-flashModel=Gemini-2.0-flash2025.09 | — | — | — | — | — | — | — | — | 66 | 70 | 56 | 36 | 30 | 0 | — | |
| Gemini-2.5-proModel=Gemini-2.5-pro2025.09 | — | — | — | — | — | — | — | — | 90 | 74 | 14 | 32 | 48 | 0 | — | |
| Gemma-3-27b-itModel=Gemma-3-27b-it2025.09 | — | — | — | — | — | — | — | — | 92 | 86 | 62 | 36 | 42 | 0 | — | |
| GPT-3.5Model=GPT-3.52025.09 | — | — | — | — | — | — | — | — | 56 | 66 | 82 | 30 | 0 | 0 | — | |
| GPT-4Model=GPT-42025.09 | — | — | — | — | — | — | — | — | 70 | 56 | 32 | 20 | 0 | 0 | — | |
| GPT-4oModel=GPT-4o2025.09 | — | — | — | — | — | — | — | — | 92 | 84 | 32 | 24 | 0 | 0 | — | |
| Llama3.1-8BModel=Llama3.1-8B2025.09 | — | — | — | — | — | — | — | — | 54 | 50 | 34 | 8 | 0 | 2 | — | |
| Mixtral-8x7BModel=Mixtral-8x7B2025.09 | — | — | — | — | — | — | — | — | 94 | 84 | 64 | 48 | 76 | 80 | — |