Jailbreak Attack on Advbench subset
96ASRCodeAttack
Evaluation results
| Method | Links | |||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| CodeAttackTarget Model=DeepSeek-R12026.05 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77.2 | |
| AE-CoTTarget Model=DeepSeek-R12026.05 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 78.8 | |
| AE-CoTTarget Model=Qwen32026.05 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80 | |
| AE-CoTTarget Model=Gemini-2.52026.05 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82.8 | |
| CodeAttackTarget Model=Qwen32026.05 | 94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.4 | |
| ICRTTarget Model=Gemini-2.52026.05 | 94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 80 | |
| H-CoTTarget Model=DeepSeek-R12026.05 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 74.4 | |
| H-CoTTarget Model=Qwen32026.05 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 74.8 | |
| AE-CoTTarget Model=o1-mini2026.05 | 92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 70.4 | |
| ADCTarget Model=GPT-3.52024.05 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| H-CoTTarget Model=Gemini-2.52026.05 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 66.4 | |
| AE-CoTTarget Model=o3-mini2026.05 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 72 | |
| GCGTarget Model=GPT-3.52024.05 | 87 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ICRTTarget Model=DeepSeek-R12026.05 | 86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 77.2 | |
| H-CoTTarget Model=o3-mini2026.05 | 86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 70.4 | |
| PAPTarget Model=DeepSeek-R12026.05 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 71.2 | |
| FlipAttackAttack Type=Black-box2024.10 | 78.12 | — | — | — | — | — | — | — | 81.92 | 80.96 | 65.19 | 78.84 | 83.85 | 86.92 | 84.42 | 62.88 | — | |
| ICRTTarget Model=Qwen32026.05 | 78 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 76.4 | |
| CL-GSOTarget Model=DeepSeek-R12026.05 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 59.2 | |
| ICRTTarget Model=o1-mini2026.05 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 61.2 | |
| ReNeLLMAttack Type=Black-box2024.10 | 66.18 | — | — | — | — | — | — | — | 94.04 | 88.27 | 89.62 | 70.77 | 83.08 | 27.12 | 9.23 | 67.31 | — | |
| ArtPromptAttack Type=Black-box2024.10 | 63.52 | — | — | — | — | — | — | — | 93.75 | 68.65 | 84.81 | 78.06 | 83.46 | 25 | 16.73 | 57.69 | — | |
| PAIRTarget Model=GPT-42024.05 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CL-GSOTarget Model=o1-mini2026.05 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 49.6 | |
| DeepInceptionAttack Type=Black-box2024.10 | 61.55 | — | — | — | — | — | — | — | 75.05 | 79.17 | 80.46 | 66.15 | 69.04 | 18.08 | 15.96 | 88.46 | — | |
| PAIRTarget Model=GPT-3.52024.05 | 60 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAPTarget Model=Gemini-2.52026.05 | 58 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 48.8 | |
| CodeChameleonAttack Type=Black-box2024.10 | 56.2 | — | — | — | — | — | — | — | 85.58 | 96.35 | 84.42 | 23.85 | 62.31 | 37.12 | 0.77 | 59.23 | — | |
| CodeAttackTarget Model=o3-mini2026.05 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 44.8 | |
| CodeAttackTarget Model=o1-mini2026.05 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 43.2 | |
| H-CoTTarget Model=o1-mini2026.05 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 60 | |
| AE-CoTTarget Model=GPT-52026.05 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 71.6 | |
| ADCTarget Model=GPT-42024.05 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CL-GSOTarget Model=Gemini-2.52026.05 | 52 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42.4 | |
| GCGTarget Model=GPT-42024.05 | 47 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| DRAAttack Type=Black-box2024.10 | 44.26 | — | — | — | — | — | — | — | 94.62 | 78.85 | 77.31 | 95 | 0 | 8.27 | 0 | 0 | — | |
| CL-GSOTarget Model=Qwen32026.05 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 28.8 | |
| H-CoTTarget Model=GPT-52026.05 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 29.6 | |
| AutoDANAttack Type=White-box2024.10 | 33.39 | — | — | — | — | — | — | — | 72.31 | 24.04 | 36.92 | 19.23 | 27.12 | 25.58 | 5.77 | 56.15 | — | |
| TAPAttack Type=Black-box2024.10 | 33.28 | — | — | — | — | — | — | — | 72.53 | 56.6 | 54.67 | 45.17 | 9.23 | 12.69 | 1.15 | 14.23 | — | |
| GPTFUZZERAttack Type=Black-box2024.10 | 33.24 | — | — | — | — | — | — | — | 40.5 | 48.85 | 44.04 | 36.92 | 34.62 | 20 | 0 | 40.96 | — | |
| CL-GSOTarget Model=o3-mini2026.05 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 25.6 | |
| Base64Attack Type=Black-box2024.10 | 31.73 | — | — | — | — | — | — | — | 71.35 | 0.38 | 82.69 | 1.35 | 13.08 | 0.19 | 0.58 | 84.23 | — | |
| PAIRAttack Type=Black-box2024.10 | 30.65 | — | — | — | — | — | — | — | 71.54 | 45.74 | 47.44 | 33.53 | 12.5 | 15.38 | 9.62 | 9.42 | — | |
| CL-GSOTarget Model=GPT-52026.05 | 30 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 24 | |
| PromptAttackAttack Type=Black-box2024.10 | 28.85 | — | — | — | — | — | — | — | 37.69 | 26.15 | 28.27 | 23.27 | 32.88 | 22.88 | 32.5 | 27.12 | — | |
| CodeAttackTarget Model=Gemini-2.52026.05 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 19.2 | |
| PAPTarget Model=GPT-52026.05 | 18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 14.4 | |
| ArtPromptTarget Model=Qwen32026.05 | 16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 12.8 | |
| ICRTTarget Model=GPT-52026.05 | 16 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 12.8 | |
| ArtPromptTarget Model=DeepSeek-R12026.05 | 14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 11.2 | |
| SelfCipherAttack Type=Black-box2024.10 | 11.88 | — | — | — | — | — | — | — | 0.58 | 0 | 0.19 | 59.62 | 25.77 | 6.73 | 0 | 2.12 | — | |
| ArtPromptTarget Model=o3-mini2026.05 | 10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 8 | |
| ICRTTarget Model=o3-mini2026.05 | 10 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 8 | |
| GCGAttack Type=White-box2024.10 | 7.5 | — | — | — | — | — | — | — | 30 | 1.73 | 1.35 | 1.54 | 3.46 | 13.46 | 0.96 | 7.5 | — | |
| ArtPromptTarget Model=o1-mini2026.05 | 6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.8 | |
| PAPTarget Model=o1-mini2026.05 | 6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.8 | |
| COLD-AttackAttack Type=White-box2024.10 | 5.72 | — | — | — | — | — | — | — | 18.65 | 2.12 | 1.35 | 1.73 | 5.58 | 11.54 | 2.12 | 2.69 | — | |
| MACAttack Type=White-box2024.10 | 4.93 | — | — | — | — | — | — | — | 18.08 | 0.38 | 0.58 | 0.96 | 2.5 | 11.73 | 1.15 | 4.04 | — | |
| PAPTarget Model=Qwen32026.05 | 4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.2 | |
| CodeAttackTarget Model=GPT-52026.05 | 2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.6 | |
| ArtPromptTarget Model=GPT-52026.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | |
| ArtPromptTarget Model=Gemini-2.52026.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | |
| PAPTarget Model=o3-mini2026.05 | 0 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 0 | |
| DACAT2I prompt format adaptation=true2026.04 | — | 32 | 30 | 28 | 26 | 36 | 38 | 40 | — | — | — | — | — | — | — | — | — | |
| Direct AttackT2I prompt format adaptation=true2026.04 | — | 30 | 26 | 24 | 26 | 34 | 32 | 34 | — | — | — | — | — | — | — | — | — | |
| ETCHLLM Engine=Qwen3-Max, VLM Engine=Qwen3-VL-Plus, Maximum number of budget (T)=22026.04 | — | 62 | 70 | 44 | 42 | 82 | 62 | 76 | — | — | — | — | — | — | — | — | — | |
| JailBrokenT2I prompt format adaptation=true2026.04 | — | 36 | 38 | 24 | 26 | 40 | 42 | 42 | — | — | — | — | — | — | — | — | — | |
| MultiLingualT2I prompt format adaptation=true2026.04 | — | 30 | 28 | 26 | 24 | 32 | 36 | 34 | — | — | — | — | — | — | — | — | — | |
| ReNeLLMT2I prompt format adaptation=true2026.04 | — | 34 | 34 | 30 | 28 | 38 | 40 | 38 | — | — | — | — | — | — | — | — | — | |
| SneakyPromptT2I prompt format adaptation=true2026.04 | — | 30 | 30 | 30 | 28 | 36 | 38 | 36 | — | — | — | — | — | — | — | — | — | |
| Template AttackT2I prompt format adaptation=true2026.04 | — | 30 | 28 | 28 | 24 | 32 | 36 | 38 | — | — | — | — | — | — | — | — | — |