Jailbreaking on AdvBench
100ASRAMIS
Evaluation results
| Method | Links | ||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| AMISTarget LLM=Llama-3.1-8B2025.11 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 84 | |
| AMISTarget LLM=GPT-4o2025.11 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87 | |
| AMISTarget LLM=Sonnet-42025.11 | 100 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 70 | |
| PiFTarget Model=Vicuna-13B-V1.52025.12 | 99.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.22 | — | |
| PiFTarget Model=Mistral-7B-Instruct2025.12 | 99.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.4 | — | |
| PiFTarget Model=Llama-3.1-8B-Instruct2025.12 | 98.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.46 | — | |
| TAPTarget LLM=Llama-3.1-8B2025.11 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 35 | |
| AMISTarget LLM=GPT-4o-mini2025.11 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 87 | |
| AMISTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMISTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AMISTarget LLM=GPT-4o, ASR Prompt=Harmbench2025.11 | 98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PiFTarget Model=GPT-4-06132025.12 | 97.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.53 | — | |
| GCGTarget Model=Mistral-7B-Instruct2025.12 | 96.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.24 | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Single, Target Model=Qwen 3-14B2025.12 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TAPTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 96 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SATA-MLMVictim LLM=DeepSeek-R1, Configuration=ensemble2024.12 | 94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.84 | — | — | — | — | |
| AMISTarget LLM=Sonnet-4, ASR Prompt=Harmbench2025.11 | 94 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PiFTarget Model=GPT-O1-Preview2025.12 | 93.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.5 | — | |
| GCGTarget Model=Vicuna-13B-V1.52025.12 | 92.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.23 | — | |
| PAIRTarget LLM=Llama-3.1-8B2025.11 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 30 | |
| TAPTarget LLM=GPT-4o-mini2025.11 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 33 | |
| SeqARTarget LLM=Llama-3.1-8B2025.11 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 82 | |
| SeqARTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 90 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Single, Target Model=Qwen 3-32B2025.12 | 89.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| ArtPromptVictim LLM=DeepSeek-R1, Configuration=ensemble2024.12 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4.78 | — | — | — | — | |
| AMISTarget LLM=Haiku-3.52025.11 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | |
| PAIRTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDAN-TurboTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Gemma-7B-IT, Defense=ESF2024.05 | 86.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget Model=GPT-4-06132025.12 | 85.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.16 | — | |
| VulMineTarget Model=Mistral-7B-Instruct, Defense=ESF2024.05 | 84.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-3-8B-Instruct, Defense=ESF2024.05 | 84.62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget LLM=GPT-4o2025.11 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 13 | |
| AutoDAN-TurboTarget LLM=Llama-3.1-8B2025.11 | 84 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 61 | |
| PiFTarget Model=Llama-2-13B-Chat2025.12 | 83.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.15 | — | |
| VulMineTarget Model=Llama-2-13B-Chat, Defense=SmoothLLM2024.05 | 83.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget LLM=GPT-4o-mini2025.11 | 82 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 21 | |
| PAIRTarget Model=Mistral-7B-Instruct2025.12 | 81.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.29 | — | |
| TAPTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 80 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget Model=Vicuna-13B-V1.52025.12 | 79.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.89 | — | |
| VulMineTarget Model=Llama-2-7B-Chat, Defense=SmoothLLM2024.05 | 79.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-2-7B-Chat, Defense=PAT2024.05 | 79.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Average, Defense=ESF2024.05 | 77.39 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAPTarget LLM=Llama-3.1-8B2025.11 | 76 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 42 | |
| VulMineTarget Model=Llama-3-8B-Instruct, Defense=SmoothLLM2024.05 | 75.38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Single, Target Model=Qwen 3-8B2025.12 | 74.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TAPTarget LLM=GPT-4o2025.11 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 13 | |
| PAIRTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 74 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Average, Defense=SmoothLLM2024.05 | 73.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget Model=GPT-O1-Preview2025.12 | 72.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.77 | — | |
| AMISTarget LLM=Haiku-3.5, ASR Prompt=Harmbench2025.11 | 72 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-3-8B-Instruct, Defense=PAT2024.05 | 70.77 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Gemma-7B-IT, Defense=PAT2024.05 | 68.46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-2-13B-Chat, Defense=ESF2024.05 | 67.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget Model=Llama-3.1-8B-Instruct2025.12 | 67.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.42 | — | |
| VulMineTarget Model=Average, Defense=PAT2024.05 | 67.23 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Multi, Target Model=Mistral 0.3-7B2025.12 | 66 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Single, Target Model=Mistral 0.3-7B2025.12 | 65.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Gemma-7B-IT, Defense=SmoothLLM2024.05 | 64.49 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-2-7B-Chat, Defense=ESF2024.05 | 63.08 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Mistral-7B-Instruct, Defense=SmoothLLM2024.05 | 62.7 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget LLM=GPT-4o, ASR Prompt=Harmbench2025.11 | 62 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Llama-2-13B-Chat, Defense=PAT2024.05 | 60 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Single, Target Model=Llama 3.1-8B2025.12 | 59.5 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VulMineTarget Model=Mistral-7B-Instruct, Defense=PAT2024.05 | 57.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget Model=Llama-2-13B-Chat2025.12 | 56.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.52 | — | |
| SATA-ELPVictim LLM=DeepSeek-R1, Configuration=ensemble2024.12 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 3.58 | — | — | — | — | |
| PAPTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDAN-TurboTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 56 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDAN-TurboTarget LLM=GPT-4o-mini2025.11 | 54 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 31 | |
| PAPTarget LLM=GPT-4o-mini2025.11 | 48 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 22 | |
| PAIRTarget LLM=Haiku-3.52025.11 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 14 | |
| TAPTarget LLM=Haiku-3.52025.11 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 13 | |
| TAPTarget LLM=GPT-4o, ASR Prompt=Harmbench2025.11 | 46 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAPTarget LLM=GPT-4o2025.11 | 44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 26 | |
| PAPTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAPTarget LLM=GPT-4o, ASR Prompt=Harmbench2025.11 | 44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| GCGTarget Model=GPT-O1-Preview2025.12 | 43.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.53 | — | |
| AutoDAN-TurboTarget LLM=Haiku-3.52025.11 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 5 | |
| PAIRTarget LLM=Haiku-3.5, ASR Prompt=Harmbench2025.11 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SeqARTarget LLM=GPT-4o-mini, ASR Prompt=Harmbench2025.11 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDAN-TurboTarget LLM=Haiku-3.5, ASR Prompt=Harmbench2025.11 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| AutoDAN-TurboTarget LLM=Sonnet-4, ASR Prompt=Harmbench2025.11 | 42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SATA-MLMVictim LLM=OpenAI o3-mini, Configuration=ensemble2024.12 | 40 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 2.82 | — | — | — | — | |
| AutoDAN-TurboTarget LLM=GPT-4o, ASR Prompt=Harmbench2025.11 | 40 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SeqARTarget LLM=GPT-4o-mini2025.11 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 10 | |
| AutoDAN-TurboTarget LLM=GPT-4o2025.11 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 16 | |
| AutoDAN-TurboTarget LLM=Sonnet-42025.11 | 38 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4 | |
| VanillaTarget LLM=Llama-3.1-8B, ASR Prompt=Harmbench2025.11 | 36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Multi, Target Model=Qwen 3-8B2025.12 | 33.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TAPTarget LLM=Haiku-3.5, ASR Prompt=Harmbench2025.11 | 32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Multi, Target Model=Qwen 3-14B2025.12 | 30.1 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| VanillaTarget LLM=Llama-3.1-8B2025.11 | 30 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 15 | |
| EMBERInteraction Type=Online Interaction, Interaction Variant=Multi, Target Model=Qwen 3-32B2025.12 | 28.3 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| PAIRTarget LLM=Sonnet-42025.11 | 28 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 4 | |
| GCGTarget Model=GPT-4-06132025.12 | 27.2 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 1.8 | — | |
| TAPTarget LLM=Sonnet-4, ASR Prompt=Harmbench2025.11 | 24 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| TAPTarget LLM=Sonnet-42025.11 | 22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | 7 | |
| PAIRTarget LLM=Sonnet-4, ASR Prompt=Harmbench2025.11 | 22 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| SFTInteraction Type=Offline Interaction, Target Model=Qwen 3-32B2025.12 | 18.4 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |