Jailbreaking on HarmBench (test)
97ASR (GPT-4o)HMNS
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| HMNSTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 97 | 94 | 2 | — | — | |
| HMNSTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 96 | 92 | 2.1 | — | — | |
| ArrAttackTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 91 | 88 | 7.7 | — | — | |
| ArrAttackTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 90 | 86 | 7.9 | — | — | |
| HMNSTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 90 | 84 | 2.1 | — | — | |
| ArrAttackTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 84 | 78 | 8.4 | — | — | |
| TempestTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 83.1 | 78.2 | 9.5 | — | — | |
| TempestTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 82 | 76.6 | 9.8 | — | — | |
| TempestTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 80 | 73.9 | 10 | — | — | |
| PrisonBreakTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 75.6 | 70.2 | 11.9 | — | — | |
| PrisonBreakTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 74.1 | 68.3 | 12.1 | — | — | |
| PrisonBreakTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 71 | 64.8 | 12.9 | — | — | |
| AutoDANTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 70.6 | 64.9 | 12.8 | — | — | |
| AutoDANTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 69.1 | 63.2 | 13.1 | — | — | |
| MasterKeyTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 68.9 | 63.4 | 10.8 | — | — | |
| ADCTarget LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 67.4 | 61.9 | 10.1 | — | — | |
| MasterKeyTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 67 | 61.2 | 10.9 | — | — | |
| ADCTarget LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 65.7 | 60.1 | 10.6 | — | — | |
| Many-shot JB (MSJ)Target LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 63.7 | 58.4 | 11 | — | — | |
| AutoDANTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 62.4 | 56.7 | 13.9 | — | — | |
| Many-shot JB (MSJ)Target LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 62.2 | 56.7 | 11.4 | — | — | |
| MasterKeyTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 60.1 | 54.2 | 11.7 | — | — | |
| ADCTarget LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 58.6 | 53.1 | 11.2 | — | — | |
| Many-shot JB (MSJ)Target LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 55.2 | 49.8 | 12.3 | — | — | |
| Foot-In-The-Door (FITD)Target LLM=LLaMA-3.1-70B, Zero-shot=true2026.04 | 43.8 | 38.4 | 16.2 | — | — | |
| Foot-In-The-Door (FITD)Target LLM=LLaMA-2-7B-Chat, Zero-shot=true2026.04 | 41.3 | 36.1 | 16.8 | — | — | |
| Foot-In-The-Door (FITD)Target LLM=Phi-3-Medium-14B (Instruct), Zero-shot=true2026.04 | 37.9 | 32.8 | 17.6 | — | — | |
| BaselineModel=Llama-3.1-8B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 37 | — | |
| BaselineModel=Mistral-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 35 | — | |
| BaselineModel=Qwen2.5-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 8 | — | |
| BaselineModel=Qwen3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 8.5 | — | |
| BaselineModel=Gemma-3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 14 | — | |
| DPO bestModel=Llama-3.1-8B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 35.5 | 20.5 | |
| DPO bestModel=Mistral-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 28 | 30.5 | |
| DPO bestModel=Qwen2.5-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 7.5 | 13 | |
| DPO bestModel=Qwen3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 11 | 3.5 | |
| DPO bestModel=Gemma-3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 18 | 22.5 | |
| SmoothLLMModel=Llama-3.1-8B, Evaluation Denominator=/nr, Sample Budget=N=2002026.05 | — | — | — | 5.6 | — | |
| SmoothLLMModel=Llama-3.1-8B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 15 | — | |
| SmoothLLMModel=Mistral-7B, Evaluation Denominator=/nr, Sample Budget=N=2002026.05 | — | — | — | 20.2 | — | |
| SmoothLLMModel=Mistral-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 58.5 | — | |
| SmoothLLMModel=Qwen2.5-7B, Evaluation Denominator=/nr, Sample Budget=N=2002026.05 | — | — | — | 5.4 | — | |
| SmoothLLMModel=Qwen2.5-7B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 20.5 | — | |
| SmoothLLMModel=Qwen3-4B, Evaluation Denominator=/nr, Sample Budget=N=2002026.05 | — | — | — | 4.5 | — | |
| SmoothLLMModel=Qwen3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 14.5 | — | |
| SmoothLLMModel=Gemma-3-4B, Evaluation Denominator=/nr, Sample Budget=N=2002026.05 | — | — | — | 16.8 | — | |
| SmoothLLMModel=Gemma-3-4B, Evaluation Denominator=/200, Sample Budget=N=2002026.05 | — | — | — | 40.5 | — |