Jailbreaking on HarmBench
84Attack Success Rate (ASR)MASKFORGE
Evaluation results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| MASKFORGEVictim model=LLaDA-Instruct2026.06 | 84 | — | — | — | 4.74 | |
| UJEM-KLTarget Model=LLaVA-1.52026.05 | 82.3 | — | — | — | — | |
| UJEM-KLTarget Model=LLaVA-1.52026.05 | 82.3 | — | — | — | — | |
| MASKFORGEVictim model=Dream-Instruct2026.06 | 80 | — | — | — | 4.49 | |
| UJEMTarget Model=LLaVA-1.52026.05 | 79.4 | — | — | — | — | |
| UJEMTarget Model=LLaVA-1.52026.05 | 79.4 | — | — | — | — | |
| UJEM-KLTarget Model=InternVL3.52026.05 | 79.1 | — | — | — | — | |
| UJEM-KLTarget Model=InternVL3.52026.05 | 79.1 | — | — | — | — | |
| MASKFORGEVictim model=LLaDA 1.52026.06 | 79 | — | — | — | 4.74 | |
| MASKFORGEVictim model=AVG.2026.06 | 79 | — | — | — | 4.64 | |
| SEATarget Model=LLaVA-1.52026.05 | 78.5 | — | — | — | — | |
| SEATarget Model=LLaVA-1.52026.05 | 78.5 | — | — | — | — | |
| UJEM-KLTarget Model=Qwen2.5-VL2026.05 | 78.4 | — | — | — | — | |
| UJEM-KLTarget Model=Qwen2.5-VL2026.05 | 78.4 | — | — | — | — | |
| MASKFORGEVictim model=LLaDA2.0-mini2026.06 | 78 | — | — | — | 4.7 | |
| ForceTarget Model=LLaVA-1.52026.05 | 77.9 | — | — | — | — | |
| ForceTarget Model=LLaVA-1.52026.05 | 77.9 | — | — | — | — | |
| ForceTarget Model=InternVL3.52026.05 | 77.3 | — | — | — | — | |
| ForceTarget Model=InternVL3.52026.05 | 77.3 | — | — | — | — | |
| DIJAVictim model=MMaDA-CoT2026.06 | 77.1 | — | — | — | 4.03 | |
| SEATarget Model=InternVL3.52026.05 | 76.8 | — | — | — | — | |
| SEATarget Model=InternVL3.52026.05 | 76.8 | — | — | — | — | |
| FigStepTarget Model=LLaVA-1.52026.05 | 76.4 | — | — | — | — | |
| FigStepTarget Model=LLaVA-1.52026.05 | 76.4 | — | — | — | — | |
| SEATarget Model=Qwen2.5-VL2026.05 | 76.1 | — | — | — | — | |
| SEATarget Model=Qwen2.5-VL2026.05 | 76.1 | — | — | — | — | |
| DIJAVictim model=LLaDA-Instruct2026.06 | 75.8 | — | — | — | 4.03 | |
| DIJAVictim model=LLaDA 1.52026.06 | 75.8 | — | — | — | 4.07 | |
| ForceTarget Model=Qwen2.5-VL2026.05 | 74.8 | — | — | — | — | |
| ForceTarget Model=Qwen2.5-VL2026.05 | 74.8 | — | — | — | — | |
| DIJAVictim model=Dream-Instruct2026.06 | 74.6 | — | — | — | 4.01 | |
| UJEMTarget Model=InternVL3.52026.05 | 74.2 | — | — | — | — | |
| UJEMTarget Model=InternVL3.52026.05 | 74.2 | — | — | — | — | |
| MASKFORGEVictim model=MMaDA-CoT2026.06 | 74 | — | — | — | 4.53 | |
| PAIRVictim model=MMaDA-CoT2026.06 | 73 | — | — | — | 3.19 | |
| FigStepTarget Model=Qwen2.5-VL2026.05 | 72.8 | — | — | — | — | |
| FigStepTarget Model=Qwen2.5-VL2026.05 | 72.8 | — | — | — | — | |
| AutoDANVictim model=MMaDA-CoT2026.06 | 72.5 | — | — | — | 3.71 | |
| UJEMTarget Model=Qwen2.5-VL2026.05 | 71.6 | — | — | — | — | |
| UJEMTarget Model=Qwen2.5-VL2026.05 | 71.6 | — | — | — | — | |
| AutoDANVictim model=LLaDA-Instruct2026.06 | 71.5 | — | — | — | 3.52 | |
| FigStepTarget Model=InternVL3.52026.05 | 70.1 | — | — | — | — | |
| FigStepTarget Model=InternVL3.52026.05 | 70.1 | — | — | — | — | |
| PADVictim model=MMaDA-CoT2026.06 | 69.7 | — | — | — | 3.98 | |
| UJATarget Model=LLaVA-1.52026.05 | 68.9 | — | — | — | — | |
| UJATarget Model=LLaVA-1.52026.05 | 68.9 | — | — | — | — | |
| AutoDANVictim model=LLaDA 1.52026.06 | 68 | — | — | — | 3.45 | |
| Set TheoryTarget Model=Gemini-3-Flash, Encoding Category=LLM-based2026.05 | 67.3 | — | — | — | — | |
| Set TheoryTarget Model=Gemini-2.0-Flash, Encoding Category=LLM-based2026.05 | 65.4 | — | — | — | — | |
| UJATarget Model=Qwen2.5-VL2026.05 | 65.3 | — | — | — | — | |
| UJATarget Model=Qwen2.5-VL2026.05 | 65.3 | — | — | — | — | |
| DIJAVictim model=AVG.2026.06 | 64.8 | — | — | — | 3.57 | |
| UJATarget Model=InternVL3.52026.05 | 63.7 | — | — | — | — | |
| UJATarget Model=InternVL3.52026.05 | 63.7 | — | — | — | — | |
| DIRECTVictim model=MMaDA-CoT2026.06 | 62.3 | — | — | — | 3.48 | |
| Formal LogicTarget Model=Gemini-2.0-Flash, Encoding Category=LLM-based2026.05 | 61.6 | — | — | — | — | |
| Set TheoryTarget Model=GPT-4o-Mini, Encoding Category=LLM-based2026.05 | 60.4 | — | — | — | — | |
| Formal LogicTarget Model=Llama-3-8B, Encoding Category=LLM-based2026.05 | 60.4 | — | — | — | — | |
| AutoDANVictim model=AVG.2026.06 | 60.3 | — | — | — | 3.3 | |
| Set TheoryTarget Model=Llama-3-8B, Encoding Category=LLM-based2026.05 | 59.1 | — | — | — | — | |
| Quantum MechanicsTarget Model=Gemini-3-Flash, Encoding Category=LLM-based2026.05 | 59.1 | — | — | — | — | |
| Formal LogicTarget Model=GPT-4o-Mini, Encoding Category=LLM-based2026.05 | 58.5 | — | — | — | — | |
| Formal LogicTarget Model=Gemini-3-Flash, Encoding Category=LLM-based2026.05 | 58.5 | — | — | — | — | |
| Formal LogicTarget Model=Claude-3.7-Sonnet, Encoding Category=LLM-based2026.05 | 57.2 | — | — | — | — | |
| Set TheoryTarget Model=Claude-3.7-Sonnet, Encoding Category=LLM-based2026.05 | 56.6 | — | — | — | — | |
| PADVictim model=Dream-Instruct2026.06 | 54.7 | — | — | — | 3.94 | |
| AutoDANVictim model=Dream-Instruct2026.06 | 53 | — | — | — | 3.2 | |
| PADVictim model=LLaDA-Instruct2026.06 | 52.7 | — | — | — | 3.76 | |
| PADVictim model=AVG.2026.06 | 52.2 | — | — | — | 3.72 | |
| PADVictim model=LLaDA 1.52026.06 | 51.7 | — | — | — | 3.65 | |
| Quantum MechanicsTarget Model=Claude-3.7-Sonnet, Encoding Category=LLM-based2026.05 | 50.9 | — | — | — | — | |
| Quantum MechanicsTarget Model=GPT-4o-Mini, Encoding Category=LLM-based2026.05 | 50.3 | — | — | — | — | |
| Quantum MechanicsTarget Model=Gemini-2.0-Flash, Encoding Category=LLM-based2026.05 | 49.7 | — | — | — | — | |
| Set TheoryTarget Model=Vicuna-13B, Encoding Category=LLM-based2026.05 | 44.7 | — | — | — | — | |
| Quantum MechanicsTarget Model=Llama-3-8B, Encoding Category=LLM-based2026.05 | 44.7 | — | — | — | — | |
| Formal LogicTarget Model=Vicuna-13B, Encoding Category=LLM-based2026.05 | 40.9 | — | — | — | — | |
| AutoDANVictim model=LLaDA2.0-mini2026.06 | 36.5 | — | — | — | 2.62 | |
| PAIRVictim model=LLaDA 1.52026.06 | 35 | — | — | — | 2 | |
| PADVictim model=LLaDA2.0-mini2026.06 | 32.3 | — | — | — | 3.28 | |
| Formal LogicTarget Model=GPT-5, Encoding Category=LLM-based2026.05 | 32.1 | — | — | — | — | |
| PAIRVictim model=AVG.2026.06 | 31.8 | — | — | — | 1.94 | |
| Formal LogicTarget Model=GPT-5-Mini, Encoding Category=LLM-based2026.05 | 31.4 | — | — | — | — | |
| PAIRVictim model=LLaDA-Instruct2026.06 | 30.5 | — | — | — | 1.85 | |
| DIRECTVictim model=AVG.2026.06 | 28.7 | — | — | — | 1.98 | |
| Set TheoryTarget Model=GPT-5, Encoding Category=LLM-based2026.05 | 27.7 | — | — | — | — | |
| DIRECTVictim model=LLaDA-Instruct2026.06 | 27.2 | — | — | — | 1.79 | |
| DIRECTVictim model=LLaDA 1.52026.06 | 26.2 | — | — | — | 1.8 | |
| Set TheoryTarget Model=GPT-5-Mini, Encoding Category=LLM-based2026.05 | 24.5 | — | — | — | — | |
| AIMVictim model=MMaDA-CoT2026.06 | 24.5 | — | — | — | 2.55 | |
| BaselineTarget Model=Vicuna-13B, Encoding Category=Unencoded2026.05 | 22 | — | — | — | — | |
| Quantum MechanicsTarget Model=Vicuna-13B, Encoding Category=LLM-based2026.05 | 21.4 | — | — | — | — | |
| DIJAVictim model=LLaDA2.0-mini2026.06 | 20.7 | — | — | — | 1.73 | |
| DIRECTVictim model=LLaDA2.0-mini2026.06 | 19.1 | — | — | — | 1.51 | |
| BaselineTarget Model=Gemini-3-Flash, Encoding Category=Unencoded2026.05 | 16.4 | — | — | — | — | |
| Quantum MechanicsTarget Model=GPT-5-Mini, Encoding Category=LLM-based2026.05 | 14.5 | — | — | — | — | |
| AIMVictim model=LLaDA-Instruct2026.06 | 14 | — | — | — | 1.9 | |
| PAIRVictim model=LLaDA2.0-mini2026.06 | 14 | — | — | — | 1.3 | |
| Quantum MechanicsTarget Model=GPT-5, Encoding Category=LLM-based2026.05 | 13.2 | — | — | — | — | |
| BaselineTarget Model=GPT-4o-Mini, Encoding Category=Unencoded2026.05 | 12.6 | — | — | — | — | |
| AIMVictim model=LLaDA 1.52026.06 | 12.5 | — | — | — | 1.85 |