Jailbreaking Attack on MM-SafetyBench
95Attack Success Rate (ASR)TAO-Attack
Evaluation results
| Method | Links | ||
|---|---|---|---|
| TAO-AttackTarget Model=Qwen2.5-VL-7B-Instruct2026.03 | 95 | 132 | |
| DMNTarget Model=Gemini-2.5-pro2026.05 | 93.75 | — | |
| DMNTarget Model=Claude Sonnet 42026.05 | 93.1 | — | |
| Multi-turn AttackModel=LLaVa-7B, Turn=32026.01 | 91.5 | — | |
| DMNTarget Model=GPT-4V2026.05 | 91.25 | — | |
| INFER+Target Model=InternVL2, Model Type=Open-source2026.07 | 90.68 | — | |
| DMNTarget Model=GPT-4o2026.05 | 90.42 | — | |
| I-GCGTarget Model=Qwen2.5-VL-7B-Instruct2026.03 | 90 | 248 | |
| DMNTarget Model=Seed1.5-VL2026.05 | 88.45 | — | |
| INFER+Target Model=Llama 3.2-Vision, Model Type=Open-source2026.07 | 87.41 | — | |
| INFER+Target Model=Qwen2-VL, Model Type=Open-source2026.07 | 86.25 | — | |
| DMNTarget Model=Average2026.05 | 86.24 | — | |
| INFER+Target Model=InternVL3.5, Model Type=Open-source2026.07 | 83.92 | — | |
| DMNTarget Model=GPT-52026.05 | 82.86 | — | |
| INFER+Target Model=GPT-4.1 mini, Model Type=Closed-source, Surrogate Model=Qwen2-VL2026.07 | 82.75 | — | |
| DMNTarget Model=o32026.05 | 82.02 | — | |
| INFERTarget Model=InternVL2, Model Type=Open-source2026.07 | 81 | — | |
| DMNTarget Model=o4-mini2026.05 | 80.65 | — | |
| DMNTarget Model=Gemini-2.5-flash2026.05 | 80.54 | — | |
| DMNTarget Model=Qwen3-VL-plus2026.05 | 79.4 | — | |
| INFERTarget Model=Qwen2-VL, Model Type=Open-source2026.07 | 79.02 | — | |
| INFER+Target Model=Gemini 2.5 Flash Lite, Model Type=Closed-source, Surrogate Model=Qwen2-VL2026.07 | 78.79 | — | |
| Multi-turn AttackModel=GPT-4o, Turn=32026.01 | 77.31 | — | |
| INFER+Target Model=Gemini 2.5 Flash, Model Type=Closed-source, Surrogate Model=Qwen2-VL2026.07 | 76.57 | — | |
| INFERTarget Model=Llama 3.2-Vision, Model Type=Open-source2026.07 | 76.46 | — | |
| CS-DJTarget Model=Qwen2-VL, Model Type=Open-source2026.07 | 75.06 | — | |
| INFERTarget Model=Gemini 2.5 Flash Lite, Model Type=Closed-source2026.07 | 74.83 | — | |
| INFER+Target Model=Qwen3-VL, Model Type=Open-source2026.07 | 74.48 | — | |
| CS-DJTarget Model=InternVL2, Model Type=Open-source2026.07 | 72.26 | — | |
| MM-SafetyBenchModel=LLaVa-7B2026.01 | 72.14 | — | |
| CS-DJTarget Model=InternVL3.5, Model Type=Open-source2026.07 | 69.7 | — | |
| SI-AttackModel=GPT-4o2026.01 | 68.57 | — | |
| INFERTarget Model=InternVL3.5, Model Type=Open-source2026.07 | 63.4 | — | |
| SI-AttackModel=LLaVa-7B2026.01 | 62.68 | — | |
| FigStepModel=LLaVa-7B2026.01 | 62.3 | — | |
| INFERTarget Model=Gemini 2.5 Flash, Model Type=Closed-source2026.07 | 57.81 | — | |
| CS-DJTarget Model=GPT-4.1 mini, Model Type=Closed-source2026.07 | 56.41 | — | |
| CS-DJTarget Model=Gemini 2.5 Flash Lite, Model Type=Closed-source2026.07 | 56.18 | — | |
| CS-DJTarget Model=Gemini 2.5 Flash, Model Type=Closed-source2026.07 | 54.66 | — | |
| INFERTarget Model=GPT-4.1 mini, Model Type=Closed-source2026.07 | 54.19 | — | |
| ImageTypoTarget Model=InternVL2, Model Type=Open-source2026.07 | 53.38 | — | |
| TVPITarget Model=Qwen2-VL, Model Type=Open-source2026.07 | 53.26 | — | |
| TVPITarget Model=InternVL2, Model Type=Open-source2026.07 | 53.26 | — | |
| ImageTypoTarget Model=Qwen2-VL, Model Type=Open-source2026.07 | 51.75 | — | |
| TVPITarget Model=Qwen3-VL, Model Type=Open-source2026.07 | 45.22 | — | |
| INFERTarget Model=Qwen3-VL, Model Type=Open-source2026.07 | 44.17 | — | |
| CS-DJTarget Model=Llama 3.2-Vision, Model Type=Open-source2026.07 | 39.86 | — | |
| TVPITarget Model=Llama 3.2-Vision, Model Type=Open-source2026.07 | 38.23 | — | |
| CS-DJTarget Model=Qwen3-VL, Model Type=Open-source2026.07 | 33.45 | — | |
| ImageTypoTarget Model=Gemini 2.5 Flash, Model Type=Closed-source2026.07 | 31 | — | |
| ImageTypoTarget Model=Llama 3.2-Vision, Model Type=Open-source2026.07 | 30.42 | — | |
| QRATarget Model=GPT-4o2026.05 | 29.4 | — | |
| QRATarget Model=Gemini-2.5-flash2026.05 | 28.93 | — | |
| QRATarget Model=GPT-4V2026.05 | 28.27 | — | |
| TVPITarget Model=Gemini 2.5 Flash, Model Type=Closed-source2026.07 | 27.51 | — | |
| QRATarget Model=Seed1.5-VL2026.05 | 26.9 | — | |
| QRATarget Model=Qwen3-VL-plus2026.05 | 25.48 | — | |
| QRATarget Model=Average2026.05 | 21.3 | — | |
| TVPITarget Model=Gemini 2.5 Flash Lite, Model Type=Closed-source2026.07 | 20.98 | — | |
| FigStepModel=GPT-4o2026.01 | 19.81 | — | |
| ImageTypoTarget Model=InternVL3.5, Model Type=Open-source2026.07 | 19.7 | — | |
| QRATarget Model=Gemini-2.5-pro2026.05 | 19.29 | — | |
| TVPITarget Model=InternVL3.5, Model Type=Open-source2026.07 | 18.53 | — | |
| QRATarget Model=o4-mini2026.05 | 16.49 | — | |
| QRATarget Model=Claude Sonnet 42026.05 | 15.24 | — | |
| ImageTypoTarget Model=Gemini 2.5 Flash Lite, Model Type=Closed-source2026.07 | 14.8 | — | |
| ImageTypoTarget Model=GPT-4.1 mini, Model Type=Closed-source2026.07 | 12.59 | — | |
| MM-SafetyBenchModel=GPT-4o2026.01 | 12.34 | — | |
| QRATarget Model=GPT-52026.05 | 11.73 | — | |
| QRATarget Model=o32026.05 | 11.31 | — | |
| TVPITarget Model=GPT-4.1 mini, Model Type=Closed-source2026.07 | 10.37 | — | |
| Mistral-Medium-32026.04 | 10 | — | |
| LLaVA-v1.6-Vicuna-7BBackbone=Vicuna-7B2026.04 | 7.31 | — | |
| ImageTypoTarget Model=Qwen3-VL, Model Type=Open-source2026.07 | 4.43 | — | |
| Kimi-K2.52026.04 | 2.69 | — | |
| Llama-3.2-11B-VisionParameters=11B, Type=Vision2026.04 | 2.69 | — | |
| Gemini-3-Flash2026.04 | 1.54 | — | |
| Qwen3-VL-Plus2026.04 | 0.77 | — | |
| Claude-Haiku-4.52026.04 | 0.77 | — | |
| GPT-5-Mini2026.04 | 0.77 | — | |
| DeepSeek-V3.22026.04 | 0.77 | — | |
| Qwen3-VL-8B-InstructParameters=8B, Alignment=Instruct2026.04 | 0.38 | — |