Targeted Attack on Image Captioning on Frontier MLLM Evaluation Set
89Attack Success Rate (ASR)M-Attack
Evaluation results
| Method | Links | ||
|---|---|---|---|
| M-AttackTarget MLLM=GPT-4.1, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 89 | 51 | |
| M-AttackTarget MLLM=Qwen3-VL-8B, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 87 | 47 | |
| M-AttackTarget MLLM=GPT-4o, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 86 | 47 | |
| M-AttackTarget MLLM=Gemini-2.5-Flash, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 77 | 38 | |
| M-AttackTarget MLLM=Gemini-3-Flash, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 69 | 32 | |
| DarkLLMTarget MLLM=GPT-4o, Variant=InternVL-2B, Attack Type=Universal2026.05 | 67.7 | 28 | |
| DarkLLMTarget MLLM=GPT-4.1, Variant=InternVL-1B, Attack Type=Universal2026.05 | 67.6 | 27 | |
| M-AttackTarget MLLM=GPT-4.1, Variant=L/14, Attack Type=Sample-wise2026.05 | 66 | 31 | |
| M-AttackTarget MLLM=GPT-4o, Variant=L/14, Attack Type=Sample-wise2026.05 | 65 | 31 | |
| DarkLLMTarget MLLM=Qwen3-VL-8B, Variant=InternVL-2B, Attack Type=Universal2026.05 | 64.7 | 29 | |
| DarkLLMTarget MLLM=GPT-4o, Variant=InternVL-1B, Attack Type=Universal2026.05 | 64.6 | 26 | |
| DarkLLMTarget MLLM=GPT-4.1, Variant=InternVL-2B, Attack Type=Universal2026.05 | 64.6 | 27 | |
| DarkLLMTarget MLLM=Gemini-2.5-Flash, Variant=InternVL-2B, Attack Type=Universal2026.05 | 63.6 | 25 | |
| M-AttackTarget MLLM=GPT-5-mini, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 63 | 29 | |
| DarkLLMTarget MLLM=Gemini-3-Flash, Variant=InternVL-2B, Attack Type=Universal2026.05 | 60.6 | 23 | |
| AnyAttackTarget MLLM=GPT-4o, Variant=coco-bi, Attack Type=Universal2026.05 | 60 | 22 | |
| AnyAttackTarget MLLM=GPT-4.1, Variant=coco-bi, Attack Type=Universal2026.05 | 59 | 25 | |
| DarkLLMTarget MLLM=GPT-5-mini, Variant=InternVL-2B, Attack Type=Universal2026.05 | 58.6 | 24 | |
| DarkLLMTarget MLLM=Gemini-2.5-Flash, Variant=InternVL-1B, Attack Type=Universal2026.05 | 57.5 | 24 | |
| AnyAttackTarget MLLM=Qwen3-VL-8B, Variant=coco-bi, Attack Type=Universal2026.05 | 56 | 24 | |
| DarkLLMTarget MLLM=GPT-5-mini, Variant=InternVL-1B, Attack Type=Universal2026.05 | 54.5 | 24 | |
| M-AttackTarget MLLM=Gemini-2.5-Flash, Variant=L/14, Attack Type=Sample-wise2026.05 | 53 | 25 | |
| AnyAttackTarget MLLM=GPT-5-mini, Variant=coco-bi, Attack Type=Universal2026.05 | 53 | 22 | |
| DarkLLMTarget MLLM=Gemini-3-Flash, Variant=InternVL-1B, Attack Type=Universal2026.05 | 52.5 | 22 | |
| M-AttackTarget MLLM=Qwen3-VL-8B, Variant=L/14, Attack Type=Sample-wise2026.05 | 52 | 28 | |
| M-AttackTarget MLLM=GPT-4.1, Variant=B/16, Attack Type=Sample-wise2026.05 | 50 | 20 | |
| DarkLLMTarget MLLM=Qwen3-VL-8B, Variant=InternVL-1B, Attack Type=Universal2026.05 | 49.5 | 24 | |
| AnyAttackTarget MLLM=GPT-4.1, Variant=flickr-bi, Attack Type=Universal2026.05 | 47 | 19 | |
| AnyAttackTarget MLLM=Gemini-3-Flash, Variant=coco-bi, Attack Type=Universal2026.05 | 46 | 18 | |
| AnyAttackTarget MLLM=Gemini-2.5-Flash, Variant=coco-bi, Attack Type=Universal2026.05 | 44 | 19 | |
| M-AttackTarget MLLM=GPT-4o, Variant=B/16, Attack Type=Sample-wise2026.05 | 43 | 19 | |
| M-AttackTarget MLLM=Gemini-3-Flash, Variant=L/14, Attack Type=Sample-wise2026.05 | 42 | 17 | |
| AnyAttackTarget MLLM=GPT-4o, Variant=flickr-bi, Attack Type=Universal2026.05 | 40 | 16 | |
| AnyAttackTarget MLLM=Qwen3-VL-8B, Variant=flickr-bi, Attack Type=Universal2026.05 | 36 | 17 | |
| M-AttackTarget MLLM=GPT-5-mini, Variant=L/14, Attack Type=Sample-wise2026.05 | 34 | 16 | |
| AnyAttackTarget MLLM=GPT-4.1, Variant=flickr-cos, Attack Type=Universal2026.05 | 34 | 14 | |
| AnyAttackTarget MLLM=GPT-5-mini, Variant=flickr-bi, Attack Type=Universal2026.05 | 32 | 16 | |
| AnyAttackTarget MLLM=Gemini-2.5-Flash, Variant=flickr-bi, Attack Type=Universal2026.05 | 32 | 14 | |
| M-AttackTarget MLLM=Qwen3-VL-8B, Variant=B/16, Attack Type=Sample-wise2026.05 | 31 | 15 | |
| AnyAttackTarget MLLM=Gemini-2.5-Flash, Variant=flickr-cos, Attack Type=Universal2026.05 | 30 | 14 | |
| AnyAttackTarget MLLM=GPT-4o, Variant=flickr-cos, Attack Type=Universal2026.05 | 29 | 14 | |
| AnyAttackTarget MLLM=GPT-5-mini, Variant=flickr-cos, Attack Type=Universal2026.05 | 27 | 12 | |
| M-AttackTarget MLLM=GPT-5-mini, Variant=B/16, Attack Type=Sample-wise2026.05 | 26 | 12 | |
| M-AttackTarget MLLM=Gemini-2.5-Flash, Variant=B/16, Attack Type=Sample-wise2026.05 | 25 | 11 | |
| AnyAttackTarget MLLM=Qwen3-VL-8B, Variant=flickr-cos, Attack Type=Universal2026.05 | 25 | 11 | |
| AnyAttackTarget MLLM=Gemini-3-Flash, Variant=flickr-bi, Attack Type=Universal2026.05 | 23 | 9 | |
| M-AttackTarget MLLM=Gemini-3-Flash, Variant=B/16, Attack Type=Sample-wise2026.05 | 21 | 8 | |
| AnyAttackTarget MLLM=Gemini-3-Flash, Variant=flickr-cos, Attack Type=Universal2026.05 | 20 | 9 | |
| AnyAttackTarget MLLM=Gemini-3-Flash, Variant=coco-cos, Attack Type=Universal2026.05 | 8 | 3 | |
| AnyAttackTarget MLLM=GPT-4o, Variant=coco-cos, Attack Type=Universal2026.05 | 7 | 4 | |
| AnyAttackTarget MLLM=Gemini-2.5-Flash, Variant=coco-cos, Attack Type=Universal2026.05 | 7 | 4 | |
| AnyAttackTarget MLLM=Qwen3-VL-8B, Variant=coco-cos, Attack Type=Universal2026.05 | 5 | 2 | |
| AnyAttackTarget MLLM=GPT-5-mini, Variant=coco-cos, Attack Type=Universal2026.05 | 5 | 4 | |
| AttackVLMTarget MLLM=Qwen3-VL-8B, Variant=L/14, Attack Type=Sample-wise2026.05 | 4 | 3 | |
| AnyAttackTarget MLLM=GPT-4.1, Variant=coco-cos, Attack Type=Universal2026.05 | 4 | 3 | |
| AttackVLMTarget MLLM=Qwen3-VL-8B, Variant=B/16, Attack Type=Sample-wise2026.05 | 3 | 2 | |
| AttackVLMTarget MLLM=GPT-4o, Variant=B/16, Attack Type=Sample-wise2026.05 | 3 | 1 | |
| AttackVLMTarget MLLM=GPT-4.1, Variant=B/16, Attack Type=Sample-wise2026.05 | 3 | 2 | |
| AttackVLMTarget MLLM=GPT-4o, Variant=L/14, Attack Type=Sample-wise2026.05 | 3 | 2 | |
| AttackVLMTarget MLLM=Qwen3-VL-8B, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 3 | 2 | |
| AttackVLMTarget MLLM=GPT-4.1, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 3 | 1 | |
| AttackVLMTarget MLLM=GPT-5-mini, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 3 | 1 | |
| AttackVLMTarget MLLM=Gemini-2.5-Flash, Variant=L/14, Attack Type=Sample-wise2026.05 | 2 | 2 | |
| AttackVLMTarget MLLM=GPT-4o, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 2 | 2 | |
| AttackVLMTarget MLLM=Gemini-2.5-Flash, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 2 | 2 | |
| AttackVLMTarget MLLM=Gemini-3-Flash, Variant=Ensemble, Attack Type=Sample-wise2026.05 | 2 | 1 | |
| AttackVLMTarget MLLM=Gemini-2.5-Flash, Variant=B/16, Attack Type=Sample-wise2026.05 | 1 | 1 | |
| AttackVLMTarget MLLM=GPT-4.1, Variant=L/14, Attack Type=Sample-wise2026.05 | 1 | 1 | |
| AttackVLMTarget MLLM=GPT-5-mini, Variant=L/14, Attack Type=Sample-wise2026.05 | 1 | 0 | |
| AttackVLMTarget MLLM=Gemini-3-Flash, Variant=L/14, Attack Type=Sample-wise2026.05 | 1 | 0 | |
| AttackVLMTarget MLLM=GPT-5-mini, Variant=B/16, Attack Type=Sample-wise2026.05 | 0 | 0 | |
| AttackVLMTarget MLLM=Gemini-3-Flash, Variant=B/16, Attack Type=Sample-wise2026.05 | 0 | 0 |