Black-box Attack on VLM Evaluation Set (test)
75.75Ensemble Success RateSGHA-Attack (Ours)
Evaluation Results
| Method | Links | |||
|---|---|---|---|---|
| SGHA-Attack (Ours)VLM Model=BLIP-2, Defense=Bit Reduction2026.02 | 75.75 | 91.9 | 73.9 | |
| SGHA-Attack (Ours)VLM Model=InstructBLIP, Defense=Bit Reduction2026.02 | 74.84 | 93.3 | 74.4 | |
| SGHA-Attack (Ours)VLM Model=UniDiffuser, Defense=Bit Reduction2026.02 | 74.52 | 97.3 | 81.7 | |
| SGHA-Attack (Ours)VLM Model=UniDiffuser, Defense=JPEG Compression2026.02 | 72.25 | 97.1 | 77.1 | |
| COAVLM Model=BLIP-2, Defense=Bit Reduction2026.02 | 69.12 | 90.5 | 56.4 | |
| COAVLM Model=UniDiffuser, Defense=Bit Reduction2026.02 | 68.33 | 98.4 | 65.6 | |
| COAVLM Model=InstructBLIP, Defense=Bit Reduction2026.02 | 68.3 | 92.2 | 56 | |
| MF-iiVLM Model=UniDiffuser, Defense=Bit Reduction2026.02 | 65.99 | 94.4 | 60 | |
| COAVLM Model=UniDiffuser, Defense=JPEG Compression2026.02 | 65.51 | 98.4 | 56.8 | |
| MF-iiVLM Model=BLIP-2, Defense=Bit Reduction2026.02 | 65.32 | 75.5 | 42.6 | |
| SGHA-Attack (Ours)VLM Model=UniDiffuser, Defense=ComDefend2026.02 | 65.05 | 91 | 56 | |
| MF-iiVLM Model=InstructBLIP, Defense=Bit Reduction2026.02 | 64.12 | 77 | 43.5 | |
| SGHA-Attack (Ours)VLM Model=LLaVA, Defense=Bit Reduction2026.02 | 64.02 | 93.5 | 76.4 | |
| MF-iiVLM Model=UniDiffuser, Defense=JPEG Compression2026.02 | 63.16 | 91.3 | 51.2 | |
| SGHA-Attack (Ours)VLM Model=BLIP-2, Defense=JPEG Compression2026.02 | 58.82 | 56.4 | 27.4 | |
| COAVLM Model=UniDiffuser, Defense=ComDefend2026.02 | 58.23 | 93.3 | 34.5 | |
| SGHA-Attack (Ours)VLM Model=InstructBLIP, Defense=ComDefend2026.02 | 57.66 | 51.7 | 23.3 | |
| SGHA-Attack (Ours)VLM Model=InstructBLIP, Defense=JPEG Compression2026.02 | 57.47 | 54 | 25.1 | |
| SGHA-Attack (Ours)VLM Model=BLIP-2, Defense=ComDefend2026.02 | 57.38 | 51.8 | 24.3 | |
| SGHA-Attack (Ours)VLM Model=UniDiffuser, Defense=DiffPure2026.02 | 56.86 | 70.1 | 27.2 | |
| AdvDiffVLMVLM Model=BLIP-2, Defense=Bit Reduction2026.02 | 56.75 | 45.9 | 19.6 | |
| AdvDiffVLMVLM Model=InstructBLIP, Defense=Bit Reduction2026.02 | 56.45 | 48.7 | 19.3 | |
| MF-iiVLM Model=UniDiffuser, Defense=ComDefend2026.02 | 56.11 | 79.1 | 28.7 | |
| SGHA-Attack (Ours)VLM Model=InstructBLIP, Defense=DiffPure2026.02 | 56.1 | 44.9 | 18.2 | |
| SGHA-Attack (Ours)VLM Model=BLIP-2, Defense=DiffPure2026.02 | 56 | 43.3 | 18.5 | |
| SGHA-Attack (Ours)VLM Model=MiniGPT-4, Defense=Bit Reduction2026.02 | 55.74 | 93.9 | 73.4 | |
| AdvDiffVLMVLM Model=BLIP-2, Defense=DiffPure2026.02 | 55.12 | 41.5 | 13.7 | |
| AdvDiffVLMVLM Model=UniDiffuser, Defense=JPEG Compression2026.02 | 54.97 | 72.2 | 24 | |
| AdvDiffVLMVLM Model=InstructBLIP, Defense=DiffPure2026.02 | 54.53 | 43 | 15 | |
| COAVLM Model=UniDiffuser, Defense=DiffPure2026.02 | 54.47 | 83.4 | 20.3 | |
| COAVLM Model=BLIP-2, Defense=JPEG Compression2026.02 | 54.46 | 60.2 | 15.6 | |
| AdvDiffVLMVLM Model=UniDiffuser, Defense=Bit Reduction2026.02 | 54.22 | 69.2 | 21.6 | |
| COAVLM Model=InstructBLIP, Defense=ComDefend2026.02 | 54.2 | 55.5 | 14.6 | |
| COAVLM Model=BLIP-2, Defense=ComDefend2026.02 | 54.13 | 55.6 | 14.4 | |
| COAVLM Model=InstructBLIP, Defense=JPEG Compression2026.02 | 54.03 | 59.3 | 15.7 | |
| SGHA-Attack (Ours)VLM Model=LLaVA, Defense=JPEG Compression2026.02 | 53.8 | 70.1 | 38.1 | |
| COAVLM Model=BLIP-2, Defense=DiffPure2026.02 | 53.37 | 48 | 11.6 | |
| COAVLM Model=InstructBLIP, Defense=DiffPure2026.02 | 53.18 | 47.8 | 11.6 | |
| AdvDiffVLMVLM Model=BLIP-2, Defense=JPEG Compression2026.02 | 52.81 | 42.5 | 11.9 | |
| AdvDiffVLMVLM Model=InstructBLIP, Defense=ComDefend2026.02 | 51.73 | 34.3 | 8.9 | |
| MF-iiVLM Model=BLIP-2, Defense=JPEG Compression2026.02 | 51.69 | 37.4 | 12.4 | |
| MF-iiVLM Model=InstructBLIP, Defense=ComDefend2026.02 | 51.61 | 32.8 | 9.7 | |
| MF-iiVLM Model=LLaVA, Defense=Bit Reduction2026.02 | 51.6 | 66.2 | 30.3 | |
| MF-iiVLM Model=BLIP-2, Defense=DiffPure2026.02 | 51.58 | 27.4 | 7.1 | |
| MF-iiVLM Model=InstructBLIP, Defense=DiffPure2026.02 | 51.26 | 27.4 | 7.6 | |
| AdvDiffVLMVLM Model=InstructBLIP, Defense=JPEG Compression2026.02 | 51.23 | 38.7 | 11.5 | |
| AdvDiffVLMVLM Model=UniDiffuser, Defense=DiffPure2026.02 | 51.1 | 62.8 | 14.4 | |
| AdvDiffVLMVLM Model=BLIP-2, Defense=ComDefend2026.02 | 50.98 | 32.3 | 8.1 | |
| MF-iiVLM Model=InstructBLIP, Defense=JPEG Compression2026.02 | 50.98 | 33.7 | 10.6 | |
| MF-iiVLM Model=BLIP-2, Defense=ComDefend2026.02 | 50.95 | 31.4 | 9 | |
| MF-iiVLM Model=UniDiffuser, Defense=DiffPure2026.02 | 50.61 | 56 | 13.2 | |
| COAVLM Model=LLaVA, Defense=Bit Reduction2026.02 | 49.86 | 76.7 | 23.2 | |
| COAVLM Model=MiniGPT-4, Defense=Bit Reduction2026.02 | 49.46 | 92.7 | 52.8 | |
| AdvDiffVLMVLM Model=UniDiffuser, Defense=ComDefend2026.02 | 49 | 62.5 | 10.5 | |
| SGHA-Attack (Ours)VLM Model=LLaVA-NeXT, Defense=Bit Reduction2026.02 | 48.43 | 78.5 | 44.4 | |
| SGHA-Attack (Ours)VLM Model=LLaVA, Defense=ComDefend2026.02 | 47.1 | 46.2 | 18 | |
| MF-iiVLM Model=MiniGPT-4, Defense=Bit Reduction2026.02 | 45.47 | 76.3 | 39.3 | |
| SGHA-Attack (Ours)VLM Model=LLaVA, Defense=DiffPure2026.02 | 45.2 | 37.4 | 11.9 | |
| SGHA-Attack (Ours)VLM Model=MiniGPT-4, Defense=ComDefend2026.02 | 44.98 | 50.3 | 22.4 | |
| MF-iiVLM Model=LLaVA, Defense=JPEG Compression2026.02 | 44.5 | 39.4 | 11.4 | |
| COAVLM Model=LLaVA, Defense=DiffPure2026.02 | 44.37 | 41.6 | 10.8 | |
| SGHA-Attack (Ours)VLM Model=MiniGPT-4, Defense=JPEG Compression2026.02 | 43.25 | 54.7 | 24.2 | |
| AdvDiffVLMVLM Model=LLaVA, Defense=JPEG Compression2026.02 | 43.13 | 34.4 | 6.8 | |
| AdvDiffVLMVLM Model=LLaVA, Defense=DiffPure2026.02 | 42.48 | 31.6 | 5.2 | |
| AdvDiffVLMVLM Model=LLaVA, Defense=Bit Reduction2026.02 | 42.47 | 34 | 6.6 | |
| COAVLM Model=MiniGPT-4, Defense=ComDefend2026.02 | 42.13 | 57.6 | 13 | |
| SGHA-Attack (Ours)VLM Model=MiniGPT-4, Defense=DiffPure2026.02 | 42.02 | 42.5 | 16.2 | |
| COAVLM Model=LLaVA, Defense=JPEG Compression2026.02 | 41.37 | 40.6 | 3.8 | |
| MF-iiVLM Model=LLaVA, Defense=DiffPure2026.02 | 41.31 | 18.1 | 2.6 | |
| COAVLM Model=LLaVA, Defense=ComDefend2026.02 | 41.04 | 35.1 | 2.6 | |
| MF-iiVLM Model=LLaVA, Defense=ComDefend2026.02 | 40.63 | 24.5 | 4.4 | |
| AdvDiffVLMVLM Model=LLaVA, Defense=ComDefend2026.02 | 40.54 | 26.1 | 2.1 | |
| COAVLM Model=MiniGPT-4, Defense=JPEG Compression2026.02 | 39.91 | 62.6 | 14.1 | |
| MF-iiVLM Model=MiniGPT-4, Defense=ComDefend2026.02 | 39.7 | 31.1 | 7.3 | |
| SGHA-Attack (Ours)VLM Model=LLaVA-NeXT, Defense=JPEG Compression2026.02 | 39.54 | 49.2 | 15.8 | |
| COAVLM Model=MiniGPT-4, Defense=DiffPure2026.02 | 39.1 | 48.8 | 9.5 | |
| AdvDiffVLMVLM Model=MiniGPT-4, Defense=Bit Reduction2026.02 | 39.01 | 44.7 | 15.8 | |
| AdvDiffVLMVLM Model=MiniGPT-4, Defense=DiffPure2026.02 | 38.99 | 38.5 | 10.8 | |
| AdvDiffVLMVLM Model=MiniGPT-4, Defense=ComDefend2026.02 | 38.33 | 31.4 | 6 | |
| AdvDiffVLMVLM Model=MiniGPT-4, Defense=JPEG Compression2026.02 | 38.13 | 38.8 | 9.8 | |
| MF-iiVLM Model=MiniGPT-4, Defense=JPEG Compression2026.02 | 37.76 | 35.5 | 8.9 | |
| MF-iiVLM Model=MiniGPT-4, Defense=DiffPure2026.02 | 37.59 | 25 | 6.4 | |
| COAVLM Model=LLaVA-NeXT, Defense=Bit Reduction2026.02 | 37.01 | 58.5 | 6.1 | |
| SGHA-Attack (Ours)VLM Model=LLaVA-NeXT, Defense=ComDefend2026.02 | 36.6 | 37.5 | 7.4 | |
| MF-iiVLM Model=LLaVA-NeXT, Defense=JPEG Compression2026.02 | 36.22 | 26.5 | 3.5 | |
| SGHA-Attack (Ours)VLM Model=LLaVA-NeXT, Defense=DiffPure2026.02 | 34.23 | 25.6 | 3.8 | |
| AdvDiffVLMVLM Model=LLaVA-NeXT, Defense=ComDefend2026.02 | 34.17 | 27.7 | 0.4 | |
| AdvDiffVLMVLM Model=LLaVA-NeXT, Defense=Bit Reduction2026.02 | 34.13 | 27.4 | 1.3 | |
| COAVLM Model=LLaVA-NeXT, Defense=DiffPure2026.02 | 33.75 | 30.2 | 3.3 | |
| COAVLM Model=LLaVA-NeXT, Defense=ComDefend2026.02 | 33.58 | 31.9 | 1 | |
| COAVLM Model=LLaVA-NeXT, Defense=JPEG Compression2026.02 | 33.26 | 32.6 | 1.7 | |
| MF-iiVLM Model=LLaVA-NeXT, Defense=Bit Reduction2026.02 | 33.01 | 39.5 | 9.5 | |
| AdvDiffVLMVLM Model=LLaVA-NeXT, Defense=DiffPure2026.02 | 32.94 | 24.8 | 1.2 | |
| MF-iiVLM Model=LLaVA-NeXT, Defense=ComDefend2026.02 | 32.76 | 20.9 | 1.3 | |
| AdvDiffVLMVLM Model=LLaVA-NeXT, Defense=JPEG Compression2026.02 | 32.67 | 26.9 | 1.9 | |
| MF-iiVLM Model=LLaVA-NeXT, Defense=DiffPure2026.02 | 31.72 | 15.2 | 0.8 |