Jailbreak Safety Evaluation on MM-Safety Bench (test)
0.18Average ASRLLaVA-v1.5-7B
Evaluation results
| Method | Links | |||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| LLaVA-v1.5-7BFinetuning Dataset=VLGuard2025.04 | 0.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Yi-VL-6BFinetuning Dataset=VLGuard2025.04 | 0.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-NeXT-Mistral-7BFinetuning Dataset=VLGuard2025.04 | 0.18 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-v1.5-13BFinetuning Dataset=VLGuard2025.04 | 0.36 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-v1.5-13BFinetuning Dataset=Ours2025.04 | 1.79 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-NeXT-Mistral-7BFinetuning Dataset=Ours2025.04 | 2.61 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IMMUNEModel=LLaVA-1.52024.11 | 3.51 | 0.46 | 1.03 | 0 | 0.4 | 0 | 13.64 | 6.98 | 9.09 | 19.05 | 0.52 | 3.02 | 5.04 | 5.17 | 4.03 | 7.02 | 0 | 0 | 6.04 | |
| Yi-VL-6BFinetuning Dataset=Ours2025.04 | 3.57 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IMMUNEModel=LLaVA-1.62024.11 | 3.94 | 0 | 1.03 | 5.15 | 0 | 0 | 4.55 | 9.23 | 3.07 | 18.02 | 0 | 0 | 3.02 | 4.09 | 5.03 | 5.04 | 0 | 2.07 | 7.07 | |
| AdaShieldModel=LLaVA-1.52024.11 | 4.81 | 9.28 | 1.03 | 3.09 | 0 | 2.27 | 0 | 9.06 | 7.03 | 4.04 | 7.07 | 1.04 | 2.03 | 11.03 | 7.05 | 3.03 | 10.01 | 1.07 | 2.08 | |
| LLaVA-v1.5-7BFinetuning Dataset=Ours2025.04 | 5.6 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FigStepTarget Model=GPT-4o, System Prompt=None2024.11 | 6.86 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QueryRelatedTarget Model=Claude-3.5-Sonnet, System Prompt=None2024.11 | 8.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FigStepTarget Model=Claude-3.5-Sonnet, System Prompt=None2024.11 | 9.32 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| IMMUNEModel=MiniGPT-4-7B2024.11 | 11.03 | 13.46 | 22.68 | 13.43 | 11.36 | 20.45 | 18.18 | 17.04 | 12.09 | 21.03 | 3.08 | 8.04 | 7.08 | 7.02 | 14.01 | 20.02 | 0 | 0 | 0 | |
| IMMUNEModel=MiniGPT-4-13B2024.11 | 14.24 | 23.78 | 22.68 | 19.59 | 9.09 | 18.18 | 29.55 | 20.08 | 14.08 | 15.06 | 11.08 | 13.08 | 13.02 | 10.08 | 18.03 | 22.08 | 6.02 | 0 | 0 | |
| MMLTarget Model=Claude-3.5-Sonnet, Encryption=Base64 Encoding, System Prompt=None2024.11 | 14.92 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoCAModel=MiniGPT-4-7B2024.11 | 18.92 | 9.28 | 42.27 | 28.87 | 6.82 | 20.45 | 18.18 | 19.08 | 12.09 | 24.07 | 6.05 | 12.07 | 10.03 | 25.04 | 16.67 | 26.07 | 25.06 | 8.33 | 25.02 | |
| AdaShieldModel=MiniGPT-4-13B2024.11 | 19.84 | 21.65 | 23.71 | 22.68 | 20.45 | 34.09 | 25.04 | 15.05 | 9.02 | 15.03 | 11.06 | 25.03 | 21.01 | 14.03 | 22.01 | 23.07 | 23.05 | 28.02 | 15.09 | |
| AdaShieldModel=MiniGPT-4-7B2024.11 | 20.05 | 11.34 | 35.05 | 18.56 | 18.18 | 22.73 | 9.09 | 16.67 | 26.03 | 15.04 | 22.67 | 8.04 | 26.04 | 8.08 | 27.09 | 14.09 | 18.18 | 34.04 | 22.07 | |
| OriginalModel=MiniGPT-4-7B2024.11 | 23.04 | 13.43 | 43.34 | 35.05 | 2.27 | 34.09 | 25.08 | 27.04 | 14.01 | 21.06 | 6.02 | 29.05 | 17.05 | 3.08 | 34.08 | 26.02 | 5.05 | 40.04 | 36.02 | |
| OriginalModel=MiniGPT-4-13B2024.11 | 23.97 | 17.53 | 40.21 | 42.27 | 13.64 | 22.73 | 27.27 | 10.04 | 15.08 | 18.03 | 9.05 | 22.09 | 21.07 | 17.03 | 20.02 | 35.08 | 19.05 | 37.03 | 41.09 | |
| FigStepModel=MiniGPT-4-7B2024.11 | 24.21 | 19.59 | 43.34 | 38.14 | 6.82 | 40.91 | 15.91 | 26.01 | 19.06 | 23.06 | 11.04 | 28.08 | 16.03 | 14.06 | 31.08 | 26.07 | 12.06 | 44.05 | 17.07 | |
| AdaShieldModel=LLaVA-1.62024.11 | 24.4 | 19.59 | 62.89 | 41.24 | 9.09 | 54.55 | 43.18 | 7.05 | 25.05 | 19.06 | 6.06 | 20.03 | 13.03 | 9.09 | 26.04 | 27.04 | 6.01 | 36.02 | 35.08 | |
| QueryRelatedTarget Model=GPT-4o, System Prompt=None2024.11 | 25.25 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoCAModel=MiniGPT-4-13B2024.11 | 26.13 | 25.03 | 50.07 | 46.03 | 14.06 | 25.03 | 43.06 | 13.02 | 25.01 | 8.33 | 16.04 | 16.67 | 16.67 | 16.67 | 25.06 | 41.67 | 25.06 | 41.67 | 25.07 | |
| QueryRelatedTarget Model=GPT-4o-Mini, System Prompt=None2024.11 | 26.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| FigStepModel=MiniGPT-4-13B2024.11 | 27.17 | 25.77 | 46.39 | 36.08 | 22.73 | 27.27 | 29.55 | 13.02 | 16.08 | 19.03 | 20.08 | 30.07 | 20.08 | 18.01 | 36.04 | 38.08 | 21.05 | 32.09 | 38.03 | |
| FigStepModel=LLaVA-1.62024.11 | 37.64 | 27.84 | 49.48 | 79.38 | 25.03 | 47.73 | 59.09 | 7.09 | 28.07 | 41.03 | 30.04 | 36.07 | 46.07 | 20.02 | 21.08 | 52.03 | 29.05 | 36.08 | 55.07 | |
| CoCAModel=LLaVA-1.62024.11 | 39.79 | 24.27 | 55.67 | 76.01 | 17.27 | 59.09 | 70.45 | 9.02 | 33.07 | 43.08 | 25.01 | 42.07 | 52.06 | 23.06 | 26.04 | 57.01 | 25.01 | 37.08 | 58.07 | |
| FigStepTarget Model=GPT-4o-Mini, System Prompt=None2024.11 | 42.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| CoCAModel=LLaVA-1.52024.11 | 44.65 | 17.13 | 80.41 | 80.41 | 25.91 | 27.77 | 9.91 | 8.07 | 52.06 | 55.06 | 10.08 | 63.19 | 67.03 | 25.07 | 45.05 | 57.07 | 26.03 | 57.08 | 59.07 | |
| FigStepModel=LLaVA-1.52024.11 | 45.54 | 25.52 | 63.92 | 73.28 | 20.43 | 75.05 | 44.09 | 16.04 | 26.05 | 38.06 | 38.03 | 51.06 | 55.01 | 21.04 | 57.04 | 59.06 | 36.08 | 62.08 | 61.04 | |
| OriginalModel=LLaVA-1.62024.11 | 47.94 | 34.02 | 80.41 | 78.35 | 20.45 | 70.45 | 70.45 | 7.07 | 42.05 | 35.03 | 20.06 | 51.02 | 52.05 | 18.01 | 63.01 | 63.05 | 30.03 | 67.04 | 72.07 | |
| FigStepTarget Model=Qwen-VL-Max, System Prompt=None2024.11 | 48.73 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Claude-3.5-Sonnet, Encryption=Image Rotation, System Prompt=None2024.11 | 50.68 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| QueryRelatedTarget Model=Qwen-VL-Max, System Prompt=None2024.11 | 51.19 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Claude-3.5-Sonnet, Encryption=Word Replacement, System Prompt=None2024.11 | 52.12 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| OriginalModel=LLaVA-1.52024.11 | 53.85 | 34.02 | 80.41 | 80.41 | 43.18 | 75.01 | 70.45 | 13.02 | 56.07 | 39.07 | 24.05 | 75.02 | 69.07 | 25.07 | 64.03 | 67.05 | 32.03 | 73.07 | 65.03 | |
| MMLTarget Model=Claude-3.5-Sonnet, Encryption=Image Mirroring, System Prompt=None2024.11 | 60 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| Yi-VL-6BFinetuning Dataset=Origin2025.04 | 93.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Qwen-VL-Max, Encryption=Base64 Encoding, System Prompt=None2024.11 | 93.98 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Qwen-VL-Max, Encryption=Word Replacement, System Prompt=None2024.11 | 95.42 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o-Mini, Encryption=Base64 Encoding, System Prompt=None2024.11 | 95.51 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-v1.5-7BFinetuning Dataset=Origin2025.04 | 96.37 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o-Mini, Encryption=Image Rotation, System Prompt=None2024.11 | 96.44 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Qwen-VL-Max, Encryption=Image Rotation, System Prompt=None2024.11 | 96.69 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o-Mini, Encryption=Word Replacement, System Prompt=None2024.11 | 97.03 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-v1.5-13BFinetuning Dataset=Origin2025.04 | 97.8 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=Qwen-VL-Max, Encryption=Image Mirroring, System Prompt=None2024.11 | 97.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o, Encryption=Image Mirroring, System Prompt=None2024.11 | 98.05 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o, Encryption=Word Replacement, System Prompt=None2024.11 | 98.11 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o-Mini, Encryption=Image Mirroring, System Prompt=None2024.11 | 98.14 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o, Encryption=Base64 Encoding, System Prompt=None2024.11 | 98.64 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| MMLTarget Model=GPT-4o, Encryption=Image Rotation, System Prompt=None2024.11 | 98.81 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | |
| LLaVA-NeXT-Mistral-7BFinetuning Dataset=Origin2025.04 | 99.88 | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — | — |