Safety Evaluation on VLGuard
0.23ASRLLaVA-1.5-7B + Mixed-SFT
Evaluation Results
| Method | Links | |||||
|---|---|---|---|---|---|---|
| LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 0.23 | — | — | — | — | |
| LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 0.23 | — | — | — | — | |
| LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 1.29 | — | — | — | — | |
| LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 2.26 | — | — | — | — | |
| LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 2.49 | — | — | — | — | |
| LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 3.87 | — | — | — | — | |
| LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 4.07 | — | — | — | — | |
| Claude-3.5-SonnetCategory=Online MLLMs2025.10 | 5.21 | — | — | — | — | |
| LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 5.54 | — | — | — | — | |
| GPT-4oCategory=Online MLLMs2025.10 | 6.11 | — | — | — | — | |
| CrossGuardCategory=MLLM Guardrails2025.10 | 7.24 | — | — | — | — | |
| Qwen2.5-VL-7BCategory=Offline MLLMs2025.10 | 9.73 | — | — | — | — | |
| LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 10.18 | — | — | — | — | |
| LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 12.92 | — | — | — | — | |
| JailDAMCategory=MLLM Guardrails2025.10 | 15.38 | — | — | — | — | |
| HiddenDetectCategory=MLLM Guardrails2025.10 | 26.02 | — | — | — | — | |
| LLaVA-1.5-7B (base)Category=Offline MLLMs2025.10 | 46.38 | — | — | — | — | |
| LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 46.83 | — | — | — | — | |
| LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 54.98 | — | — | — | — | |
| LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 64.25 | — | — | — | — | |
| LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.03 | 65.66 | — | — | — | — | |
| LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 72.9 | — | — | — | — | |
| LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.03 | 73.75 | — | — | — | — | |
| Llama-Guard3-VisionCategory=MLLM Guardrails2025.10 | 89.82 | — | — | — | — | |
| LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 90.27 | — | — | — | — | |
| LlavaGuardCategory=MLLM Guardrails2025.10 | 90.42 | — | — | — | — | |
| LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.03 | 90.72 | — | — | — | — | |
| LLaVA-1.5-13BFine-tuning Strategy=Full fine-tuning, Safety Approach=Baseline2025.03 | — | 68.1 | 91.86 | — | — | |
| LLaVA-1.5-13BFine-tuning Strategy=LoRA fine-tuning, Safety Approach=Baseline2025.03 | — | 67.87 | 93.89 | — | — | |
| LLaVA-1.5-7BFine-tuning Strategy=Full fine-tuning, Safety Approach=Baseline2025.03 | — | 64.25 | 90.27 | — | — | |
| LLaVA-1.5-7BFine-tuning Strategy=LoRA fine-tuning, Safety Approach=Baseline2025.03 | — | 64.72 | 95.25 | — | — | |
| Mixed-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 0.23 | 54.98 | — | — | |
| Mixed-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 0.45 | 69.23 | — | — | |
| Mixed-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 0.45 | 57.01 | — | — | |
| Mixed-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 0.45 | 52.94 | — | — | |
| MM-RLHFArch.=Qwen2.5 VL 7B2025.05 | — | — | — | 78.4 | 3.8 | |
| NPO-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 2.49 | 12.92 | — | — | |
| NPO-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 4.56 | 18.29 | — | — | |
| NPO-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 1.89 | 11.7 | — | — | |
| NPO-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 3.36 | 13.53 | — | — | |
| Posthoc-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 0.23 | 46.83 | — | — | |
| Posthoc-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 0.23 | 51.81 | — | — | |
| Posthoc-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 1.58 | 69.23 | — | — | |
| Posthoc-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 0.23 | 42.08 | — | — | |
| Qwen2.5 VLArch.=Qwen2.5 VL-3B2025.05 | — | — | — | 57.4 | 4.1 | |
| Qwen2.5 VLArch.=Qwen2.5 VL-7B2025.05 | — | — | — | 47 | 4.2 | |
| RMU-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 1.29 | 10.18 | — | — | |
| RMU-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 3.87 | 11.14 | — | — | |
| RMU-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 1.29 | 8.96 | — | — | |
| RMU-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 2.75 | 10.18 | — | — | |
| Self-ReminderArch.=Qwen2.5 VL-7B2025.05 | — | — | — | 68 | 2.8 | |
| SPA-VLArch.=Qwen2.5 VL 7B2025.05 | — | — | — | 97 | 2.4 | |
| Unsafe-FilterBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 65.66 | 90.72 | — | — | |
| Unsafe-FilterBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 67.19 | 93.89 | — | — | |
| Unsafe-FilterBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03 | — | 67.65 | 92.99 | — | — | |
| Unsafe-FilterBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03 | — | 66.97 | 94.34 | — | — | |
| VideoLLaMA3Arch.=VideoLLaMA3-2B2025.05 | — | — | — | 42.4 | 3.8 | |
| VideoLLaMA3Arch.=VideoLLaMA3-7B2025.05 | — | — | — | 58.4 | 3.8 | |
| VideoSafety-R1Arch.=Qwen2.5VL-3B2025.05 | — | — | — | 94.2 | 4.5 | |
| VideoSafety-R1Arch.=Qwen2.5VL-7B2025.05 | — | — | — | 99.6 | 4.2 | |
| VideoSafety-R1Arch.=VideoLLaMA-2B2025.05 | — | — | — | 86.7 | 4.1 | |
| VLGuardArch.=Qwen2.5 VL 7B2025.05 | — | — | — | 100 | 2.7 |