List your company
ResearchFormerly WizWand

Safety Evaluation on VLGuard

0.23ASRLLaVA-1.5-7B + Mixed-SFT
-3.389621.042745.47569.9073Mar 14, 2025Apr 19, 2025May 26, 2025Jul 2, 2025Aug 7, 2025Sep 13, 2025Oct 20, 2025
Updated 6mo ago

Evaluation results

MethodLinks
LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.030.23————
LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.030.23————
LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.031.29————
LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.032.26————
LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.032.49————
LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.033.87————
LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.034.07————
Claude-3.5-SonnetCategory=Online MLLMs2025.105.21————
LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.035.54————
GPT-4oCategory=Online MLLMs2025.106.11————
CrossGuardCategory=MLLM Guardrails2025.107.24————
Qwen2.5-VL-7BCategory=Offline MLLMs2025.109.73————
LLaVA-1.5-7B + RMU-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0310.18————
LLaVA-1.5-7B + NPO-UnlearningBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0312.92————
JailDAMCategory=MLLM Guardrails2025.1015.38————
HiddenDetectCategory=MLLM Guardrails2025.1026.02————
LLaVA-1.5-7B (base)Category=Offline MLLMs2025.1046.38————
LLaVA-1.5-7B + Post-hocBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0346.83————
LLaVA-1.5-7B + Mixed-SFTBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0354.98————
LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.0364.25————
LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=Before one-word, Shot setting=3-shot2025.0365.66————
LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.0372.9————
LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=After GCG, Shot setting=3-shot2025.0373.75————
Llama-Guard3-VisionCategory=MLLM Guardrails2025.1089.82————
LLaVA-1.5-7BBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0390.27————
LlavaGuardCategory=MLLM Guardrails2025.1090.42————
LLaVA-1.5-7B + Unsafe-FilterBackbone=LLaVA-1.5-7B, Attack Type=After one-word, Shot setting=3-shot2025.0390.72————
LLaVA-1.5-13BFine-tuning Strategy=Full fine-tuning, Safety Approach=Baseline2025.03—68.191.86——
LLaVA-1.5-13BFine-tuning Strategy=LoRA fine-tuning, Safety Approach=Baseline2025.03—67.8793.89——
LLaVA-1.5-7BFine-tuning Strategy=Full fine-tuning, Safety Approach=Baseline2025.03—64.2590.27——
LLaVA-1.5-7BFine-tuning Strategy=LoRA fine-tuning, Safety Approach=Baseline2025.03—64.7295.25——
Mixed-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03—0.2354.98——
Mixed-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03—0.4569.23——
Mixed-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03—0.4557.01——
Mixed-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03—0.4552.94——
MM-RLHFArch.=Qwen2.5 VL 7B2025.05———78.43.8
NPO-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03—2.4912.92——
NPO-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03—4.5618.29——
NPO-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03—1.8911.7——
NPO-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03—3.3613.53——
Posthoc-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03—0.2346.83——
Posthoc-SFTBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03—0.2351.81——
Posthoc-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03—1.5869.23——
Posthoc-SFTBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03—0.2342.08——
Qwen2.5 VLArch.=Qwen2.5 VL-3B2025.05———57.44.1
Qwen2.5 VLArch.=Qwen2.5 VL-7B2025.05———474.2
RMU-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03—1.2910.18——
RMU-UnlearningBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03—3.8711.14——
RMU-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03—1.298.96——
RMU-UnlearningBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03—2.7510.18——
Self-ReminderArch.=Qwen2.5 VL-7B2025.05———682.8
SPA-VLArch.=Qwen2.5 VL 7B2025.05———972.4
Unsafe-FilterBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=Full fine-tuning2025.03—65.6690.72——
Unsafe-FilterBase Model=LLaVA-1.5-7B, Fine-tuning Strategy=LoRA fine-tuning2025.03—67.1993.89——
Unsafe-FilterBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=Full fine-tuning2025.03—67.6592.99——
Unsafe-FilterBase Model=LLaVA-1.5-13B, Fine-tuning Strategy=LoRA fine-tuning2025.03—66.9794.34——
VideoLLaMA3Arch.=VideoLLaMA3-2B2025.05———42.43.8
VideoLLaMA3Arch.=VideoLLaMA3-7B2025.05———58.43.8
VideoSafety-R1Arch.=Qwen2.5VL-3B2025.05———94.24.5
VideoSafety-R1Arch.=Qwen2.5VL-7B2025.05———99.64.2
VideoSafety-R1Arch.=VideoLLaMA-2B2025.05———86.74.1
VLGuardArch.=Qwen2.5 VL 7B2025.05———1002.7