Malicious prompt detection on JailBreakV & GPT4V π = 0.005
98.06AUROCVLMGuard
Evaluation results
| Method | Links | ||
|---|---|---|---|
| VLMGuardBackbone=Qwen, Type=Activation, Single inference=true, Single LM=true2024.10 | 98.06 | 94.79 | |
| VLMGuardBackbone=LLaVA, Type=Activation, Single inference=true, Single LM=true2024.10 | 96.58 | 93 | |
| Gradient CuffBackbone=Qwen, Type=Uncertainty, Single inference=false, Single LM=true2024.10 | 94.56 | 80.56 | |
| GPT-5.4Backbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 94.18 | 89.4 | |
| ASTRABackbone=Qwen, Type=Activation, Single inference=true, Single LM=true2024.10 | 94.03 | 82.7 | |
| Gradient CuffBackbone=LLaVA, Type=Uncertainty, Single inference=false, Single LM=true2024.10 | 92.64 | 78.55 | |
| Qwen2.5-VL-72B-InstructBackbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 91.76 | 85.33 | |
| HiddenDetectBackbone=LLaVA, Type=Activation, Single inference=true, Single LM=true2024.10 | 90.27 | 88.53 | |
| ASTRABackbone=LLaVA, Type=Activation, Single inference=true, Single LM=true2024.10 | 90.13 | 83.86 | |
| InternVL3-78B-InstructBackbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 89.22 | 82.51 | |
| Self-detectionBackbone=Qwen, Type=LLM, Single inference=true, Single LM=true2024.10 | 86.16 | 72.52 | |
| LlamaGuard3-VisionBackbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 85.39 | 69.57 | |
| PerplexityBackbone=Qwen, Type=Uncertainty, Single inference=true, Single LM=true2024.10 | 84.84 | 63.68 | |
| Self-detectionBackbone=LLaVA, Type=LLM, Single inference=true, Single LM=true2024.10 | 84.53 | 71.14 | |
| HiddenDetectBackbone=Qwen, Type=Activation, Single inference=true, Single LM=true2024.10 | 83.32 | 76.57 | |
| PerplexityBackbone=LLaVA, Type=Uncertainty, Single inference=true, Single LM=true2024.10 | 81.19 | 61.4 | |
| GradSafeBackbone=Qwen, Type=Uncertainty, Single inference=true, Single LM=true2024.10 | 79.02 | 78.32 | |
| GradSafeBackbone=LLaVA, Type=Uncertainty, Single inference=true, Single LM=true2024.10 | 77.35 | 77.1 | |
| MirrorCheckBackbone=Qwen, Type=Denoising, Single inference=true, Single LM=false2024.10 | 66.4 | 57.25 | |
| Ovis2-34BBackbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 64.83 | 71.4 | |
| JailGuardBackbone=Qwen, Type=Mutation, Single inference=false, Single LM=true2024.10 | 64.25 | 62.15 | |
| MirrorCheckBackbone=LLaVA, Type=Denoising, Single inference=true, Single LM=false2024.10 | 62.93 | 54.8 | |
| JailGuardBackbone=LLaVA, Type=Mutation, Single inference=false, Single LM=true2024.10 | 62.71 | 61.34 | |
| CIDERBackbone=Qwen, Type=Denoising, Single inference=true, Single LM=false2024.10 | 59.4 | 59.44 | |
| CIDERBackbone=LLaVA, Type=Denoising, Single inference=true, Single LM=false2024.10 | 57.34 | 59.99 | |
| LLaVAGuardBackbone=N/A, Type=LLM, Single inference=true, Single LM=false2024.10 | 51.65 | 59.62 |