Text-Based Jailbreak Attack on JailbreakV-28K (test)
75.23ASR (None-Template)Original
Evaluation results
| Method | Links | |||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| OriginalModel=MiniGPT-4-7B, Defense Strategy=None, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 75.23 | 54.12 | 83.62 | 72.44 | 37.33 | 85.63 | 73.18 | 45.27 | 86.64 | 72.09 | 55.42 | 86.64 | 67.15 | |
| OriginalModel=MiniGPT-4-13B, Defense Strategy=None, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 74.18 | 47.12 | 90.67 | 80.23 | 53.34 | 85.63 | 79.29 | 52.45 | 83.78 | 79.07 | 53.22 | 87.65 | 70.62 | |
| OriginalModel=LLaVA-1.6, Defense Strategy=None, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 66.12 | 37.45 | 78.58 | 67.34 | 37.56 | 77.57 | 69.23 | 40.78 | 82.61 | 66.67 | 39.45 | 81.6 | 60.27 | |
| FigStepModel=LLaVA-1.5, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 64.17 | 28.34 | 62.16 | 62.23 | 32.18 | 68.91 | 58.39 | 37.27 | 72.97 | 61.09 | 31.42 | 68.91 | 52.46 | |
| AdaShieldModel=MiniGPT-4-13B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 63.22 | 23.18 | 59.44 | 64.47 | 31.12 | 74.55 | 69.15 | 27.35 | 68.91 | 65.09 | 23.41 | 53.39 | 50.55 | |
| CoCAModel=LLaVA-1.6, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 61.23 | 39.17 | 62.16 | 61.34 | 41.28 | 52.7 | 63.11 | 35.22 | 55.41 | 61.09 | 37.36 | 52.7 | 51.37 | |
| FigStepModel=LLaVA-1.6, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 61.12 | 39.27 | 62.16 | 62.34 | 40.18 | 54.05 | 63.41 | 35.22 | 56.76 | 61.09 | 38.43 | 52.7 | 51.64 | |
| CoCAModel=LLaVA-1.5, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 59.18 | 26.34 | 35.13 | 58.42 | 30.29 | 35.13 | 49.22 | 32.11 | 28.37 | 59.07 | 23.39 | 37.83 | 39.87 | |
| OriginalModel=LLaVA-1.5, Defense Strategy=None, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 58.12 | 39.44 | 76.56 | 61.47 | 38.35 | 74.55 | 58.22 | 42.11 | 77.57 | 59.33 | 40.26 | 74.55 | 56.39 | |
| AdaShieldModel=MiniGPT-4-7B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 47.23 | 22.14 | 32.24 | 41.37 | 18.46 | 27.2 | 58.15 | 12.28 | 30.22 | 46.09 | 19.43 | 32.24 | 32.21 | |
| FigStepModel=MiniGPT-4-7B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 47.22 | 17.18 | 32.43 | 42.13 | 5.37 | 24.32 | 37.29 | 8.45 | 18.92 | 43.08 | 17.41 | 43.24 | 27.74 | |
| OriginalModel=Qwen-VL, Defense Strategy=None, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 46.12 | 3.27 | 12.09 | 52.23 | 6.18 | 9.07 | 53.34 | 3.42 | 7.05 | 53.11 | 4.36 | 15.11 | 22.99 | |
| FigStepModel=MiniGPT-4-13B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 44.13 | 20.28 | 58.11 | 43.22 | 18.47 | 47.3 | 40.36 | 27.14 | 48.65 | 42.09 | 19.42 | 58.11 | 37.41 | |
| AdaShieldModel=LLaVA-1.6, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 38.42 | 0 | 11.08 | 38.13 | 1.56 | 18.13 | 39.29 | 3.22 | 19.14 | 42.78 | 0.48 | 16.12 | 19.23 | |
| CoCAModel=MiniGPT-4-13B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 35.29 | 21.34 | 36.49 | 40.18 | 22.27 | 40.54 | 35.12 | 20.41 | 35.14 | 48.22 | 21.36 | 52.7 | 33.21 | |
| CoCAModel=MiniGPT-4-7B, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 35.18 | 18.27 | 22.97 | 40.34 | 21.14 | 31.08 | 35.29 | 18.42 | 27.03 | 48.11 | 21.36 | 40.54 | 29.74 | |
| FigStepModel=Qwen-VL, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 32.18 | 0 | 1.35 | 40.42 | 0 | 5.41 | 30.23 | 0 | 5.41 | 44.12 | 2.23 | 1.35 | 14.42 | |
| AdaShieldModel=LLaVA-1.5, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 32.14 | 0 | 5.41 | 31.36 | 0 | 4.23 | 30.22 | 0 | 5.41 | 29.67 | 0 | 10.81 | 12.86 | |
| AdaShieldModel=Qwen-VL, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 30.17 | 0 | 18.13 | 28.34 | 1.29 | 15.11 | 23.28 | 1.37 | 7.05 | 42.11 | 0.41 | 18.13 | 15.88 | |
| CoCAModel=Qwen-VL, Defense Strategy=Inference-time, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 29.12 | 6.27 | 13.51 | 27.34 | 7.18 | 5.41 | 27.29 | 1.42 | 13.51 | 28.47 | 13.36 | 13.51 | 15.69 | |
| IMMUNEModel=MiniGPT-4-13B, Defense Strategy=Safety model alignment, k=10, alpha=1, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 25.17 | 18.29 | 59.44 | 29.34 | 19.41 | 57.42 | 29.22 | 23.18 | 58.43 | 23.47 | 23.37 | 58.43 | 32.94 | |
| IMMUNEModel=MiniGPT-4-7B, Defense Strategy=Safety model alignment, k=10, alpha=1, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 18.23 | 6.12 | 44.59 | 11.34 | 8.27 | 29.72 | 17.18 | 8.45 | 27.02 | 16.09 | 10.37 | 43.24 | 18.34 | |
| IMMUNEModel=Qwen-VL, Defense Strategy=Safety model alignment, k=10, alpha=1, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 10.27 | 2.18 | 5.41 | 21.34 | 2.29 | 4.03 | 18.22 | 2.35 | 5.41 | 20.17 | 3.37 | 7.05 | 8.58 | |
| IMMUNEModel=LLaVA-1.5, Defense Strategy=Safety model alignment, k=10, alpha=1, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 9.23 | 0 | 0 | 8.14 | 0 | 0 | 1.47 | 0 | 0 | 5.32 | 0 | 0 | 2.1 | |
| IMMUNEModel=LLaVA-1.6, Defense Strategy=Safety model alignment, k=10, alpha=1, Jailbreak Classifier=Llama-Guard-3 Jailbreak Judge2024.11 | 5.23 | 0 | 0 | 8.14 | 0 | 0 | 8.45 | 0 | 0 | 4.67 | 0 | 0 | 2.45 |