Jailbreak Attack Success Rate on SorryBench
8.9ASR (SorryBench)ChatBug-M
Evaluation results
| Method | Links | ||
|---|---|---|---|
| ChatBug-MTarget Model=Gemma-2, Platform=Ollama completion API2025.10 | 8.9 | — | |
| DITarget Model=Gemma-22025.10 | 10.2 | — | |
| ChatBug-OTarget Model=Gemma-22025.10 | 10.2 | — | |
| EnumAttackModel=gemini-2.0-flash, Grammar Guardrail=llama-guard-3-8b2025.03 | 15.5 | — | |
| EnumAttackModel=gpt-4o, Grammar Guardrail=llama-guard-3-8b2025.03 | 16.6 | — | |
| EnumAttackModel=gpt-4o-mini, Grammar Guardrail=llama-guard-3-8b2025.03 | 16.8 | — | |
| VCTarget Model=Gemma-22025.10 | 19.3 | — | |
| ChatBug-MTarget Model=Phi-4, Platform=Ollama completion API2025.10 | 24.3 | — | |
| BaselineModel=gemini-2.0-flash, Grammar Guardrail=llama-guard-3-8b2025.03 | 29.3 | — | |
| BaselineModel=gemini-2.0-flash, Grammar Guardrail=None2025.03 | 29.3 | — | |
| VCTarget Model=Llama-3.32025.10 | 29.8 | — | |
| ChatBug-OTarget Model=Llama-3.32025.10 | 30.2 | — | |
| VCTarget Model=Qwen-2.52025.10 | 31.1 | — | |
| DITarget Model=Llama-3.32025.10 | 31.6 | — | |
| ChatBug-MTarget Model=Llama-3.3, Platform=Ollama completion API2025.10 | 32.3 | — | |
| DITarget Model=Qwen-2.52025.10 | 32.3 | — | |
| BaselineModel=gpt-4o, Grammar Guardrail=llama-guard-3-8b2025.03 | 33.9 | — | |
| BaselineModel=gpt-4o, Grammar Guardrail=None2025.03 | 33.9 | — | |
| GPTFuzzerTarget Model=Phi-42025.10 | 33.9 | — | |
| ChatBug-MTarget Model=Qwen-2.5, Platform=Ollama completion API2025.10 | 37 | — | |
| ChatBug-OTarget Model=Qwen-2.52025.10 | 37.7 | — | |
| DITarget Model=Phi-42025.10 | 37.7 | — | |
| ChatBug-OTarget Model=Phi-42025.10 | 37.7 | — | |
| PAPTarget Model=Gemma-22025.10 | 42 | — | |
| BaselineModel=gpt-4o-mini, Grammar Guardrail=llama-guard-3-8b2025.03 | 42.5 | — | |
| BaselineModel=gpt-4o-mini, Grammar Guardrail=None2025.03 | 42.5 | — | |
| MetaBreakTarget Model=Qwen-2.52025.10 | 45.2 | — | |
| MetaBreakTarget Model=Gemma-22025.10 | 45.5 | — | |
| PAPTarget Model=Phi-42025.10 | 53 | — | |
| VCTarget Model=Phi-42025.10 | 58.6 | — | |
| PAPTarget Model=Llama-3.32025.10 | 60.9 | — | |
| GPTFuzzerTarget Model=Qwen-2.52025.10 | 62.5 | — | |
| GPTFuzzerTarget Model=Llama-3.32025.10 | 67.7 | — | |
| PAP*Target Model=Gemma-2, MetaBreak Integration=true2025.10 | 69.1 | 27.1 | |
| GPTFuzzer*Target Model=Phi-4, MetaBreak Integration=true2025.10 | 69.1 | 35.2 | |
| MetaBreakTarget Model=Phi-42025.10 | 69.8 | — | |
| PAPTarget Model=Qwen-2.52025.10 | 71.1 | — | |
| GPTFuzzer*Target Model=Llama-3.3, MetaBreak Integration=true2025.10 | 81.8 | 14.1 | |
| GPTFuzzerTarget Model=Gemma-22025.10 | 82.3 | — | |
| PAP*Target Model=Qwen-2.5, MetaBreak Integration=true2025.10 | 83.4 | 12.3 | |
| GPTFuzzer*Target Model=Qwen-2.5, MetaBreak Integration=true2025.10 | 83.4 | 20.9 | |
| DictAttackModel=gpt-4o-mini, Grammar Guardrail=llama-guard-3-8b, k=12025.03 | 83.8 | — | |
| PAP*Target Model=Llama-3.3, MetaBreak Integration=true2025.10 | 85 | 24.1 | |
| DictAttackModel=gpt-4o-mini, Grammar Guardrail=None, k=12025.03 | 85.2 | — | |
| PAP*Target Model=Phi-4, MetaBreak Integration=true2025.10 | 86.8 | 33.8 | |
| MetaBreakTarget Model=Llama-3.32025.10 | 87.3 | — | |
| EnumAttackModel=gemini-2.0-flash, Grammar Guardrail=None2025.03 | 89.5 | — | |
| DictAttackModel=gpt-5, Grammar Guardrail=enabled2025.03 | 91.6 | — | |
| GPTFuzzer*Target Model=Gemma-2, MetaBreak Integration=true2025.10 | 92.7 | 10.4 | |
| DictAttackModel=gpt-5, Grammar Guardrail=disabled2025.03 | 93.2 | — | |
| DictAttackModel=gemini-2.0-flash, Grammar Guardrail=llama-guard-3-8b, k=12025.03 | 93.9 | — | |
| DictAttackModel=gpt-oss-120b, Grammar Guardrail=enabled2025.03 | 94.8 | — | |
| DictAttackModel=gemini-2.0-flash, Grammar Guardrail=None, k=12025.03 | 95.5 | — | |
| DictAttackModel=gpt-oss-120b, Grammar Guardrail=disabled2025.03 | 96.4 | — | |
| EnumAttackModel=gpt-4o, Grammar Guardrail=None2025.03 | 96.4 | — | |
| EnumAttackModel=gpt-4o-mini, Grammar Guardrail=None2025.03 | 96.4 | — | |
| DictAttackModel=deepseek-r1, Grammar Guardrail=enabled2025.03 | 96.9 | — | |
| DictAttackModel=gpt-4o, Grammar Guardrail=llama-guard-3-8b, k=12025.03 | 96.9 | — | |
| DictAttackModel=gemini-2.5-pro, Grammar Guardrail=enabled2025.03 | 97.6 | — | |
| DictAttackModel=deepseek-r1, Grammar Guardrail=disabled2025.03 | 98.6 | — | |
| DictAttackModel=gpt-4o, Grammar Guardrail=None, k=12025.03 | 98.6 | — | |
| DictAttackModel=gemini-2.5-pro, Grammar Guardrail=disabled2025.03 | 99.3 | — |