Attack Success Rate (ASR) on AdvBench (test)
91.35ASR (GPT-3.5 Turbo)ReNeLLM
Evaluation results
| Method | Links | |||||||||
|---|---|---|---|---|---|---|---|---|---|---|
| ReNeLLMAttack Method Category=Black-box Attack Method2024.10 | 91.35 | 83.85 | 68.08 | 85.38 | 55.77 | 2.88 | 1.54 | 64.23 | 56.64 | |
| FlipAttackAttack Method Category=Black-box Attack Method2024.10 | 88.65 | 94.04 | 86.73 | 90.77 | 61.92 | 88.08 | 27.5 | 94.04 | 78.97 | |
| CodeChameleonAttack Method Category=Black-box Attack Method2024.10 | 84.62 | 92.64 | 22.27 | 92.67 | 51.54 | 20.77 | 0.58 | 87.69 | 56.6 | |
| AutoDANAttack Method Category=White-box Attack Method2024.10 | 81.73 | 31.92 | 26.54 | 46.92 | 27.31 | 1.35 | 3.27 | 77.31 | 37.04 | |
| TAPAttack Method Category=Black-box Attack Method2024.10 | 60.54 | 36.81 | 40.97 | 61.63 | 6.54 | 0 | 0.77 | 29.42 | 29.58 | |
| PAIRAttack Method Category=Black-box Attack Method2024.10 | 59.68 | 23.96 | 27.18 | 47.83 | 3.46 | 0 | 2.12 | 2.12 | 20.79 | |
| Base64Attack Method Category=Black-box Attack Method2024.10 | 45 | 0.19 | 0.77 | 57.88 | 3.08 | 0.19 | 0 | 1.92 | 13.63 | |
| GCGAttack Method Category=White-box Attack Method2024.10 | 42.88 | 0.38 | 1.73 | 1.15 | 2.5 | 0 | 0 | 10.58 | 7.4 | |
| DeepInceptionAttack Method Category=Black-box Attack Method2024.10 | 41.13 | 5.83 | 27.27 | 40.04 | 20.38 | 0 | 1.92 | 49.81 | 23.3 | |
| GPTFuzzerAttack Method Category=Black-box Attack Method2024.10 | 37.79 | 51.35 | 42.5 | 66.73 | 41.35 | 0 | 0 | 73.27 | 39.12 | |
| MACAttack Method Category=White-box Attack Method2024.10 | 36.15 | 0.19 | 0.77 | 0.58 | 1.92 | 0 | 0 | 10 | 6.2 | |
| COLD-AttackAttack Method Category=White-box Attack Method2024.10 | 34.23 | 0.19 | 0.77 | 0.19 | 10.92 | 0.19 | 0.77 | 6.54 | 5.6 | |
| ArtPromptAttack Method Category=Black-box Attack Method2024.10 | 14.06 | 1.92 | 1.75 | 4.42 | 0.77 | 0.58 | 0.38 | 19.62 | 5.44 | |
| PromptAttackAttack Method Category=Black-box Attack Method2024.10 | 13.46 | 0.96 | 0.96 | 1.92 | 0 | 0 | 0 | 0 | 2.16 | |
| DRAAttack Method Category=Black-box Attack Method2024.10 | 9.42 | 22.12 | 31.73 | 40.96 | 2.69 | 0 | 0 | 56.54 | 20.43 | |
| SelfCipherAttack Method Category=Black-box Attack Method2024.10 | 0 | 0 | 41.73 | 0 | 0 | 0 | 0 | 0 | 5.22 |