Adversarial Attack on AdvBench (test)
100ASRAutoDAN
Evaluation results
| Method | Links | ||||
|---|---|---|---|---|---|
| AutoDANTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 100 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 100 | — | — | — | |
| PCAPType=MoE, Attacker Model=Kimi K2.5, Personas=6, Strategies per persona=22026.05 | 100 | 97.5 | 4.18 | 88 | |
| Semantic Representation AttackTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 99.81 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 99.81 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 99.81 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 99.62 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 99.62 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 99.62 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 99.62 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 99.62 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 99.42 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 99.42 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 99.42 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 99.42 | — | — | — | |
| PCAPType=MoE, Attacker Model=Mixtral 8x22B, Personas=6, Strategies per persona=22026.05 | 99.3 | 96.3 | 2.49 | 99.09 | |
| PCAPType=Dense, Attacker Model=Granite 3.3 8B, Personas=6, Strategies per persona=22026.05 | 98.7 | 95.3 | 1.98 | 169.06 | |
| PCAPType=MoE, Attacker Model=Granite 4.0 H Tiny, Personas=6, Strategies per persona=22026.05 | 98 | 94.3 | 2.41 | 135.6 | |
| Semantic Representation AttackTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 97.31 | — | — | — | |
| PCAPType=Dense, Attacker Model=Llama 3.3 70B, Personas=6, Strategies per persona=22026.05 | 97.3 | 93.3 | 2.98 | 90.26 | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 96.92 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 96.73 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 96.73 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 95.96 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 95.19 | — | — | — | |
| TAPType=MoE, Attacker Model=Kimi K2.52026.05 | 94 | 89 | 1.24 | 29.8 | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=12026.05 | 91.73 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=12026.05 | 90.58 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 90 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 89.62 | — | — | — | |
| TAPType=Dense, Attacker Model=Granite 3.3 8B2026.05 | 88.7 | 82.6 | 1.05 | 37.16 | |
| Semantic Representation AttackTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=12026.05 | 88.27 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 87.69 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 87.12 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 85.19 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 84.04 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 83.85 | — | — | — | |
| Semantic Representation AttackTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=12026.05 | 79.81 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 78.65 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 78.65 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 78.27 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 78.27 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 78.27 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 77.5 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 77.12 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=12026.05 | 76.54 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 74.81 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 70.38 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 69.81 | — | — | — | |
| TAPType=MoE, Attacker Model=Mixtral 8x22B2026.05 | 69.3 | 61.5 | 0.79 | 53.13 | |
| BEASTTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 67.31 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 66.73 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 64.04 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 63.85 | — | — | — | |
| TAPType=Dense, Attacker Model=Llama 3.3 70B2026.05 | 57.7 | 48.7 | 0.72 | 56.24 | |
| BEASTTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 50.96 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 47.5 | — | — | — | |
| Semantic Representation AttackTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=12026.05 | 44.62 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=52026.05 | 44.04 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 42.69 | — | — | — | |
| TAPType=MoE, Attacker Model=Granite 4.0 H Tiny2026.05 | 41 | 33.8 | 0.43 | 69.49 | |
| AutoDANTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 38.85 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 38.46 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 38.27 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 34.62 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 34.62 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 34.42 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 34.23 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 32.88 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 31.73 | — | — | — | |
| GCGTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=52026.05 | 31.54 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 30.96 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 29.04 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 26.54 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 25.19 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=52026.05 | 23.85 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 21.15 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Vicuna 7B, Defense Intensity=32026.05 | 15.77 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 15.19 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=32026.05 | 14.04 | — | — | — | |
| BEASTTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 13.46 | — | — | — | |
| GCGTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=32026.05 | 9.23 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 9.23 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 8.65 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 8.27 | — | — | — | |
| AutoDANTime Budget=15s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 8.08 | — | — | — | |
| AutoDANTime Budget=30s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 8.08 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 7.88 | — | — | — | |
| AutoDANTime Budget=60s, Target Model=Guanaco 7B, Defense Intensity=12026.05 | 7.88 | — | — | — | |
| BEASTTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=32026.05 | 7.69 | — | — | — | |
| GCGTime Budget=60s, Target Model=Mistral 7B, Defense Intensity=52026.05 | 6.54 | — | — | — | |
| Semantic Representation AttackTime Budget=60s, Target Model=Vicuna 13B, Defense Intensity=12026.05 | 6.15 | — | — | — | |
| BEASTTime Budget=15s, Target Model=Vicuna 7B, Defense Intensity=12026.05 | 5.77 | — | — | — |