HarmBench
Benchmarks
Task NameDataset NameSOTA ResultTrendResults
HarmBench Visual Riddle
6.3ASR (K=1)
6
HarmBench Textual Riddle
67.9ASR (K=1)
6
HarmBench Visual Text Replacement
3.6ASR (K=1)
6
HarmBench Visual Object Replacement
34.5ASR (K=1)
6
HarmBench Textual Replacement
55.4ASR (K=1)
6
HarmBench Visual Cipher
90.6ASR (K=1)
6
HarmBench Textual Cipher
74.8ASR (K=1)
6
HarmBench example-based Llama3 RR (8B)
0Attack Success Rate
6
HarmBench example-based Llama2 7B
32Attack Success Rate (ASR)
6
HarmBench Llama-3-8B
19MSS
5
HarmBench Mistral-7B
9MSS
5
HarmBench Llama-3.1-8B
19MSS
5
Harmbench Target: Grok-3
95.67ASR
5
Harmbench Target: Gemini-2.0-flash
78.33Attack Success Rate (ASR)
5
Harmbench Target: GPT-4o
82.67ASR
5
HarmBench
15,198Attack Execution Time (s)
5
HarmBench Claude-Sonnet-3.5 (held-out test)
60ASR
5
HarmBench Llama-3-8B (test)
0.98ASR
5
HarmBench Llama-2-7B (test)
36ASR
5
HarmBench Harmful 200 Standard Behaviors 13
13.8ASR (Length 25)
4
Harmful behaviors (HarmBench) 200 standard behaviors 13
0.3ASR (L=25)
4
HarmBench
1Attack Success Rate (ASR)
4
HarmBench Responses
27.26Macro Accuracy
4
HarmBench prompts
0.543Macro Accuracy
4
HarmBench Judge
63.12Attack Success Rate (ASR)
4