HarmBench
Benchmarks
TaskDatasetSOTA resultTrendResults
HarmBench Standard set
98CodeCha
8
HarmBench Hades
1Safety Score (1-ASR)
8
HarmBench-QR
0.998Safety Score (1-ASR)
8
HarmBench FigStep
1Safety Score (1-ASR)
8
HarmBench Direct
99.75Safety Score (1-ASR)
8
HarmBench (50 randomly sampled questions)
84ASR
8
HarmBench (test)
98.74ASR (%)
8
HarmBench (test)
81AASR
8
HarmBench (test)
14Harm Score (HS)
7
HarmBench target: Mistral-7B-v2-RR
70.31Attack Success Rate
7
HarmBench target: LLaMA-3-8B-RR
67.81Attack Success Rate
7
HarmBench
96.44Safety F1
7
HarmBench
100MCA Accuracy
7
Harmbench TrojanPraise (full)
4.15Harmful Score
7
HarmBench Contextual
100String-match Refusal Rate
6
HarmBench Standard
98.2Refusal Rate (String Match)
6
HarmBench Ctx
80.6Refusal Rate
6
HarmBench Std
80.8Refusal Rate
6
HarmBench out
67.29ASR (OM)
6
HarmBench (held-out prompts)
50.9ASR
6
HarmBench (overall)
5ASR
6
HarmBench FigStep
3ASR (K=1)
6
HarmBench HADES
1ASR (K=1)
6
HarmBench SD+TYPO
45.3ASR (K=1)
6
HarmBench SD
6.3ASR (K=1)
6