List your company

ResearchFormerly WizWand

HarmBench

Benchmarks

TaskDatasetSOTA resultTrendResults

HarmBench Visual Riddle

6.3ASR (K=1)

6

HarmBench Textual Riddle

67.9ASR (K=1)

6

HarmBench Visual Text Replacement

3.6ASR (K=1)

6

HarmBench Visual Object Replacement

34.5ASR (K=1)

6

HarmBench Textual Replacement

55.4ASR (K=1)

6

HarmBench Visual Cipher

90.6ASR (K=1)

6

HarmBench Textual Cipher

74.8ASR (K=1)

6

Jailbreak Attack

HarmBench example-based Llama3 RR (8B)

0Attack Success Rate

6

Jailbreak Attack

HarmBench example-based Llama2 7B

32Attack Success Rate (ASR)

6

Embedding Attack

HarmBench Llama-3-8B

19MSS

5

Embedding Attack

HarmBench Mistral-7B

9MSS

5

Embedding Attack

HarmBench Llama-3.1-8B

19MSS

5

Jailbreak Attack

Harmbench Target: Grok-3

95.67ASR

5

Jailbreak Attack

Harmbench Target: Gemini-2.0-flash

78.33Attack Success Rate (ASR)

5

Jailbreak Attack

Harmbench Target: GPT-4o

82.67ASR

5

Jailbreak Attack Efficiency

HarmBench

15,198Attack Execution Time (s)

5

HarmBench Claude-Sonnet-3.5 (held-out test)

60ASR

5

HarmBench Llama-3-8B (test)

0.98ASR

5

HarmBench Llama-2-7B (test)

36ASR

5

HarmBench Harmful 200 Standard Behaviors 13

13.8ASR (Length 25)

4

Random Insertion Attack

Harmful behaviors (HarmBench) 200 standard behaviors 13

0.3ASR (L=25)

4

Adversarial Robustness Evaluation

HarmBench

1Attack Success Rate (ASR)

4

Multi-label Safety Categorization

HarmBench Responses

27.26Macro Accuracy

4

Multi-label Safety Categorization

HarmBench prompts

0.543Macro Accuracy

4

Jailbreak Attack

HarmBench Judge

63.12Attack Success Rate (ASR)

4