MM-SafetyBench
Benchmarks
TaskDatasetSOTA resultTrendResults
MM-SafetyBench
0Average ASR
98
MM-SafetyBench
95Attack Success Rate (ASR)
82
MM-SafetyBench
0.2Attack Success Rate (ASR)
56
MM-SafetyBench 1.0 (test)
96Attack Success Rate
48
MM-SafetyBench (full)
95.42ASR
40
MM-SafetyBench
2.73Safety Score
34
MM-SafetyBench
100WASR
32
MM-SafetyBench (tiny)
99.16ASR
25
MM-SafetyBench v1.0 (test)
0.6ASR
24
MM-SafetyBench
99.18AUROC
23
MM-SafetyBench (test)
68.95Helpfulness Score
20
MM-SafetyBench SD+TYPO
81.4ASR
18
MM-SafetyBench TYPO
79.6Attack Success Rate (ASR)
18
MM-SafetyBench SD
80.6ASR
18
MM-SafetyBench OOD
0.71ASR
16
MM-SafetyBench (avg)
99MS-R
15
MM-SafetyBench Physical Harm scenario
6ASR
15
MM-SafetyBench (full)
1.03ASR (Illegal Activity - S)
12
MM-SafetyBench SD_TYPO
12Average ASR
10
MM-SafetyBench SD
0.09Average ASR
10
MM-SafetyBench OCR (test)
0Illegal Activity Rate
10
MM-SafetyBench (MMSB)
1.8Attack Success Rate (V-T AVG)
9
MM-SafetyBench unseen attack types
2.35ASR (SD)
9
MM-SafetyBench++
100Illegal Activity Unsafe Refusal Rate
9
MM-SafetyBench SD + TYPO + SD_TYPO (test)
0.08ASR Score
8