MLLMU-Bench
Benchmarks
TaskDatasetSOTA resultTrendResults
MLLMU-Bench Forget Set
63.33Accuracy
51
MLLMU-Bench 5% (forget)
55Accuracy (Classification)
42
MLLMU-Bench (Forget Set)
64.5Rouge Score
37
MLLMU-Bench Forget Set
54.67Classification Accuracy
36
MLLMU-Bench (test)
47.86Classification Accuracy
34
MLLMU-Bench (Retain Set)
67.07Accuracy
32
MLLMU-Bench (test)
52.5Accuracy
32
MLLMU-Bench (Forget Set)
26.09Cloze Accuracy
32
MLLMU-Bench Retain Set 10% ratio
34.4Cloze Accuracy
30
MLLMU-Bench 10% ratio (test)
40Cloze Accuracy
30
MLLMU-Bench forget set, 10% ratio
40Cloze Accuracy
30
MLLMU-Bench 1.0 (Retain Set)
30Cloze Accuracy
30
MLLMU-Bench 1.0 (test)
38.37Cloze Accuracy
30
MLLMU-Bench forget set 1.0
35.14Cloze Accuracy
30
MLLMU-Bench (Retain Set)
53.1ROUGE-L
30
MLLMU-Bench (Retain Set)
24.52Accuracy
30
MLLMU-Bench (test)
34.5ROUGE-L
30
MLLMU-Bench (test)
13.04Accuracy
30
MLLMU-Bench Real Celebrity
56.41Class Acc
28
MLLMU-Bench LLaVA-1.5-7B (test 2)
62.8Forget Rate
24
MLLMU-Bench LLaVA-1.5-7B (test 1)
65.4Forget Rate
24
MLLMU-Bench 5% forget (Real)
78.59Classification Accuracy
21
MLLMU-Bench 5% forget (test)
62.5Classification Accuracy
21
MLLMU-Bench 5% forget
0.01Contextual Refusal Rate
18
MLLMU-Bench
29.6Forget VQA Accuracy
16