List your company
ResearchFormerly WizWand

Multimodal Machine Unlearning Evaluation on MLLMU-Bench Forget Set

54.67Classification AccuracyVanilla
35.53440.50245.4750.438Feb 21, 2025Apr 7, 2025May 23, 2025Jul 8, 2025Aug 23, 2025Oct 8, 2025Nov 23, 2025
Updated 7mo ago

Evaluation results

MethodLinks
VanillaForget Ratio=15%, Backbone=Idefics-2-8B2025.0254.670.636.4246.33
VanillaForget Ratio=10%, Backbone=Idefics-2-8B2025.0254.480.6456.2746.55
VanillaForget Ratio=5%, Backbone=Idefics-2-8B2025.0253.80.636.2244.75
VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1151.870.5756.3426.62
VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1151.70.6456.7825.81
PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1149.730.5474.6326
VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1149.150.5946.426.97
PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1148.410.5614.7526.55
KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1147.60.5414.5723.44
NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1147.40.5155.0520.9
KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1146.80.5745.0420.46
PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1146.80.5584.5123.81
NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1145.610.5253.4122.76
NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1145.520.5094.3920.63
PromptingForget Ratio=5%, Backbone=Idefics-2-8B2025.0245.450.4923.9142.61
MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1144.850.5183.2819.42
KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1144.80.5794.1222.69
GABackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1144.40.4853.3817.19
GABackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1143.850.513.5120.91
Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1143.60.5073.0516
Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1143.470.5184.818.78
NPOForget Ratio=5%, Backbone=Idefics-2-8B2025.0243.290.5014.8739.77
SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.1143.280.5023.1216.85
MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1143.120.5233.6420.18
SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1143.080.4744.1718.02
MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1142.280.5313.7821.45
Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1141.60.5083.1618.79
SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.1141.030.4913.7720.14
GABackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.1140.930.5824.6217.33
Grad. Diff.Forget Ratio=5%, Backbone=Idefics-2-8B2025.0240.380.4263.9632.24
KL MinimizationForget Ratio=5%, Backbone=Idefics-2-8B2025.0239.690.4593.3936.79
GAForget Ratio=10%, Backbone=Idefics-2-8B2025.0237.810.4593.0931.05
MANUForget Ratio=5%, Backbone=Idefics-2-8B2025.0237.130.4133.232.11
MANUForget Ratio=10%, Backbone=Idefics-2-8B2025.0237.010.4673.4529.98
MANUForget Ratio=15%, Backbone=Idefics-2-8B2025.0236.490.3663.1730.67
GAForget Ratio=5%, Backbone=Idefics-2-8B2025.0236.270.4052.930.07