Multimodal Machine Unlearning Evaluation on MLLMU-Bench (test)
47.86Classification AccuracyVanilla
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 47.86 | 0.539 | 4.89 | 23.01 | |
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 47.53 | 0.502 | 4.08 | 25.33 | |
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 47.41 | 0.51 | 5.2 | 25.43 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 47.29 | 0.479 | 4.21 | 24.11 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 46.81 | 0.483 | 3.67 | 24.56 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 46.42 | 0.408 | 4.25 | 21.66 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 45.2 | 0.396 | 4.54 | 20.04 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 44.87 | 0.415 | 4.18 | 21.99 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 44.44 | 0.347 | 3.91 | 20 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 43.95 | 0.358 | 3.84 | 19.35 | |
| BaselineModel=LLaVA-1.5-7B, Stage=Baseline2026.07 | 43.52 | 0.516 | 5.2 | 25.73 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 43.41 | 0.383 | 3.83 | 16.19 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 43.2 | 0.439 | 3.78 | 21.09 | |
| POPSModel=LLaVA-1.5-7B, Attacked Method=KL Minimization, Stage=POPS2026.07 | 43.05 | 0.451 | 3.8 | 20.8 | |
| POPSModel=LLaVA-1.5-7B, Attacked Method=Gradient Ascent, Stage=POPS2026.07 | 42.9 | 0.461 | 4.72 | 18.2 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 42.75 | 0.42 | 3.29 | 20.5 | |
| KL MinimizationModel=LLaVA-1.5-7B, Stage=Unlearned2026.07 | 42.75 | 0.42 | 3.29 | 20.5 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 42.67 | 0.331 | 3.72 | 18.81 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 42.18 | 0.401 | 3.61 | 18.11 | |
| POPSModel=LLaVA-1.5-7B, Attacked Method=Gradient Diff, Stage=POPS2026.07 | 41.7 | 0.475 | 3.55 | 17.46 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 40.87 | 0.432 | 3.35 | 16.92 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 40.6 | 0.421 | 3.19 | 15.77 | |
| Gradient AscentModel=LLaVA-1.5-7B, Stage=Unlearned2026.07 | 40.2 | 0.387 | 3.83 | 14.51 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 40.15 | 0.445 | 3.52 | 17.88 | |
| MANUForget Ratio=10%, Backbone=Idefics-2-8B2025.02 | 39.75 | 0.355 | 3.51 | 21.88 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 39.64 | 0.371 | 3.7 | 17.67 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 39.33 | 0.439 | 4.01 | 17.88 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 39.08 | 0.414 | 3.07 | 14.5 | |
| Gradient DiffModel=LLaVA-1.5-7B, Stage=Unlearned2026.07 | 39.08 | 0.414 | 3.07 | 14.5 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 38.4 | 0.384 | 3.47 | 16.47 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 38.32 | 0.421 | 3.08 | 17.11 | |
| MANUForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 35.55 | 0.361 | 3.91 | 20.97 | |
| MANUForget Ratio=15%, Backbone=Idefics-2-8B2025.02 | 35.32 | 0.333 | 3.1 | 16.66 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 34.21 | 0.407 | 3.01 | 19.78 |