Multimodal Machine Unlearning Evaluation on MLLMU-Bench Forget Set
54.67Classification AccuracyVanilla
Evaluation Results
| Method | Links | ||||
|---|---|---|---|---|---|
| VanillaForget Ratio=15%, Backbone=Idefics-2-8B2025.02 | 54.67 | 0.63 | 6.42 | 46.33 | |
| VanillaForget Ratio=10%, Backbone=Idefics-2-8B2025.02 | 54.48 | 0.645 | 6.27 | 46.55 | |
| VanillaForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 53.8 | 0.63 | 6.22 | 44.75 | |
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 51.87 | 0.575 | 6.34 | 26.62 | |
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 51.7 | 0.645 | 6.78 | 25.81 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 49.73 | 0.547 | 4.63 | 26 | |
| VanillaBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 49.15 | 0.594 | 6.4 | 26.97 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 48.41 | 0.561 | 4.75 | 26.55 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 47.6 | 0.541 | 4.57 | 23.44 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 47.4 | 0.515 | 5.05 | 20.9 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 46.8 | 0.574 | 5.04 | 20.46 | |
| PromptingBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 46.8 | 0.558 | 4.51 | 23.81 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 45.61 | 0.525 | 3.41 | 22.76 | |
| NPOBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 45.52 | 0.509 | 4.39 | 20.63 | |
| PromptingForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 45.45 | 0.492 | 3.91 | 42.61 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 44.85 | 0.518 | 3.28 | 19.42 | |
| KL Min.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 44.8 | 0.579 | 4.12 | 22.69 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 44.4 | 0.485 | 3.38 | 17.19 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 43.85 | 0.51 | 3.51 | 20.91 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 43.6 | 0.507 | 3.05 | 16 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 43.47 | 0.518 | 4.8 | 18.78 | |
| NPOForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 43.29 | 0.501 | 4.87 | 39.77 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=5%2025.11 | 43.28 | 0.502 | 3.12 | 16.85 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 43.12 | 0.523 | 3.64 | 20.18 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 43.08 | 0.474 | 4.17 | 18.02 | |
| MMUnlearnerBackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 42.28 | 0.531 | 3.78 | 21.45 | |
| Grad. Diff.Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 41.6 | 0.508 | 3.16 | 18.79 | |
| SINEPROJECT(NPO)Backbone=LLaVA-1.5-7B, Deletion Ratio=10%2025.11 | 41.03 | 0.491 | 3.77 | 20.14 | |
| GABackbone=LLaVA-1.5-7B, Deletion Ratio=15%2025.11 | 40.93 | 0.582 | 4.62 | 17.33 | |
| Grad. Diff.Forget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 40.38 | 0.426 | 3.96 | 32.24 | |
| KL MinimizationForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 39.69 | 0.459 | 3.39 | 36.79 | |
| GAForget Ratio=10%, Backbone=Idefics-2-8B2025.02 | 37.81 | 0.459 | 3.09 | 31.05 | |
| MANUForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 37.13 | 0.413 | 3.2 | 32.11 | |
| MANUForget Ratio=10%, Backbone=Idefics-2-8B2025.02 | 37.01 | 0.467 | 3.45 | 29.98 | |
| MANUForget Ratio=15%, Backbone=Idefics-2-8B2025.02 | 36.49 | 0.366 | 3.17 | 30.67 | |
| GAForget Ratio=5%, Backbone=Idefics-2-8B2025.02 | 36.27 | 0.405 | 2.9 | 30.07 |