Multimodal Language Model Unlearning on MLLMU-Bench 1.0 (Retain Set)
30Cloze AccuracyRMU
Evaluation results
| Method | Links | |||
|---|---|---|---|---|
| RMUArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 30 | 5.05 | 50.1 | |
| ICU-TArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 26.96 | 41.7 | 40.6 | |
| KL_MinArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 26.36 | 39.53 | 50.9 | |
| BaseArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 25.65 | 42.98 | 50 | |
| Noise-TArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 25.65 | 42.22 | 48.1 | |
| POArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 24.82 | 39.01 | 26 | |
| VGIDArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 24.46 | 42.6 | 47.2 | |
| NPOArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 22.81 | 41.81 | 49.8 | |
| GAArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 21.86 | 40.28 | 46.8 | |
| RMUArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 20 | 47.4 | 50.9 | |
| GA_DiffArchitecture=LLaVA-1.5-7B, Forget Ratio=15%2026.05 | 19.19 | 53.8 | 49.8 | |
| GAArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 12.06 | 32.86 | 39.1 | |
| Noise-TArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 11.05 | 59.3 | 34.1 | |
| ICU-TArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 10.55 | 61 | 37.2 | |
| VGIDArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 10.3 | 61.11 | 33.3 | |
| GA_DiffArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 5.08 | 21.32 | 47.1 | |
| KL_MinArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 4.96 | 17.39 | 25.7 | |
| ICU-TArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 4.86 | 26.76 | 43.9 | |
| GA_DiffArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 4.55 | 53.74 | 46.5 | |
| POArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 4.55 | 53.13 | 45.6 | |
| RMUArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 4.55 | 51.52 | 46 | |
| POArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 4.25 | 27.28 | 51.3 | |
| BaseArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 4.04 | 53.54 | 48.7 | |
| NPOArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 4.04 | 53.94 | 49.9 | |
| VGIDArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 4.02 | 27.52 | 48.1 | |
| GAArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 3.54 | 52.93 | 49.3 | |
| KL_MinArchitecture=Qwen2.5-VL-3B, Forget Ratio=15%2026.05 | 3.54 | 53.94 | 48.8 | |
| BaseArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 3.31 | 27.28 | 49.8 | |
| NPOArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 3.19 | 27.71 | 48.7 | |
| Noise-TArchitecture=LLaVA-1.5-13B, Forget Ratio=15%2026.05 | 3.19 | 29.69 | 49.7 |