# Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models Source: [Which Concepts to Forget and How to Refuse? Decomposing Concepts for Continual Unlearning in Large Vision-Language Models]() Collection index: [llms.txt]() Data updated: 2026-03-25T05:00:11.826Z Authors: Hyundong Jin, Dongyoon Han, Eunwoo Kim Published: 2026-03-23T02:07:40.000Z ## Abstract Continual unlearning poses the challenge of enabling large vision-language models to selectively refuse specific image-instruction pairs in response to sequential deletion requests, while preserving general utility. However, sequential unlearning updates distort shared representations, creating spurious associations between vision-language pairs and refusal behaviors that hinder precise identification of refusal targets, resulting in inappropriate refusals. To address this challenge, we propose a novel continual unlearning framework that grounds refusal behavior in fine-grained descriptions of visual and textual concepts decomposed from deletion targets. We first identify which visual-linguistic concept combinations characterize each forget category through a concept modulator, then determine how to generate appropriate refusal responses via a mixture of refusal experts, termed refusers, each specialized for concept-aligned refusal generation. To generate concept-specific refusal responses across sequential tasks, we introduce a multimodal, concept-driven routing scheme that reuses refusers for tasks sharing similar concepts and adapts underutilized ones for novel concepts. Extensive experiments on vision-language benchmarks demonstrate that the proposed framework outperforms existing methods by generating concept-grounded refusal responses and preserving the general utility across unlearning sequences. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Knowledge Unlearning on 16-task Sequential Unlearning Forgotten Data Avg](): Knowledge Unlearning; 16-task Sequential Unlearning Forgotten Data Avg; CRR: 90.9. - [Knowledge Retention on 16-task Sequential Unlearning Retained Data Last](): Knowledge Retention; 16-task Sequential Unlearning Retained Data Last; Specificity: 96.54. - [Knowledge Retention on Sequential Unlearning 16-task Retained Data Avg](): Knowledge Retention; Sequential Unlearning 16-task Retained Data Avg; Specificity: 95.09. - [Knowledge Unlearning on 16-task Sequential Unlearning Forgotten Data Last](): Knowledge Unlearning; 16-task Sequential Unlearning Forgotten Data Last; Context-aware Refusal Rate (CRR): 84.54. ## Original sources and implementation [Original paper]() [PDF]()