# Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs Source: [Dictionary-Aligned Concept Control for Safeguarding Multimodal LLMs]() Collection index: [llms.txt]() Data updated: 2026-04-14T05:00:04.234Z Authors: Jinqi Luo, Jinyu Yang, Tal Neiman, Lei Fan, Bing Yin, Son Tran, Mubarak Shah, Ren\\'e Vidal Published: 2026-04-10T01:01:56.000Z ## Abstract Multimodal Large Language Models (MLLMs) have been shown to be vulnerable to malicious queries that can elicit unsafe responses. Recent work uses prompt engineering, response classification, or finetuning to improve MLLM safety. Nevertheless, such approaches are often ineffective against evolving malicious patterns, may require rerunning the query, or demand heavy computational resources. Steering the activations of a frozen model at inference time has recently emerged as a flexible and effective solution. However, existing steering methods for MLLMs typically handle only a narrow set of safety-related concepts or struggle to adjust specific concepts without affecting others. To address these challenges, we introduce Dictionary-Aligned Concept Control (DACO), a framework that utilizes a curated concept dictionary and a Sparse Autoencoder (SAE) to provide granular control over MLLM activations. First, we curate a dictionary of 15,000 multimodal concepts by retrieving over 400,000 caption-image stimuli and summarizing their activations into concept directions. We name the dataset DACO-400K. Second, we show that the curated dictionary can be used to intervene activations via sparse coding. Third, we propose a new steering approach that uses our dictionary to initialize the training of an SAE and automatically annotate the semantics of the SAE atoms for safeguarding MLLMs. Experiments on multiple MLLMs (e.g., QwenVL, LLaVA, InternVL) across safety benchmarks (e.g., MM-SafetyBench, JailBreakV) show that DACO significantly improves MLLM safety while maintaining general-purpose capabilities. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Response Safety on MM-SafetyBench (avg)](): Response Safety; MM-SafetyBench (avg); MS-R: 99. - [Response Safety on JailBreakV-28K (avg)](): Response Safety; JailBreakV-28K (avg); JBV-R Score: 0.975. - [Multimodal Open-ended Utility Evaluation on MM-Vet v2 (test)](): Multimodal Open-ended Utility Evaluation; MM-Vet v2 (test); Utility Score: 63.7. - [Multimodal Open-ended Utility Evaluation on MM-Vet v1 (test)](): Multimodal Open-ended Utility Evaluation; MM-VET v1 (test); Utility Score: 65. - [General-Purpose Utility on General-Purpose Utility Evaluation v1 (avg)](): General-Purpose Utility; General-Purpose Utility Evaluation v1 (avg); Fluency: 90.5. - [Multi-discipline Multimodal Understanding on MMMU](): Multi-discipline Multimodal Understanding; MMMU; Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()