# Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection Source: [Leave My Images Alone: Preventing Multi-Modal Large Language Models from Analyzing Images via Visual Prompt Injection]() Collection index: [llms.txt]() Data updated: 2026-04-14T05:00:04.234Z Authors: Zedian Shao, Hongbin Liu, Yuepeng Hu, Neil Zhenqiang Gong Published: 2026-04-10T06:37:46.000Z ## Abstract Multi-modal large language models (MLLMs) have emerged as powerful tools for analyzing Internet-scale image data, offering significant benefits but also raising critical safety and societal concerns. In particular, open-weight MLLMs may be misused to extract sensitive information from personal images at scale, such as identities, locations, or other private details. In this work, we propose ImageProtector, a user-side method that proactively protects images before sharing by embedding a carefully crafted, nearly imperceptible perturbation that acts as a visual prompt injection attack on MLLMs. As a result, when an adversary analyzes a protected image with an MLLM, the MLLM is consistently induced to generate a refusal response such as "I'm sorry, I can't help with that request." We empirically demonstrate the effectiveness of ImageProtector across six MLLMs and four datasets. Additionally, we evaluate three potential countermeasures, Gaussian noise, DiffPure, and adversarial training, and show that while they partially mitigate the impact of ImageProtector, they simultaneously degrade model accuracy and/or efficiency. Our study focuses on the practically important setting of open-weight MLLMs and large-scale automated image analysis, and highlights both the promise and the limitations of perturbation-based privacy protection. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Refusal Rate Evaluation on VQA v2](): Refusal Rate Evaluation; VQA v2; Refusal Rate: 0.77. - [Refusal Rate Evaluation on GQA](): Refusal Rate Evaluation; GQA; Refusal Rate: 77. - [Refusal Rate Evaluation on CelebA](): Refusal Rate Evaluation; CelebA; Refusal Rate: 81. - [Refusal Rate Evaluation on TextVQA](): Refusal Rate Evaluation; TextVQA; Refusal Rate: 70. - [Visual Question Answering on VQA v2 (Probing Refusal Metrics)](): Visual Question Answering; VQA v2; Refusal Rate (Exact): 94. ## Original sources and implementation [Original paper]() [PDF]()