# When Background Matters: Breaking Medical Vision Language Models by Transferable Attack

Source: [When Background Matters: Breaking Medical Vision Language Models by Transferable Attack](<https://www.sota2.com/research/paper/69e7049c7d0172140ec7ed5b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-24T06:00:01.997Z

Authors: Akash Ghosh, Subhadip Baidya, Sriparna Saha, Xiuying Chen

Published: 2026-04-19T08:17:43.000Z

## Abstract

Vision-Language Models (VLMs) are increasingly used in clinical diagnostics, yet their robustness to adversarial attacks remains largely unexplored, posing serious risks. Existing medical attacks focus on secondary objectives such as model stealing or adversarial fine-tuning, while transferable attacks from natural images introduce visible distortions that clinicians can easily detect. To address this, we propose MedFocusLeak, a highly transferable black-box multimodal attack that induces incorrect yet clinically plausible diagnoses while keeping perturbations imperceptible. The method injects coordinated perturbations into non-diagnostic background regions and employs an attention distraction mechanism to shift the model's focus away from pathological areas. Extensive evaluations across six medical imaging modalities show that MedFocusLeak achieves state-of-the-art performance, generating misleading yet realistic diagnostic outputs across diverse VLMs. We further introduce a unified evaluation framework with novel metrics that jointly capture attack success and image fidelity, revealing a critical weakness in the reasoning capabilities of modern clinical VLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Adversarial Attack on Medical Imaging Dataset 1,000 images 1.0 (test)](<https://www.sota2.com/research/sota/adversarial-attack-on-medical-imaging-dataset-1-000-images-1-0-test.md>): Adversarial Attack; Medical Imaging Dataset 1,000 images 1.0 (test); MTR: 79.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.17318>)

[PDF](<https://arxiv.org/pdf/2604.17318.pdf>)

[Implementation](<https://github.com/AkashGhosh/When-Background-Matters-Breaking-Medical-Vision-Language-Models-by-Transferable-Attack>)