# To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models

Source: [To Agree or To Be Right? The Grounding-Sycophancy Tradeoff in Medical Vision-Language Models](<https://www.sota2.com/research/paper/69c36bfc0636e5bf4afc6424>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-03-26T05:00:04.122Z

Authors: OFM Riaz Rahman Aranya, Kevin Desai

Published: 2026-03-23T22:51:54.000Z

## Abstract

Vision-language models (VLMs) adapted to the medical domain have shown strong performance on visual question answering benchmarks, yet their robustness against two critical failure modes, hallucination and sycophancy, remains poorly understood, particularly in combination. We evaluate six VLMs (three general-purpose, three medical-specialist) on three medical VQA datasets and uncover a grounding-sycophancy tradeoff: models with the lowest hallucination propensity are the most sycophantic, while the most pressure-resistant model hallucinates more than all medical-specialist models. To characterize this tradeoff, we propose three metrics: L-VASE, a logit-space reformulation of VASE that avoids its double-normalization; CCS, a confidence-calibrated sycophancy score that penalizes high-confidence capitulation; and Clinical Safety Index (CSI), a unified safety index that combines grounding, autonomy, and calibration via a geometric mean. Across 1,151 test cases, no model achieves a CSI above 0.35, indicating that none of the evaluated 7-8B parameter VLMs is simultaneously well-grounded and robust to social pressure. Our findings suggest that joint evaluation of both properties is necessary before these models can be considered for clinical use. Code is available at https://github.com/UTSA-VIRLab/AgreeOrRight

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Medical Visual Question Answering on VQA-RAD (L-VASE, CCS, CSI)](<https://www.sota2.com/research/sota/medical-visual-question-answering-on-vqa-rad-l-vase-ccs-csi.md>): Medical Visual Question Answering; VQA-RAD; L-VASE: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.22623>)

[PDF](<https://arxiv.org/pdf/2603.22623.pdf>)