# To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs

Source: [To See or To Please: Uncovering Visual Sycophancy and Split Beliefs in VLMs](<https://www.sota2.com/research/paper/69bcd46d0636e5bf4afc28b1>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-02T05:02:08.515Z

Authors: Rui Hong, Shuxue Quan

Published: 2026-03-19T00:15:05.000Z

## Abstract

When VLMs answer correctly, do they genuinely rely on visual information? We introduce a Tri-Layer Diagnostic Framework with three per-sample metrics: Latent Anomaly Detection, Visual Necessity Score, and Competition Score, which disentangle perception, dependency, and alignment failures. Across 9 VLMs and 9,000 model-sample pairs under counterfactual blind, noise, and conflict interventions, 72.9% of samples exhibit Visual Sycophancy, a Split Beliefs pattern in which internal evidence is preserved yet a hallucinated answer is decoded, while zero samples show Robust Refusal, indicating that current alignment training has eliminated refusal as a decoding outcome. Scaling within the Qwen-VL family, both within- and across-generation, monotonically reduces Language Shortcuts but amplifies Visual Sycophancy, showing that scale and newer post-training alone cannot resolve the grounding problem. Diagnostic scores further enable a training-free selective-prediction strategy yielding up to +9.5 percentage points accuracy at 50% coverage.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Selective Prediction on To See or To Please Diagnostic](<https://www.sota2.com/research/sota/selective-prediction-on-to-see-or-to-please-diagnostic.md>): Selective Prediction; To See or To Please Diagnostic; Baseline Accuracy: 73.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.18373>)

[PDF](<https://arxiv.org/pdf/2603.18373.pdf>)
