# Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Models Source: [Risk-aware Selective Prompting for Hallucination Mitigation in Large Vision-Language Models]() Collection index: [llms.txt]() Data updated: 2026-05-29T05:00:05.519Z Authors: Yuang Huang, Yafeng Zhang, Yu Zilan Published: 2026-05-27T08:14:59.000Z ## Abstract Prompt-based verification is widely used to mitigate hallucinations in large vision-language models (LVLMs), yet when it helps remains poorly understood. We systematically study verification prompting across two representative LVLM architectures and hallucination benchmarks, and find that it is a risk-bearing intervention: its corrections increase with input difficulty, while newly introduced errors persist across difficulty levels. As a result, always-on prompting helps on hard inputs but offers little benefit -- and can harm -- easier ones. Our analysis further shows that this behavior is associated with a conservative output shift. Verification prompts redistribute attention from visual tokens toward instruction tokens and induce a distinct middle-layer entropy pattern absent in a neutral-prompt control, suggesting instruction-conditioned attention redistribution rather than uniformly improved visual grounding. Motivated by this input-dependent risk, we propose Risk-aware Selective Prompting (RSP), a training-free approach that uses pre-generation uncertainty signals to trigger verification selectively. RSP mitigates the degradation of always-on prompting while preserving baseline performance, and reveals that effective selection signals vary across architectures. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Object Hallucination Detection on POPE random](): Object Hallucination Detection; POPE (Random); F1 Score: 89.1. - [Object Hallucination Detection on POPE popular](): Object Hallucination Detection; POPE Popular; F1 Score: 86.5. - [Object Hallucination on POPE Adversarial](): Object Hallucination; POPE Adversarial; Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]()