SAGE: Sink-Aware Grounded Decoding for Multimodal Hallucination Mitigation
About
Large vision-language models (VLMs) frequently suffer from hallucinations, generating content that is inconsistent with visual inputs. Existing methods typically address this problem through post-hoc filtering, additional training objectives, or external verification, but they do not intervene during the decoding process when hallucinations arise. In this work, we introduce SAGE, a Sink-Aware Grounded Decoding framework that mitigates hallucinations by dynamically modulating self-attention during generation. Hallucinations are strongly correlated with attention sink tokens - punctuation or function tokens that accumulate disproportionate attention despite carrying limited semantic content. SAGE leverages these tokens as anchors to monitor grounding reliability in real time. At each sink trigger, the method extracts semantic concepts from the generated sequence, estimates their visual grounding using both self-attention maps and gradient-based attribution, and measures their spatial agreement. Based on this signal, self-attention distributions are adaptively sharpened or broadened to reinforce grounded regions or suppress unreliable ones. Extensive experiments across diverse hallucination benchmarks demonstrate that SAGE consistently outperforms existing decoding strategies, achieving substantial reductions in hallucination while preserving descriptive coverage, without requiring model retraining or architectural modifications. Our method achieves an average relative improvement of 10.65% on MSCOCO and 7.19% on AMBER across diverse VLM architectures, demonstrating consistent gains in hallucination mitigation.
Key insights
What problem does SAGE address and how?
VLMs hallucinate by over-attending to attention sink tokens (punctuation/function tokens) instead of visual evidence. SAGE treats sink tokens as grounding checkpoints, comparing self-attention maps with Grad-CAM attribution to modulate attention during decoding.
How does SAGE differ from prior decoding methods?
Unlike prior methods that suppress sinks or use contrastive decoding, SAGE reinterprets sinks as verification triggers and combines attention with gradient-based attribution. It achieves 10.65% average relative improvement on MSCOCO CHAIR across five VLMs versus six baselines.
What are the practical benefits of SAGE?
SAGE offers an inference-time way to reduce object hallucinations without retraining. The paper reports 10.8% average runtime overhead on one A100 GPU in its evaluation, illustrating a measured accuracy-cost trade-off rather than a documented production deployment.
What limitations do the authors acknowledge?
SAGE relies on sink tokens occurring during generation, making it ineffective for short-answer tasks like VQA or classification. Both attention and Grad-CAM signals can be simultaneously noisy, and concept extraction may fail on ambiguous text. Runtime overhead grows with long sequences.