# When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models

Source: [When Sinks Help or Hurt: Unified Framework for Attention Sink in Large Vision-Language Models](<https://www.sota2.com/research/paper/69d48f997d0172140ec7434b>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-07-29T05:00:05.450Z

Authors: Jiho Choi, Jaemin Kim, Sanghwan Kim, Seunghoon Hong, Jin-Hwi Park

Published: 2026-04-01T09:59:09.000Z

## Abstract

Attention sinks are defined as tokens that attract disproportionate attention. While these have been studied in single modality transformers, their cross-modal impact in Large Vision-Language Models (LVLM) remains largely unexplored: are they redundant artifacts or essential global priors? This paper first categorizes visual sinks into two distinct categories: ViT-emerged sinks (V-sinks), which propagate from the vision encoder, and LLM-emerged sinks (L-sinks), which arise within deep LLM layers. Based on the new definition, our analysis reveals a fundamental performance trade-off: while sinks effectively encode global scene-level priors, their dominance can suppress the fine-grained visual evidence required for local perception. Furthermore, we identify specific functional layers where modulating these sinks most significantly impacts downstream performance. To leverage these insights, we propose Layer-wise Sink Gating (LSG), a lightweight, plug-and-play module that dynamically scales the attention contributions of V-sink and the rest visual tokens. LSG is trained via standard next-token prediction, requiring no task-specific supervision while keeping the LVLM backbone frozen. In most layers, LSG yields improvements on representative multimodal benchmarks, effectively balancing global reasoning and precise local evidence.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Spatial Reasoning on CVBench (Single Score Metric)](<https://www.sota2.com/research/sota/spatial-reasoning-on-cvbench-single-score-metric.md>): Spatial Reasoning; CVBench; Score: 7.3.

- [Multimodal Reasoning on MMStar (Score %)](<https://www.sota2.com/research/sota/multimodal-reasoning-on-mmstar-score-percent.md>): Multimodal Reasoning; MMStar; Score: 0.9.

- [OCR Evaluation on OCRBench](<https://www.sota2.com/research/sota/ocr-evaluation-on-ocrbench.md>): OCR Evaluation; OCRBench; Score: 3.

- [Mathematical Reasoning on MathVista](<https://www.sota2.com/research/sota/mathematical-reasoning-on-mathvista.md>): Mathematical Reasoning; MathVista; Score: 1.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.03316>)

[PDF](<https://arxiv.org/pdf/2604.03316.pdf>)