# SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models

Source: [SinkRouter: Sink-Aware Routing for Efficient Long-Context Decoding in Large Language and Multimodal Models](<https://www.sota2.com/research/paper/69e704767d0172140ec7ec01>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-22T05:21:28.943Z

Authors: Junnan Liu, Xinyan Liu, Peifeng Gao, Zhaobo Qi, Beichen Zhang, Weigang Zhang, Antoni Bert Chen

Published: 2026-04-18T07:23:22.000Z

## Abstract

In long-context decoding for LLMs and LMMs, attention becomes increasingly memory-bound because each decoding step must load a large amount of KV-cache data from GPU memory. Existing acceleration strategies often trade efficiency for accuracy by relying on heuristic pruning that may discard useful information. At a deeper level, they also tend to indiscriminately preserve all high-scoring tokens, treat early tokens as indispensable anchors, or rely on heuristic head routing, reflecting an insufficient mechanistic understanding of the attention sink phenomenon. In this paper, we show that the attention sink phenomenon corresponds to a stable, reachable, and error-controllable fixed point constructed during training. Based on this insight, we propose SinkRouter, a training-free selective routing framework that detects the sink signal and skips computations that would otherwise produce near-zero output. To translate this mechanism into real-world acceleration, we develop a hardware-aware Triton kernel with block-level branching and Split-K parallelism. We conduct extensive evaluations on a diverse suite of long-context benchmarks, including LongBench, InfiniteBench, CVBench, MileBench, and MMVP, using both text-only and multimodal backbones such as Llama-3.1-8B, Llama-3.1-70B, Yi-9B-200K, LLaVA-1.5-7B, and LLaVA-1.5-13B. Across these settings, SinkRouter consistently improves decoding efficiency while maintaining competitive accuracy, and reaches 2.03x speedup with a 512K context.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-context understanding on InfiniteBench](<https://www.sota2.com/research/sota/long-context-understanding-on-infinitebench.md>): Long-context Understanding; InfiniteBench; Math Score (F): 0.5.

- [Multimodal Long-Context Understanding on MileBench](<https://www.sota2.com/research/sota/multimodal-long-context-understanding-on-milebench.md>): MultiModal Long-Context Understanding; MileBench; Object Existence: 51.5.

- [Multimodal Visual Perception on MMVP 1.0 (test)](<https://www.sota2.com/research/sota/multimodal-visual-perception-on-mmvp-1-0-test.md>): Multimodal Visual Perception; MMVP 1.0 (test); Accuracy: 60.73.

- [Visual Perception on CV-Bench 2D 1.0 (test)](<https://www.sota2.com/research/sota/visual-perception-on-cv-bench-2d-1-0-test.md>): Visual Perception; CV-Bench 2D 1.0 (test); Accuracy: 58.97.

- [Visual Perception on CV-Bench 3D 1.0 (test)](<https://www.sota2.com/research/sota/visual-perception-on-cv-bench-3d-1-0-test.md>): Visual Perception; CV-Bench 3D 1.0 (test); Accuracy: 61.92.

- [Long-context Language Understanding on LongBench (Specific Metric Subset including NQA and Delta Avg)](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-specific-metric-subset-including-nqa-and-delta-avg.md>): Long-context Language Understanding; LongBench; NQA: 34.43.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2604.16883>)

[PDF](<https://arxiv.org/pdf/2604.16883.pdf>)