# ART: Attention Run-time Termination for Efficient Large Language Model Decoding

Source: [ART: Attention Run-time Termination for Efficient Large Language Model Decoding](<https://www.sota2.com/research/paper/6a1e635edbd7150506f8502c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-18T05:00:06.089Z

Authors: Chen Qiu, Guozhong Li, Cristian McGee, Aritra Dutta, Panos Kalnis

Published: 2026-04-15T06:55:14.000Z

## Abstract

Long-context decoding in Large Language Models (LLMs) is constrained by the cost of accessing and processing the Key-Value (KV) cache. Despite evidence that attention outputs depend jointly on keys and values, most existing KV management methods rely on key-only pruning, since incorporating values incurs prohibitive overhead. In this paper, we propose Attention Run-time Termination (ART), a lightweight run-time mechanism that tracks accumulated attention outputs during kernel execution and terminates subsequent KV block accesses once further contributions become negligible. Rather than replacing KV selection, ART dynamically terminates redundant KV traversal on top of existing dense or sparse attention policies. We introduce a stability-based criterion that monitors both magnitude and directional changes of intermediate attention outputs and provideds a theoretical characterization of the resulting truncation error. Experiments on the LongBench and RULER Needle-in-a-Haystack tasks show that ART increases the generation throughput of existing KV-cache methods by up to 20%, without compromising the result quality.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-context language understanding on LongBench (Full Metric Suite)](<https://www.sota2.com/research/sota/long-context-language-understanding-on-longbench-full-metric-suite.md>): Long-context Language Understanding; LongBench; Narrative Score (Nrtv.): 24.57.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2606.00024>)

[PDF](<https://arxiv.org/pdf/2606.00024.pdf>)
