# TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks

Source: [TriAxialKV: Toward Extreme Low-Precision KV-Cache Quantization for Agentic Inference Tasks](<https://www.sota2.com/research/paper/6a0bee91dbc836b87d82c3b3>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-20T05:00:13.583Z

Authors: Hanzhang Shen, Haoran Wu, Yiren Zhao, Robert Mullins

Published: 2026-05-16T21:58:28.000Z

## Abstract

Agentic workloads have emerged as a major workload for LLM inference. They differ significantly from chat-only workloads, requiring long-context processing, the ability to handle multimodal inputs, and structured multi-turn interactions with tool calling capabilities. As a result, their context exhibits structure that can carry different importance along three key axes: temporal recency to the current turn, modality such as text or image tokens, and semantic role such as user queries, tool calls, observations, or reasoning. These axes capture distinct token behaviors and lead to different sensitivities to KV-cache compression. However, existing KV-cache quantization methods are typically homogeneous or exploit only heterogeneity on a single dimension, such as temporal proximity or modality, overlooking the interactions among them. To this end, we introduce TriAxialKV, a novel mixed-precision KV-cache quantization scheme that assigns each token a triaxial tag, calibrates per-tag sensitivity, and allocates INT2/INT4 bitwidths under a fixed memory budget. We implement TriAxialKV as an end-to-end serving system, comprising calibration, mixed-precision quantization and memory management, and custom fused Triton decode kernels. When using Qwen3-VL-32B-Thinking as a computer-use agent operating the OSWorld, TriAxialKV matches the accuracy of SGLang with BF16 KV cache while supporting 4.5$\\times$ KV cache size and achieving 30% higher end-to-end throughput, when running on real GPU systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multimodal Task Accuracy on OSWorld](<https://www.sota2.com/research/sota/multimodal-task-accuracy-on-osworld.md>): Multimodal Task Accuracy; OSWorld; Multimodal Task Accuracy: 41.43.

- [Task Accuracy on OSWorld](<https://www.sota2.com/research/sota/task-accuracy-on-osworld.md>): Task accuracy; OSWorld; Task Accuracy: 41.43.

- [Function Calling on BFCL Memory](<https://www.sota2.com/research/sota/function-calling-on-bfcl-memory.md>): Function Calling; BFCL Memory; Task Accuracy: 25.11.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.17170>)

[PDF](<https://arxiv.org/pdf/2605.17170.pdf>)
