# SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference

Source: [SparseVILA: Decoupling Visual Sparsity for Efficient VLM Inference](<https://www.sota2.com/research/paper/68fa7c5e1e209131cea8dc56>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-28T10:39:22.431Z

Authors: Samir Khaki, Junxian Guo, Jiaming Tang, Shang Yang, Yukang Chen, Konstantinos N. Plataniotis, Yao Lu, Song Han, Zhijian Liu

Published: 2025-10-20T17:35:47.000Z

## Abstract

Vision Language Models (VLMs) have rapidly advanced in integrating visual and textual reasoning, powering applications across high-resolution image understanding, long-video analysis, and multi-turn conversation. However, their scalability remains limited by the growing number of visual tokens that dominate inference latency. We present SparseVILA, a new paradigm for efficient VLM inference that decouples visual sparsity across the prefilling and decoding stages. SparseVILA distributes sparsity across stages by pruning redundant visual tokens during prefill and retrieving only query-relevant tokens during decoding. This decoupled design matches leading prefill pruning methods while preserving multi-turn fidelity by retaining most of the visual cache so that query-aware tokens can be retrieved at each conversation round. Built on an AWQ-optimized inference pipeline, SparseVILA achieves up to 4.0 times faster prefilling, 2.5 times faster decoding, and an overall 2.6 times end-to-end speedup on long-context video tasks -- while improving accuracy on document-understanding and reasoning tasks. By decoupling query-agnostic pruning and query-aware retrieval, SparseVILA establishes a new direction for efficient multimodal inference, offering a training-free, architecture-agnostic framework for accelerating large VLMs without sacrificing capability.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Long-Video Understanding on VideoMME (1-60m, w/o subtitles)](<https://www.sota2.com/research/sota/long-video-understanding-on-videomme-1-60m-w-o-subtitles.md>): Long Video Understanding; VideoMME 1~60m, w/o subtitles; Score: 66.3.

- [Long-Video Understanding on LongVideoBench 8s~60m](<https://www.sota2.com/research/sota/long-video-understanding-on-longvideobench-8s-60m.md>): Long Video Understanding; LongVideoBench 8s~60m; Score: 60.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2510.17777>)

[PDF](<https://arxiv.org/pdf/2510.17777.pdf>)