# OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance

Source: [OmniDrop: Layer-wise Token Pruning for Omni-modal LLMs via Query-Guidance](<https://www.sota2.com/research/paper/6a06a876dbc836b87d829f24>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-05-16T05:00:05.655Z

Authors: Yeo Jeong Park, Hyemi Jang, Minseo Choi, Jongsun Lee, Jooyoung Choi, Yongkweon Jeon

Published: 2026-05-14T06:54:37.000Z

## Abstract

Omni-modal large language models have demonstrated remarkable potential in holistic multimodal understanding; however, the token explosion caused by high-resolution audio and video inputs remains a critical bottleneck for real-time applications and long-form reasoning. Existing omni-modal token compression methods typically prune tokens at the input embedding level, relying on audio-video similarity or temporal co-occurrence as proxies for semantic relevance. In practice, such assumptions are often unreliable. To address this limitation, we propose OmniDrop, a training-free, layer-wise token pruning framework that progressively prunes audiovisual tokens within the LLM decoder layers rather than at the input-level, allowing early layers to preserve sufficient omni-modal information fusion before aggressively removing tokens in deeper layers. We further utilize text queries as guidance for modality-agnostic and task-adaptive token pruning. We also introduce a temporal diversity score that encourages balanced token survival to preserve global temporal context. Experimental results across various audiovisual benchmarks demonstrate that OmniDrop outperforms all baselines by up to 3.58 points while reducing prefill latency by up to 40% and memory usage by up to 14.7%.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Video Understanding on VideoMME (Performance, Latency, and Memory)](<https://www.sota2.com/research/sota/video-understanding-on-videomme-performance-latency-and-memory.md>): Video Understanding; VideoMME; VideoMME Performance: 66.52.

- [Multi-modal Understanding on WorldSense](<https://www.sota2.com/research/sota/multi-modal-understanding-on-worldsense.md>): Multi-modal Understanding; WorldSense; WorldSense Performance: 46.78.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.14458>)

[PDF](<https://arxiv.org/pdf/2605.14458.pdf>)
