IPCV: Information-Preserving Compression for MLLM Visual Encoders
About
Multimodal Large Language Models (MLLMs) deliver strong vision-language performance but at high computational cost, driven by numerous visual tokens processed by the Vision Transformer (ViT) encoder. Existing token pruning strategies are inadequate: LLM-stage token pruning overlooks the ViT's overhead, while conventional ViT token pruning, without language guidance, risks discarding textually critical visual cues and introduces feature distortions amplified by the ViT's bidirectional attention. To meet these challenges, we propose IPCV, a training-free, information-preserving compression framework for MLLM visual encoders. IPCV enables aggressive token pruning inside the ViT via Neighbor-Guided Reconstruction (NGR) that temporarily reconstructs pruned tokens to participate in attention with minimal overhead, then fully restores them before passing to the LLM. Besides, we introduce Attention Stabilization (AS) to further alleviate the negative influence from token pruning by approximating the K/V of pruned tokens. It can be directly applied to previous LLM-side token pruning methods to enhance their performance. Extensive experiments show that IPCV substantially reduces end-to-end computation and outperforms state-of-the-art training-free token compression methods across diverse image and video benchmarks. Our code is available at https://github.com/Perkzi/IPCV.
Key insights
How does IPCV reduce visual processing costs?
Vision-language models spend substantial computation processing image tokens. IPCV removes redundant tokens inside the vision encoder, then reconstructs them before the language model. Neighbor-guided reconstruction and attention stabilization help preserve visual information without retraining.
How much quality survives token compression?
In the paper's Qwen2-VL comparison at 50% token retention, IPCV retains 97.8% of the uncompressed model's average benchmark performance, versus 86.3% for ToMe. These are relative performance figures, not absolute task accuracies.
What is IPCV useful for in practice?
IPCV is a drop-in module to accelerate MLLM inference on existing architectures. The paper demonstrates benchmark efficiency on image and video tasks but documents no production deployment.
What remains to be improved?
The authors propose refining reconstruction and studying attention stabilization further to improve the balance between semantic preservation and computational efficiency. The reported results establish performance on the tested models and benchmarks.