# LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer

Source: [LLaVA-UHD v2: an MLLM Integrating High-Resolution Semantic Pyramid via Hierarchical Window Transformer](<https://www.sota2.com/research/paper/68fa77831e209131cea73bba>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:20:40.032Z

Authors: Yipeng Zhang, Yifan Liu, Zonghao Guo, Yidan Zhang, Xuesong Yang, Xiaoying Zhang, Chi Chen, Jun Song, Bo Zheng, Yuan Yao, Zhiyuan Liu, Tat-Seng Chua, Maosong Sun

Published: 2024-12-18T14:07:46.000Z

## Abstract

Vision transformers (ViTs) are widely employed in multimodal large language models (MLLMs) for visual encoding. However, they exhibit inferior performance on tasks regarding fine-grained visual perception. We attribute this to the limitations of ViTs in capturing diverse multi-modal visual levels, such as low-level details. To address this issue, we present LLaVA-UHD v2, an MLLM with advanced perception abilities by introducing a well-designed vision-language projector, the Hierarchical window (Hiwin) transformer. Hiwin transformer enhances MLLM's ability to capture diverse multi-modal visual granularities, by incorporating our constructed high-resolution semantic pyramid. Specifically, Hiwin transformer comprises two key modules: (i) a visual detail injection module, which progressively injects low-level visual details into high-level language-aligned semantics features, thereby forming an inverse semantic pyramid (ISP), and (ii) a hierarchical window attention module, which leverages cross-scale windows to condense multi-level semantics from the ISP. Extensive experiments show that LLaVA-UHD v2 outperforms compared MLLMs on a wide range of benchmarks. Notably, our design achieves an average boost of 3.7% across 14 benchmarks compared with the baseline method, 9.3% on DocVQA for instance. All the data and code will be publicly available to facilitate future research.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Visual Question Answering on GQA](<https://www.sota2.com/research/sota/visual-question-answering-on-gqa.md>): Visual Question Answering; GQA; Accuracy: 71.3.

- [Multimodal Understanding on MMBench (test)](<https://www.sota2.com/research/sota/multimodal-understanding-on-mmbench-test.md>): Multimodal Understanding; MMBench (test); Accuracy: 68.2.

- [Multimodal Understanding on SEED-Bench](<https://www.sota2.com/research/sota/multimodal-understanding-on-seed-bench.md>): Multimodal Understanding; SEED-Bench; Accuracy: 70.

- [Visual Question Answering on VQAv2](<https://www.sota2.com/research/sota/visual-question-answering-on-vqav2.md>): Visual Question Answering; VQAv2; Accuracy: 65.4.

- [Science Question Answering on ScienceQA SQA-IMG (test)](<https://www.sota2.com/research/sota/science-question-answering-on-scienceqa-sqa-img-test.md>): Science Question Answering; ScienceQA SQA-IMG (test); SQA-IMG Accuracy: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2412.13871>)

[PDF](<https://arxiv.org/pdf/2412.13871.pdf>)
