# OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding

Source: [OmniV-Med: Scaling Medical Vision-Language Model for Universal Visual Understanding](<https://www.sota2.com/research/paper/68fa78191e209131cea76c2f>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T04:32:25.977Z

Authors: Songtao Jiang, Yuan Wang, Sibo Song, Yan Zhang, Zijie Meng, Bohan Lei, Jian Wu, Jimeng Sun, Zuozhu Liu

Published: 2025-04-20T17:53:56.000Z

## Abstract

The practical deployment of medical vision-language models (Med-VLMs) necessitates seamless integration of textual data with diverse visual modalities, including 2D/3D images and videos, yet existing models typically employ separate encoders for different modalities. To address this limitation, we present OmniV-Med, a unified framework for multimodal medical understanding. Our technical contributions are threefold: First, we construct OmniV-Med-Instruct, a comprehensive multimodal medical dataset containing 252K instructional samples spanning 14 medical image modalities and 11 clinical tasks. Second, we devise a rotary position-adaptive encoder that processes multi-resolution 2D/3D images and videos within a unified architecture, diverging from conventional modality-specific encoders. Third, we introduce a medical-aware token pruning mechanism that exploits spatial-temporal redundancy in volumetric data (e.g., consecutive CT slices) and medical videos, effectively reducing 60\\% of visual tokens without performance degradation. Empirical evaluations demonstrate that OmniV-Med-7B achieves state-of-the-art performance on 7 benchmarks spanning 2D/3D medical imaging and video understanding tasks. Notably, our lightweight variant (OmniV-Med-1.5B) attains comparable performance while requiring only 8 RTX3090 GPUs for training and supporting efficient long-video inference. Data, code and model will be released.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Observation on 3D-RAD](<https://www.sota2.com/research/sota/image-observation-on-3d-rad.md>): Image Observation; 3D-RAD; BLEU: 16.42.

- [Anomaly Detection on 3D-RAD](<https://www.sota2.com/research/sota/anomaly-detection-on-3d-rad.md>): Anomaly Detection; 3D-RAD; BLEU: 13.47.

- [Longitudinal Temporal Diagnosis on 3D-RAD](<https://www.sota2.com/research/sota/longitudinal-temporal-diagnosis-on-3d-rad.md>): Longitudinal Temporal Diagnosis; 3D-RAD; Accuracy: 24.23.

- [Existence Detection on 3D-RAD](<https://www.sota2.com/research/sota/existence-detection-on-3d-rad.md>): Existence Detection; 3D-RAD; Accuracy: 28.66.

- [Medical Measurement on 3D-RAD](<https://www.sota2.com/research/sota/medical-measurement-on-3d-rad.md>): Medical Measurement; 3D-RAD; BLEU: 2.52.

- [Static Temporal Diagnosis on 3D-RAD](<https://www.sota2.com/research/sota/static-temporal-diagnosis-on-3d-rad.md>): Static Temporal Diagnosis; 3D-RAD; Accuracy: 22.96.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2504.14692>)

[PDF](<https://arxiv.org/pdf/2504.14692.pdf>)