Comprehensive language-image pre-training for 3D medical image understanding
About
Vision-language pre-training, i.e., aligning images with paired text, is a powerful paradigm to create encoders that can be directly used for tasks such as classification, retrieval, and segmentation. In the 3D medical image domain, these capabilities allow vision-language encoders (VLEs) to support radiologists by retrieving patients with similar abnormalities, predicting likelihoods of abnormality, or, with downstream adaptation, generating radiological reports. While the methodology holds promise, data availability and domain-specific hurdles limit the capabilities of current 3D VLEs. In this paper, we overcome these challenges by injecting additional supervision via a report generation objective and combining vision-language with vision-only pre-training. This allows us to leverage both image-only and paired image-text 3D datasets, increasing the total amount of data to which our model is exposed. Through these additional objectives, paired with best practices of the 3D medical imaging domain, we develop the Comprehensive Language-Image Pre-training (COLIPRI) encoder family. Our COLIPRI encoders achieve state-of-the-art performance in report generation, semantic segmentation, classification probing, and zero-shot classification. The model is available at https://huggingface.co/microsoft/colipri.
Key insights
What problem does COLIPRI address?
3D medical vision-language models lack paired image-text data and face engineering hurdles from large CT volumes. COLIPRI combines contrastive alignment, report generation, masked image reconstruction, and a novel opposite-sentence loss to exploit both paired and image-only datasets.
How does it differ from prior work?
Unlike prior 3D VLEs using only contrastive learning, COLIPRI adds report generation and vision-only MAE objectives. On CT-RATE, COLIPRI-CRM improves AUPRC by 6 points and AUROC by 3.5 over Merlin, and retrieval R@5 from 2.9% to 35.1%.
What are the practical uses?
COLIPRI supports radiology workflows through zero-shot abnormality classification, patient retrieval by similar findings, and report generation. These are benchmark demonstrations on CT-RATE and RAD-ChestCT; no clinical deployment is documented.
What limitations remain?
The RRG objective yields only slight improvements, and a large gap between classification probes and zero-shot classification persists. Clinical metrics remain below practice thresholds; the authors note larger datasets and stratified evaluation are needed.