# UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks

Source: [UniBoost: Unsupervised Unimodal Pre-training for Boosting Zero-shot Vision-Language Tasks](<https://www.sota2.com/research/paper/68fa6f9e1e209131cea46a98>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:24:25.039Z

Authors: Yanan Sun, Zihan Zhong, Qi Fan, Chi-Keung Tang, Yu-Wing Tai

Published: 2023-06-07T18:26:22.000Z

## Abstract

Large-scale joint training of multimodal models, e.g., CLIP, have demonstrated great performance in many vision-language tasks. However, image-text pairs for pre-training are restricted to the intersection of images and texts, limiting their ability to cover a large distribution of real-world data, where noise can also be introduced as misaligned pairs during pre-processing. Conversely, unimodal models trained on text or image data alone through unsupervised techniques can achieve broader coverage of diverse real-world data and are not constrained by the requirement of simultaneous presence of image and text. In this paper, we demonstrate that using large-scale unsupervised unimodal models as pre-training can enhance the zero-shot performance of image-text pair models. Our thorough studies validate that models pre-trained as such can learn rich representations of both modalities, improving their ability to understand how images and text relate to each other. Our experiments show that unimodal pre-training outperforms state-of-the-art CLIP-based models by 6.5% (52.3% $\\rightarrow$ 58.8%) on PASCAL-5$^i$ and 6.2% (27.2% $\\rightarrow$ 33.4%) on COCO-20$^i$ semantic segmentation under zero-shot setting respectively. By learning representations of both modalities, unimodal pre-training offers broader coverage, reduced misalignment errors, and the ability to capture more complex features and patterns in the real-world data resulting in better performance especially for zero-shot vision-language tasks.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Semantic Segmentation on PASCAL-5i (5-Fold mIoU Evaluation)](<https://www.sota2.com/research/sota/semantic-segmentation-on-pascal-5i-5-fold-miou-evaluation.md>): Semantic segmentation; PASCAL-5i; Mean mIoU: 56.6.

- [Semantic Segmentation on COCO 20i (4-Fold CV mIoU)](<https://www.sota2.com/research/sota/semantic-segmentation-on-coco-20i-4-fold-cv-miou.md>): Semantic segmentation; COCO-20i; mIoU (Mean): 32.8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2306.04715>)

[PDF](<https://arxiv.org/pdf/2306.04715.pdf>)
