# MILAN: Masked Image Pretraining on Language Assisted Representation

Source: [MILAN: Masked Image Pretraining on Language Assisted Representation](<https://www.sota2.com/research/paper/68fa6e441e209131cea3ed9d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T00:54:03.791Z

Authors: Zejiang Hou, Fei Sun, Yen-Kuang Chen, Yuan Xie, Sun-Yuan Kung

Published: 2022-08-11T21:58:36.000Z

## Abstract

Self-attention based transformer models have been dominating many computer vision tasks in the past few years. Their superb model qualities heavily depend on the excessively large labeled image datasets. In order to reduce the reliance on large labeled datasets, reconstruction based masked autoencoders are gaining popularity, which learn high quality transferable representations from unlabeled images. For the same purpose, recent weakly supervised image pretraining methods explore language supervision from text captions accompanying the images. In this work, we propose masked image pretraining on language assisted representation, dubbed as MILAN. Instead of predicting raw pixels or low level features, our pretraining objective is to reconstruct the image features with substantial semantic signals that are obtained using caption supervision. Moreover, to accommodate our reconstruction target, we propose a more effective prompting decoder architecture and a semantic aware mask sampling mechanism, which further advance the transfer performance of the pretrained model. Experimental results demonstrate that MILAN delivers higher accuracy than the previous works. When the masked autoencoder is pretrained and finetuned on ImageNet-1K dataset with an input resolution of 224x224, MILAN achieves a top-1 accuracy of 85.4% on ViT-Base, surpassing previous state-of-the-arts by 1%. In the downstream semantic segmentation task, MILAN achieves 52.7 mIoU using ViT-Base on ADE20K dataset, outperforming previous masked pretraining results by 4 points.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K (Fine-tuning)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-fine-tuning.md>): Image Classification; ImageNet-1K (fine-tuning); Accuracy (FT): 85.4.

- [Semantic Segmentation on SODA-IR (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-soda-ir-test.md>): Semantic segmentation; SODA-IR (test); mIoU: 68.93.

- [Semantic Segmentation on MFNet-IR (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-mfnet-ir-val.md>): Semantic segmentation; MFNet-IR (val); mIoU: 45.08.

- [Semantic Segmentation on MFNet-IR (test)](<https://www.sota2.com/research/sota/semantic-segmentation-on-mfnet-ir-test.md>): Semantic segmentation; MFNet-IR (test); mIoU: 47.11.

- [Semantic Segmentation on ADE20K (mIoU)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-miou.md>): Semantic segmentation; ADE20K; mIoU: 52.7.

- [Semantic Segmentation on ADE20K](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k.md>): Semantic segmentation; ADE20K; mIoU: 52.7.

- [Semantic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-val.md>): Semantic segmentation; ADE20K (val); mIoU: 52.7.

- [Image Classification on ImageNet-1K 1.0 (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-1-0-val.md>): Image Classification; ImageNet-1K 1.0 (val); Top-1 Accuracy: 78.9.

- [Instance Segmentation on COCO 2017 (val)](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017-val.md>): Instance Segmentation; COCO 2017 (val); APm: not recorded.

- [Object Detection on MS-COCO 2017 (val)](<https://www.sota2.com/research/sota/object-detection-on-ms-coco-2017-val.md>): Object Detection; MS-COCO 2017 (val); mAP: not recorded.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2208.06049>)

[PDF](<https://arxiv.org/pdf/2208.06049.pdf>)
