# MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution

Source: [MSPE: Multi-Scale Patch Embedding Prompts Vision Transformers to Any Resolution](<https://www.sota2.com/research/paper/68fa73671e209131cea5c41d>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T04:50:44.212Z

Authors: Wenzhuo Liu, Fei Zhu, Shijie Ma, Cheng-Lin Liu

Published: 2024-05-28T14:50:12.000Z

## Abstract

Although Vision Transformers (ViTs) have recently advanced computer vision tasks significantly, an important real-world problem was overlooked: adapting to variable input resolutions. Typically, images are resized to a fixed resolution, such as 224x224, for efficiency during training and inference. However, uniform input size conflicts with real-world scenarios where images naturally vary in resolution. Modifying the preset resolution of a model may severely degrade the performance. In this work, we propose to enhance the model adaptability to resolution variation by optimizing the patch embedding. The proposed method, called Multi-Scale Patch Embedding (MSPE), substitutes the standard patch embedding with multiple variable-sized patch kernels and selects the best parameters for different resolutions, eliminating the need to resize the original image. Our method does not require high-cost training or modifications to other parts, making it easy to apply to most ViT models. Experiments in image classification, segmentation, and detection tasks demonstrate the effectiveness of MSPE, yielding superior performance on low-resolution inputs and performing comparably on high-resolution inputs with existing methods.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K (Top-1 Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-top-1-accuracy.md>): Image Classification; ImageNet-1K; Top-1 Acc: 85.66.

- [Object Detection on COCO 2017](<https://www.sota2.com/research/sota/object-detection-on-coco-2017.md>): Object Detection; COCO 2017; AP (Box): 47.

- [Instance Segmentation on COCO 2017](<https://www.sota2.com/research/sota/instance-segmentation-on-coco-2017.md>): Instance Segmentation; COCO 2017; APm: 42.

- [Semantic Segmentation on ADE20K](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k.md>): Semantic segmentation; ADE20K; mIoU: 45.39.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2405.18240>)

[PDF](<https://arxiv.org/pdf/2405.18240.pdf>)
