InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions
About
Compared to the great progress of large-scale vision transformers (ViTs) in recent years, large-scale models based on convolutional neural networks (CNNs) are still in an early state. This work presents a new large-scale CNN-based foundation model, termed InternImage, which can obtain the gain from increasing parameters and training data like ViTs. Different from the recent CNNs that focus on large dense kernels, InternImage takes deformable convolution as the core operator, so that our model not only has the large effective receptive field required for downstream tasks such as detection and segmentation, but also has the adaptive spatial aggregation conditioned by input and task information. As a result, the proposed InternImage reduces the strict inductive bias of traditional CNNs and makes it possible to learn stronger and more robust patterns with large-scale parameters from massive data like ViTs. The effectiveness of our model is proven on challenging benchmarks including ImageNet, COCO, and ADE20K. It is worth mentioning that InternImage-H achieved a new record 65.4 mAP on COCO test-dev and 62.9 mIoU on ADE20K, outperforming current leading CNNs and ViTs. The code will be released at https://github.com/OpenGVLab/InternImage.
Key insights
What problem does InternImage solve?
CNNs lagged behind vision transformers for large-scale foundation models. InternImage uses deformable convolution v3 (DCNv3) to add adaptive spatial aggregation and long-range dependencies, letting CNNs scale to 1 billion parameters and 427 million images.
How does it differ from prior CNNs and ViTs?
Unlike large-kernel CNNs, InternImage uses a 3×3 dynamic sparse kernel. InternImage-H achieves 65.4 mAP on COCO test-dev, beating SwinV2-G by 2.3 points with 27% fewer parameters, evaluated across ImageNet, COCO, and ADE20K.
What are the practical uses of these findings?
InternImage provides strong backbones for classification, detection, and segmentation benchmarks. The paper documents benchmark results on tasks like autonomous driving datasets (BDD100K) but does not document any production deployment.
What limitations do the authors acknowledge?
The authors state latency remains an issue for DCN-based operators in high-speed downstream tasks. They also note large-scale CNNs are still in early development and hope InternImage serves as a starting point.