# Co-training $2^L$ Submodels for Visual Recognition

Source: [Co-training $2^L$ Submodels for Visual Recognition](<https://www.sota2.com/research/paper/68fa6e2d1e209131cea3e5dc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:50:46.791Z

Authors: Hugo Touvron, Matthieu Cord, Maxime Oquab, Piotr Bojanowski, Jakob Verbeek, Herv\\'e J\\'egou

Published: 2022-12-09T14:38:09.000Z

## Abstract

We introduce submodel co-training, a regularization method related to co-training, self-distillation and stochastic depth. Given a neural network to be trained, for each sample we implicitly instantiate two altered networks, \`\`submodels'', with stochastic depth: we activate only a subset of the layers. Each network serves as a soft teacher to the other, by providing a loss that complements the regular loss provided by the one-hot label. Our approach, dubbed cosub, uses a single set of weights, and does not involve a pre-trained external model or temporal averaging.   Experimentally, we show that submodel co-training is effective to train backbones for recognition tasks such as image classification and semantic segmentation. Our approach is compatible with multiple architectures, including RegNet, ViT, PiT, XCiT, Swin and ConvNext. Our training strategy improves their results in comparable settings. For instance, a ViT-B pretrained with cosub on ImageNet-21k obtains 87.4% top-1 acc. @448 on ImageNet-val.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet-1K (val) (Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-val-accuracy.md>): Image Classification; ImageNet-1k (val); Top-1 Accuracy: 88.

- [Image Classification on ImageNet V2](<https://www.sota2.com/research/sota/image-classification-on-imagenet-v2.md>): Image Classification; ImageNet V2; Top-1 Acc: 80.

- [Semantic Segmentation on ADE20K (val)](<https://www.sota2.com/research/sota/semantic-segmentation-on-ade20k-val.md>): Semantic segmentation; ADE20K (val); mIoU: 55.7.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2212.04884>)

[PDF](<https://arxiv.org/pdf/2212.04884.pdf>)