# Object-level Self-Distillation for Vision Pretraining

Source: [Object-level Self-Distillation for Vision Pretraining](<https://www.sota2.com/research/paper/68fa792a1e209131cea7c5d9>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-27T06:53:53.053Z

Authors: \\c{C}a\\u{g}lar H{\\i}zl{\\i}, \\c{C}a\\u{g}atay Y{\\i}ld{\\i}z, Pekka Marttinen

Published: 2025-06-04T15:50:09.000Z

## Abstract

State-of-the-art vision pretraining methods rely on image-level self-distillation from object-centric datasets such as ImageNet, implicitly assuming each image contains a single object. This assumption does not always hold: many ImageNet images already contain multiple objects. Further, it limits scalability to scene-centric datasets that better mirror real-world complexity. We address these challenges by introducing Object-level Self-DIStillation (ODIS), a pretraining approach that shifts the self-distillation granularity from whole images to individual objects. Using object-aware cropping and masked attention, ODIS isolates object-specific regions, guiding the transformer toward semantically meaningful content and transforming a noisy, scene-level task into simpler object-level sub-tasks. We show that this approach improves visual representations both at the image and patch levels. Using masks at inference time, our method achieves an impressive $82.6\\%$ $k$-NN accuracy on ImageNet1k with ViT-Large.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.05409>)

[PDF](<https://arxiv.org/pdf/2506.05409.pdf>)