# Are we done with ImageNet?

Source: [Are we done with ImageNet?](<https://www.sota2.com/research/paper/68fa67ba1e209131cea18229>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-25T22:21:05.859Z

Authors: Lucas Beyer, Olivier J. H\\'enaff, Alexander Kolesnikov, Xiaohua Zhai, A\\"aron van den Oord

Published: 2020-06-12T13:17:25.000Z

## Abstract

Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Multi-label Image Classification on Shankar et al.](<https://www.sota2.com/research/sota/multi-label-image-classification-on-shankar-et-al.md>): Multi-label Image Classification; Shankar et al.; Top-1 Accuracy: 85.2.

- [Multi-label Image Classification on ReaL](<https://www.sota2.com/research/sota/multi-label-image-classification-on-real.md>): Multi-label Image Classification; ReaL; Top-1 Acc: 83.6.

- [Image Classification on WebVision-1000 (val)](<https://www.sota2.com/research/sota/image-classification-on-webvision-1000-val.md>): Image Classification; WebVision-1000 (val); Top-1 Acc: 72.1.

- [Image Classification on ImageNet V2](<https://www.sota2.com/research/sota/image-classification-on-imagenet-v2.md>): Image Classification; ImageNet V2; Top-1 Acc: 79.1.

- [Image Classification on ImageNet (val)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-val.md>): Image Classification; ImageNet (val); Top-1 Acc: 78.1.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2006.07159>)

[PDF](<https://arxiv.org/pdf/2006.07159.pdf>)