Are we done with ImageNet?
About
Yes, and no. We ask whether recent progress on the ImageNet classification benchmark continues to represent meaningful generalization, or whether the community has started to overfit to the idiosyncrasies of its labeling procedure. We therefore develop a significantly more robust procedure for collecting human annotations of the ImageNet validation set. Using these new labels, we reassess the accuracy of recently proposed ImageNet classifiers, and find their gains to be substantially smaller than those reported on the original labels. Furthermore, we find the original ImageNet labels to no longer be the best predictors of this independently-collected set, indicating that their usefulness in evaluating vision models may be nearing an end. Nevertheless, we find our annotation procedure to have largely remedied the errors in the original labels, reinforcing ImageNet as a powerful benchmark for future research in visual recognition.
Key insights
What problem does this paper address and how?
It asks whether ImageNet classification gains reflect real generalization or overfitting to label quirks. The authors collect new multi-label annotations (ReaL) using model proposals and human raters, then re-evaluate recent classifiers.
How does this differ from prior work?
Unlike prior replications collecting new images, this work re-annotates existing validation labels. It shows recent model gains shrink on ReaL labels; the regression slope drops from 0.86 to 0.51 across 19 models.
What are the practical uses of these findings?
The released ReaL labels enable more accurate benchmarking of vision models. The authors also demonstrate sigmoid loss and cleaned training data yield up to 2.5% ImageNet accuracy gains for ResNet-50 at 900 epochs.
What limitations and future work do the authors note?
Fine-grained animal classes remain hard for non-expert raters, requiring expert annotation. The authors note ReaL accuracy can be made more stringent if models approach near-perfect performance.