# Benchmarking a Benchmark: How Reliable is MS-COCO?

Source: [Benchmarking a Benchmark: How Reliable is MS-COCO?](<https://www.sota2.com/research/paper/68fa71231e209131cea4f6cc>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:58:41.298Z

Authors: Eric Zimmermann, Justin Szeto, Jerome Pasquero, Frederic Ratle

Published: 2023-11-05T16:55:40.000Z

## Abstract

Benchmark datasets are used to profile and compare algorithms across a variety of tasks, ranging from image classification to segmentation, and also play a large role in image pretraining algorithms. Emphasis is placed on results with little regard to the actual content within the dataset. It is important to question what kind of information is being learned from these datasets and what are the nuances and biases within them. In the following work, Sama-COCO, a re-annotation of MS-COCO, is used to discover potential biases by leveraging a shape analysis pipeline. A model is trained and evaluated on both datasets to examine the impact of different annotation conditions. Results demonstrate that annotation styles are important and that annotation pipelines should closely consider the task of interest. The dataset is made publicly available at https://www.sama.com/sama-coco-dataset/ .

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2311.02709>)

[PDF](<https://arxiv.org/pdf/2311.02709.pdf>)