# Auditing LLM Benchmarks with Item Response Theory Source: [Auditing LLM Benchmarks with Item Response Theory]() Collection index: [llms.txt]() Data updated: 2026-06-02T05:00:05.014Z Authors: Sander Land, Daniel M. Bikel Published: 2026-05-28T19:38:44.000Z ## Abstract LLM benchmark labels are frozen at release and silently propagated into downstream benchmarks, errors and all. We introduce an Item Response Theory-based indicator that surfaces likely mislabels at 95% precision in the top 200 examples across seven preference and multiple-choice benchmarks using responses from 114 models, outperforming a supervised classifier. We trace these errors to mechanical labeling heuristics, upstream annotation mistakes inherited unchanged from source datasets, and fundamentally ambiguous items without a defensible single label. The same model fit reveals that reward models specialize in stylistic preference rather than factual knowledge, and identifies one frontier reward model that agrees with detected mislabels at 78% accuracy versus 38% for its peers, consistent with benchmark contamination or benchmark-specific over-optimization. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Mislabel detection on Weak Reference Labels Mislabels](): Mislabel Detection; Weak Reference Labels Mislabels; AP: 84.3. - [Mislabel detection on Weak Reference Labels Mislabels + subjective](): Mislabel Detection; Weak Reference Labels Mislabels + subjective; AP: 76.2. ## Original sources and implementation [Original paper]() [PDF]()