# Statistical Inference for Fairness Auditing

Source: [Statistical Inference for Fairness Auditing](<https://www.sota2.com/research/paper/68fa6f9e1e209131cea46a6c>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:23:24.960Z

Authors: John J. Cherian, Emmanuel J. Cand\\\`es

Published: 2023-05-05T17:54:22.000Z

## Abstract

Before deploying a black-box model in high-stakes problems, it is important to evaluate the model's performance on sensitive subpopulations. For example, in a recidivism prediction task, we may wish to identify demographic groups for which our prediction model has unacceptably high false positive rates or certify that no such groups exist. In this paper, we frame this task, often referred to as "fairness auditing," in terms of multiple hypothesis testing. We show how the bootstrap can be used to simultaneously bound performance disparities over a collection of groups with statistical guarantees. Our methods can be used to flag subpopulations affected by model underperformance, and certify subpopulations for which the model performs adequately. Crucially, our audit is model-agnostic and applicable to nearly any performance metric or group fairness criterion. Our methods also accommodate extremely rich -- even infinite -- collections of subpopulations. Further, we generalize beyond subpopulations by showing how to assess performance over certain distribution shifts. We test the proposed methods on benchmark datasets in predictive inference and algorithmic fairness and find that our audits can provide interpretable and trustworthy guarantees.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Confidence region coverage estimation on Model synthetic 5.1](<https://www.sota2.com/research/sota/confidence-region-coverage-estimation-on-model-synthetic-5-1.md>): Confidence region coverage estimation; Model synthetic 5.1; Coverage Probability: 93.3.

- [Confidence region coverage estimation on Model synthetic 5.2](<https://www.sota2.com/research/sota/confidence-region-coverage-estimation-on-model-synthetic-5-2.md>): Confidence region coverage estimation; Model synthetic 5.2; Coverage Probability: 92.5.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2305.03712>)

[PDF](<https://arxiv.org/pdf/2305.03712.pdf>)