# Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run

Source: [Detectability in Diversity: Improved Canary Crafting for Privacy Auditing in One Run](<https://www.sota2.com/research/paper/6a167ad8dbd7150506f802ec>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-06-06T05:00:11.965Z

Authors: Mathieu Dagr\\'eou, Aur\\'elien Bellet

Published: 2026-05-26T17:06:14.000Z

## Abstract

Privacy auditing aims to empirically assess privacy leakage in machine learning models using membership inference attacks (MIAs), and to derive lower bounds on differential privacy (DP) parameters. Recent one-run auditing methods address the high cost of standard approaches by relying on a single training run with multiple "canary" points whose inclusion or exclusion must be detected by the auditor. In this work, we study the problem of efficiently crafting canaries for one-run privacy auditing. Motivated by recent theoretical insights suggesting that interference between canaries contributes to weaker leakage estimates compared to multi-run methods, we propose to optimize canaries to be both highly detectable and minimally interfering. Our approach combines a greedy initialization based on influence functions with a bilevel optimization procedure that maximizes distinguishability while promoting diversity in embedding space, enabling the use of computationally efficient bilevel algorithms. Experiments show that our method achieves stronger privacy leakage estimates at a lower computational cost than existing canary crafting approaches.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [One-run DP privacy auditing on CIFAR10 (test)](<https://www.sota2.com/research/sota/one-run-dp-privacy-auditing-on-cifar10-test.md>): One-run DP privacy auditing; CIFAR10 (test); Mean Score: 99.

- [Membership Inference Attack on CIFAR10 (TPR @ 0.05 FPR)](<https://www.sota2.com/research/sota/membership-inference-attack-on-cifar10-tpr-at-0-05-fpr.md>): Membership Inference Attack auditing; CIFAR10; Mean TPR @ 0.05 FPR: 25.

- [One-run privacy auditing on CIFAR10](<https://www.sota2.com/research/sota/one-run-privacy-auditing-on-cifar10.md>): One-run privacy auditing; CIFAR10; Mean TPR @ 0.05 FPR: 100.

- [Privacy Auditing on CIFAR10 (test)](<https://www.sota2.com/research/sota/privacy-auditing-on-cifar10-test.md>): Privacy Auditing; CIFAR10 (test); Mean: 8.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2605.27292>)

[PDF](<https://arxiv.org/pdf/2605.27292.pdf>)