# How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models Source: [How Faithful is your Synthetic Data? Sample-level Metrics for Evaluating and Auditing Generative Models]() Collection index: [llms.txt]() Data updated: 2025-10-26T00:23:36.215Z Authors: Ahmed M. Alaa, Boris van Breugel, Evgeny Saveliev, Mihaela van der Schaar Published: 2021-02-17T18:25:30.000Z ## Abstract Devising domain- and model-agnostic evaluation metrics for generative models is an important and as yet unresolved problem. Most existing metrics, which were tailored solely to the image synthesis setup, exhibit a limited capacity for diagnosing the different modes of failure of generative models across broader application domains. In this paper, we introduce a 3-dimensional evaluation metric, ($\\alpha$-Precision, $\\beta$-Recall, Authenticity), that characterizes the fidelity, diversity and generalization performance of any generative model in a domain-agnostic fashion. Our metric unifies statistical divergence measures with precision-recall analysis, enabling sample- and distribution-level diagnoses of model fidelity and diversity. We introduce generalization as an additional, independent dimension (to the fidelity-diversity trade-off) that quantifies the extent to which a model copies training data -- a crucial performance indicator when modeling sensitive data with requirements on privacy. The three metric components correspond to (interpretable) probabilistic quantities, and are estimated via sample-level binary classification. The sample-level nature of our metric inspires a novel use case which we call model auditing, wherein we judge the quality of individual samples generated by a (black-box) model, discarding low-quality samples and hence improving the overall model performance in a post-hoc manner. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Dataset Quality Metric Correlation on Synthetic Data Quality YOLO11n mAP50](): Dataset Quality Metric Correlation; Synthetic Data Quality YOLO11n mAP50 (evaluation); PCC: 0.3482. ## Original sources and implementation [Original paper]() [PDF]()