# Privacy Assessment on Reconstructed Images: Are Existing Evaluation Metrics Faithful to Human Perception? Source: [Privacy Assessment on Reconstructed Images: Are Existing Evaluation Metrics Faithful to Human Perception?]() Collection index: [llms.txt]() Data updated: 2026-01-04T04:37:02.542Z Authors: Xiaoxiao Sun, Nidham Gazagnadou, Vivek Sharma, Lingjuan Lyu, Hongdong Li, Liang Zheng Published: 2023-09-22T17:58:04.000Z ## Abstract Hand-crafted image quality metrics, such as PSNR and SSIM, are commonly used to evaluate model privacy risk under reconstruction attacks. Under these metrics, reconstructed images that are determined to resemble the original one generally indicate more privacy leakage. Images determined as overall dissimilar, on the other hand, indicate higher robustness against attack. However, there is no guarantee that these metrics well reflect human opinions, which, as a judgement for model privacy leakage, are more trustworthy. In this paper, we comprehensively study the faithfulness of these hand-crafted metrics to human perception of privacy information from the reconstructed images. On 5 datasets ranging from natural images, faces, to fine-grained classes, we use 4 existing attack methods to reconstruct images from many different classification models and, for each reconstructed image, we ask multiple human annotators to assess whether this image is recognizable. Our studies reveal that the hand-crafted metrics only have a weak correlation with the human evaluation of privacy leakage and that even these metrics themselves often contradict each other. These observations suggest risks of current metrics in the community. To address this potential risk, we propose a learning-based measure called SemSim to evaluate the Semantic Similarity between the original and reconstructed images. SemSim is trained with a standard triplet loss, using an original image as an anchor, one of its recognizable reconstructed images as a positive sample, and an unrecognizable one as a negative. By training on human annotations, SemSim exhibits a greater reflection of privacy leakage on the semantic level. We show that SemSim has a significantly higher correlation with human judgment compared with existing metrics. Moreover, this strong correlation generalizes to unseen datasets, models and attack methods. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Correlation of Privacy Leakage Assessment Metrics on CIFAR-100 (test)](): Metric correlation with human assessment of privacy leakage; CIFAR-100 (test); Spearman's rho: -0.8637. - [Metric correlation with human assessment of privacy leakage on Caltech-101 (test)](): Metric correlation with human assessment of privacy leakage; Caltech-101 (test); Spearman's rho: -0.8182. - [Metric correlation with human assessment of privacy leakage on Imagenette (test)](): Metric correlation with human assessment of privacy leakage; ImageNette (test); Spearman's Rho: -0.8257. - [Metric correlation with human assessment of privacy leakage on CelebA (test)](): Metric correlation with human assessment of privacy leakage; CelebA (test); Spearman's Rho: -0.8263. - [Metric correlation with human assessment of privacy leakage on Stanford Dogs (test)](): Metric correlation with human assessment of privacy leakage; Stanford Dogs (test); Spearman Correlation: -0.712. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()