# Can We Infer Confidential Properties of Training Data from LLMs?

Source: [Can We Infer Confidential Properties of Training Data from LLMs?](<https://www.sota2.com/research/paper/68fa7be61e209131cea8b294>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-02-11T05:00:13.386Z

Authors: Pengrun Huang, Chhavi Yadav, Kamalika Chaudhuri, Ruihan Wu

Published: 2025-06-12T05:42:06.000Z

## Abstract

Large language models (LLMs) are increasingly fine-tuned on domain-specific datasets to support applications in fields such as healthcare, finance, and law. These fine-tuning datasets often have sensitive and confidential dataset-level properties -- such as patient demographics or disease prevalence -- that are not intended to be revealed. While prior work has studied property inference attacks on discriminative models (e.g., image classification models) and generative models (e.g., GANs for image data), it remains unclear if such attacks transfer to LLMs. In this work, we introduce PropInfer, a benchmark task for evaluating property inference in LLMs under two fine-tuning paradigms: question-answering and chat-completion. Built on the ChatDoctor dataset, our benchmark includes a range of property types and task configurations. We further propose two tailored attacks: a prompt-based generation attack and a shadow-model attack leveraging word frequency signals. Empirical evaluations across multiple pretrained LLMs show the success of our attacks, revealing a previously unrecognized vulnerability in LLMs.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [QA (MAE) on ChatDoctor](<https://www.sota2.com/research/sota/qa-mae-on-chatdoctor.md>): QA; ChatDoctor; MAE: 0.0396.

- [Property Inference Defense on ChatDoctor CC](<https://www.sota2.com/research/sota/property-inference-defense-on-chatdoctor-cc.md>): Property Inference Defense; ChatDoctor CC; Generation Attack MAE: 0.0919.

- [Property Inference Defense on ChatDoctor QA](<https://www.sota2.com/research/sota/property-inference-defense-on-chatdoctor-qa.md>): Property Inference Defense; ChatDoctor QA; Generation Attack MAE: 0.1301.

- [QA on MedCalc](<https://www.sota2.com/research/sota/qa-on-medcalc.md>): QA; MedCalc; MAE: 0.0653.

- [Property Inference Defense on MedCalc QA](<https://www.sota2.com/research/sota/property-inference-defense-on-medcalc-qa.md>): Property Inference Defense; MedCalc QA; Generation Attack MAE: 0.0653.

- [CC on ChatDoctor](<https://www.sota2.com/research/sota/cc-on-chatdoctor.md>): CC; ChatDoctor; MAE: 0.0719.

- [CC on MedCalc](<https://www.sota2.com/research/sota/cc-on-medcalc.md>): CC; MedCalc; MAE: 0.0253.

- [Property Inference Defense on MedCalc CC](<https://www.sota2.com/research/sota/property-inference-defense-on-medcalc-cc.md>): Property Inference Defense; MedCalc CC; Generation Attack MAE: 0.0264.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2506.10364>)

[PDF](<https://arxiv.org/pdf/2506.10364.pdf>)
