# Analyzing Leakage of Personally Identifiable Information in Language Models

Source: [Analyzing Leakage of Personally Identifiable Information in Language Models](<https://www.sota2.com/research/paper/68fa6f291e209131cea442c4>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T01:14:37.532Z

Authors: Nils Lukas, Ahmed Salem, Robert Sim, Shruti Tople, Lukas Wutschitz, Santiago Zanella-B\\'eguelin

Published: 2023-02-01T16:04:48.000Z

## Abstract

Language Models (LMs) have been shown to leak information about training data through sentence-level membership inference and reconstruction attacks. Understanding the risk of LMs leaking Personally Identifiable Information (PII) has received less attention, which can be attributed to the false assumption that dataset curation techniques such as scrubbing are sufficient to prevent PII leakage. Scrubbing techniques reduce but do not prevent the risk of PII leakage: in practice scrubbing is imperfect and must balance the trade-off between minimizing disclosure and preserving the utility of the dataset. On the other hand, it is unclear to which extent algorithmic defenses such as differential privacy, designed to guarantee sentence- or user-level privacy, prevent PII disclosure. In this work, we introduce rigorous game-based definitions for three types of PII leakage via black-box extraction, inference, and reconstruction attacks with only API access to an LM. We empirically evaluate the attacks against GPT-2 models fine-tuned with and without defenses in three domains: case law, health care, and e-mails. Our main contributions are (i) novel attacks that can extract up to 10$\\times$ more PII sequences than existing attacks, (ii) showing that sentence-level differential privacy reduces the risk of PII disclosure but still leaks about 3% of PII sequences, and (iii) a subtle connection between record-level membership inference and PII reconstruction. Code to reproduce all experiments in the paper is available at https://github.com/microsoft/analysing\_pii\_leakage.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [PII Extraction (Email Addresses) on TREC (train)](<https://www.sota2.com/research/sota/pii-extraction-email-addresses-on-trec-train.md>): PII Extraction (Email Addresses); TREC (train); Unique PII Count: 844.

- [PII Extraction (Email Addresses) on Enron (train)](<https://www.sota2.com/research/sota/pii-extraction-email-addresses-on-enron-train.md>): PII Extraction (Email Addresses); Enron (train); Unique PII Count: 5100.

- [PII Extraction (Phone Numbers) on TREC (train)](<https://www.sota2.com/research/sota/pii-extraction-phone-numbers-on-trec-train.md>): PII Extraction (Phone Numbers); TREC (train); Unique PII Count: 147.

- [PII Extraction (Phone Numbers) on Enron (train)](<https://www.sota2.com/research/sota/pii-extraction-phone-numbers-on-enron-train.md>): PII Extraction (Phone Numbers); Enron (train); Count of Unique Extracted PII: 2329.

- [PII Extraction (Personal Names) on Enron (train)](<https://www.sota2.com/research/sota/pii-extraction-personal-names-on-enron-train.md>): PII Extraction (Personal Names); Enron (train); Unique PII Count: 27142.

- [PII Extraction (Personal Names) on TREC (train)](<https://www.sota2.com/research/sota/pii-extraction-personal-names-on-trec-train.md>): PII Extraction (Personal Names); TREC (train); Unique PII Items Extracted: 4005.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2302.00539>)

[PDF](<https://arxiv.org/pdf/2302.00539.pdf>)
