# Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models

Source: [Distilling Human-Aligned Privacy Sensitivity Assessment from Large Language Models](<https://www.sota2.com/research/paper/69cca6c70636e5bf4afcbded>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-04-04T06:00:01.767Z

Authors: Gabriel Loiseau, Damien Sileo, Damien Riquet, Maxime Meyer, Marc Tommasi

Published: 2026-03-31T09:40:58.000Z

## Abstract

Accurate privacy evaluation of textual data remains a critical challenge in privacy-preserving natural language processing. Recent work has shown that large language models (LLMs) can serve as reliable privacy evaluators, achieving strong agreement with human judgments; however, their computational cost and impracticality for processing sensitive data at scale limit real-world deployment. We address this gap by distilling the privacy assessment capabilities of Mistral Large 3 (675B) into lightweight encoder models with as few as 150M parameters. Leveraging a large-scale dataset of privacy-annotated texts spanning 10 diverse domains, we train efficient classifiers that preserve strong agreement with human annotations while dramatically reducing computational requirements. We validate our approach on human-annotated test data and demonstrate its practical utility as an evaluation metric for de-identification systems.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Privacy Agreement on 250-text benchmark (test)](<https://www.sota2.com/research/sota/privacy-agreement-on-250-text-benchmark-test.md>): Privacy Agreement; 250-text benchmark (test); Krippendorff's Alpha: 0.737.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2603.29497>)

[PDF](<https://arxiv.org/pdf/2603.29497.pdf>)

[Implementation](<https://github.com/gabrielloiseau/privacy-distillation>)
