# Pre-training Differentially Private Models with Limited Public Data

Source: [Pre-training Differentially Private Models with Limited Public Data](<https://www.sota2.com/research/paper/68fa756a1e209131cea67b7a>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2025-10-26T03:33:53.801Z

Authors: Zhiqi Bu, Xinwei Zhang, Mingyi Hong, Sheng Zha, George Karypis

Published: 2024-02-28T23:26:27.000Z

## Abstract

The superior performance of large foundation models relies on the use of massive amounts of high-quality data, which often contain sensitive, private and copyrighted material that requires formal protection. While differential privacy (DP) is a prominent method to gauge the degree of security provided to the models, its application is commonly limited to the model fine-tuning stage, due to the performance degradation when applying DP during the pre-training stage. Consequently, DP is yet not capable of protecting a substantial portion of the data used during the initial pre-training process.   In this work, we first provide a theoretical understanding of the efficacy of DP training by analyzing the per-iteration loss improvement. We make a key observation that DP optimizers' performance degradation can be significantly mitigated by the use of limited public data, which leads to a novel DP continual pre-training strategy. Empirically, using only 10\\% of public data, our strategy can achieve DP accuracy of 41.5\\% on ImageNet-21k (with $\\epsilon=8$), as well as non-DP accuracy of 55.7\\% and and 60.0\\% on downstream tasks Places365 and iNaturalist-2021, respectively, on par with state-of-the-art standard pre-training and substantially outperforming existing DP pre-trained models. Our DP pre-trained models are released in fastDP library (https://github.com/awslabs/fast-differential-privacy/releases/tag/v2.1)

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2402.18752>)

[PDF](<https://arxiv.org/pdf/2402.18752.pdf>)
