# TAN Without a Burn: Scaling Laws of DP-SGD

Source: [TAN Without a Burn: Scaling Laws of DP-SGD](<https://www.sota2.com/research/paper/68fa6f6a1e209131cea45711>)

Collection index: [llms.txt](<https://www.sota2.com/research/llms.txt>)

Data updated: 2026-01-04T03:43:07.661Z

Authors: Tom Sander, Pierre Stock, Alexandre Sablayrolles

Published: 2022-10-07T08:44:35.000Z

## Abstract

Differentially Private methods for training Deep Neural Networks (DNNs) have progressed recently, in particular with the use of massive batches and aggregated data augmentations for a large number of training steps. These techniques require much more computing resources than their non-private counterparts, shifting the traditional privacy-accuracy trade-off to a privacy-accuracy-compute trade-off and making hyper-parameter search virtually impossible for realistic scenarios. In this work, we decouple privacy analysis and experimental behavior of noisy training to explore the trade-off with minimal computational requirements. We first use the tools of R\\'enyi Differential Privacy (RDP) to highlight that the privacy budget, when not overcharged, only depends on the total amount of noise (TAN) injected throughout training. We then derive scaling laws for training models with DP-SGD to optimize hyper-parameters with more than a $100\\times$ reduction in computational budget. We apply the proposed method on CIFAR-10 and ImageNet and, in particular, strongly improve the state-of-the-art on ImageNet with a +9 points gain in top-1 accuracy for a privacy budget epsilon=8.

## Recorded benchmark results

A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage.

- [Image Classification on ImageNet (test) (Standard Accuracy)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-test-standard-accuracy.md>): Image Classification; ImageNet (test); Top-1 Accuracy: 39.2.

- [Image Classification on ImageNet 1k (test)](<https://www.sota2.com/research/sota/image-classification-on-imagenet-1k-test.md>): Image Classification; ImageNet 1k (test); Top-1 Accuracy: 39.2.

## Original sources and implementation

[Original paper](<https://arxiv.org/abs/2210.03403>)

[PDF](<https://arxiv.org/pdf/2210.03403.pdf>)

[Implementation](<https://github.com/facebookresearch/tan>)