# Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy Source: [Scalable and Efficient Training of Large Convolutional Neural Networks with Differential Privacy]() Collection index: [llms.txt]() Data updated: 2026-01-02T13:11:49.500Z Authors: Zhiqi Bu, Jialin Mao, Shiyun Xu Published: 2022-05-21T22:01:12.000Z ## Abstract Large convolutional neural networks (CNN) can be difficult to train in the differentially private (DP) regime, since the optimization algorithms require a computationally expensive operation, known as the per-sample gradient clipping. We propose an efficient and scalable implementation of this clipping on convolutional layers, termed as the mixed ghost clipping, that significantly eases the private training in terms of both time and space complexities, without affecting the accuracy. The improvement in efficiency is rigorously studied through the first complexity analysis for the mixed ghost clipping and existing DP training algorithms. Extensive experiments on vision classification tasks, with large ResNet, VGG, and Vision Transformers, demonstrate that DP training with mixed ghost clipping adds $1\\sim 10\\%$ memory overhead and $\<2\\times$ slowdown to the standard non-private training. Specifically, when training VGG19 on CIFAR10, the mixed ghost clipping is $3\\times$ faster than state-of-the-art Opacus library with $18\\times$ larger maximum batch size. To emphasize the significance of efficient DP training on convolutional layers, we achieve 96.7\\% accuracy on CIFAR10 and 83.0\\% on CIFAR100 at $\\epsilon=1$ using BEiT, while the previous best results are 94.8\\% and 67.4\\%, respectively. We open-source a privacy engine (\\url{https://github.com/woodyx218/private\_vision}) that implements DP training of CNN with a few lines of code. ## Recorded benchmark results A research result applies to its stated dataset, split, metric and evaluation configuration. Compare like-for-like settings, preserve the metric's unit and direction, and distinguish paper-reported results from independent reproduction. The best result recorded here is not a claim of exhaustive, current global coverage. - [Image Classification on CIFAR-10 resized to 224 × 224 (test)](): Image Classification; CIFAR-10 resized to 224 × 224 (test); Accuracy: 97.4. - [Image Classification on CIFAR-100 resized to 224x224 (test)](): Image Classification; CIFAR-100 resized to 224 × 224 (test); Accuracy: 88.4. - [Image Classification on CIFAR-100 (test)](): Image Classification; CIFAR-100 (test); Accuracy: not recorded. - [Image Classification on CIFAR-10 (test)](): Image Classification; CIFAR-10 (test); Accuracy: not recorded. ## Original sources and implementation [Original paper]() [PDF]() [Implementation]()