Tempered Sigmoid Activations for Deep Learning with Differential Privacy

• DOI: 10.1609/aaai.v35i10.17123
• Publication Date: 2022-09-08T19:18:09Z
• Authors: Nicolas Papernot; Abhradeep Thakurta; Shuang Song; Steve Chien; Úlfar Erlingsson
• Keywords: Differential Privacy; MNIST database; Sigmoid function

Because learning sometimes involves sensitive data, machine algorithms have been extended to offer differential privacy for training data. In practice, this has mostly an afterthought, with privacy-preserving models obtained by re-running a different optimizer, but using the model architectures that already performed well in non-privacy-preserving setting. This approach leads less than ideal privacy/utility tradeoffs, as we show here. To improve these prior work introduces variants of weaken guarantee proved increase utility. We is not necessary and instead propose utility be improved choosing activation functions designed explicitly training. A crucial operation differentially private SGD gradient clipping, which along modifying optimization path (at times resulting not-optimizing single objective function), may also introduce both significant bias variance process. empirically identify exploding gradients arising from ReLU one main sources this. demonstrate analytically experimentally how general family bounded functions, tempered sigmoids, consistently outperform currently established choice: unbounded like ReLU. Using paradigm, achieve new state-of-the-art accuracy on MNIST, FashionMNIST, CIFAR10 without any modification procedure fundamentals or analysis. While changes make are simple retrospect, simplicity our facilitates its implementation adoption meaningfully while still providing strong guarantees original framework privacy.