Double Momentum and Error Feedback for Clipping with Fast Rates and Differential Privacy

• DOI: 10.48550/arxiv.2502.11682
• Publication Date: 2025-02-17
• Authors: Rustem Islamov; Samuel Horváth; Aurélien Lucchi; Peter Richtárik; Eduard Gorbunov
• Keywords: Clipping (morphology); Differential Privacy; Momentum (technical analysis)

Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both at once: they either have optimal DP but rely on restrictive assumptions such as bounded gradients/bounded data heterogeneity, or ensure strong optimization performance lack guarantees. To address this gap the literature, we propose analyze new called Clip21-SGD2M based novel combination of clipping, heavy-ball momentum, Error Feedback. In particular, non-convex smooth distributed problems with clients having arbitrarily heterogeneous data, prove that has convergence rate also near (local-)DP neighborhood. Our numerical experiments logistic regression training neural networks highlight superiority over baselines terms given DP-budget.