Towards Stability and Generalization Bounds in Decentralized Minibatch Stochastic Gradient Descent

• DOI: 10.1609/aaai.v38i14.29477
• Publication Date: 2024-03-25T11:32:05Z
• Authors: Jiahuan Wang; Hong Chen
• Keywords: Stochastic Gradient Descent; Descent (aeronautics)

Decentralized Stochastic Gradient Descent (D-SGD) represents an efficient communication approach tailored for mastering insights from vast, distributed datasets. Inspired by parallel optimization paradigms, the incorporation of minibatch serves to diminish variance, consequently expediting process. Nevertheless, as per our current understanding, existing literature has not thoroughly explored learning theory foundation Minibatch (DM-SGD). In this paper, we try address theoretical gap investigating generalization properties DM-SGD. We establish sharper bounds DM-SGD algorithm with replacement (without replacement) on (non)convex and (non)smooth cases. Moreover, results consistently recover Centralized (C-SGD). addition, derive analysis Zero-Order (ZO) version