Fall of Empires: Breaking Byzantine-tolerant SGD by Inner Product Manipulation

• DOI: 10.48550/arxiv.1903.03936
• Publication Date: 2019-01-01
• Authors: Xie, Cong; Koyejo, Sanmi; Gupta, Indranil
• Keywords: FOS: Computer and information sciences; Computer Science - Machine Learning; Computer Science - Cryptography and Security; Computer Science - Distributed, Parallel, and Cluster Computing; Statistics - Machine Learning; 0202 electrical engineering, electronic engineering, information engineering; Machine Learning (stat.ML); 02 engineering and technology; Distributed, Parallel, and Cluster Computing (cs.DC); Cryptography and Security (cs.CR); Machine Learning (cs.LG)

Recently, new defense techniques have been developed to tolerate Byzantine failures for distributed machine learning. The Byzantine model captures workers that behave arbitrarily, including malicious and compromised workers. In this paper, we break two prevailing Byzantine-tolerant techniques. Specifically we show robust aggregation methods for synchronous SGD -- coordinate-wise median and Krum -- can be broken using new attack strategies based on inner product manipulation. We prove our results theoretically, as well as show empirical validation.