LEAF: A Benchmark for Federated Settings

• DOI: 10.48550/arxiv.1812.01097
• Publication Date: 2018-01-01
• Authors: Sebastian Caldas; Sai Meher Karthik...; Peter Wu; Tian Li; Jakub Konečný; H. Brendan McMahan; Virginia Smith; Ameet Talwalkar
• Keywords: Benchmarking; Benchmark (surveying); Implementation; Federated Learning

Modern federated networks, such as those comprised of wearable devices, mobile phones, or autonomous vehicles, generate massive amounts data each day. This wealth can help to learn models that improve the user experience on device. However, scale and heterogeneity presents new challenges in research areas learning, meta-learning, multi-task learning. As machine learning community begins tackle these challenges, we are at a critical time ensure developments made grounded with realistic benchmarks. To this end, propose LEAF, modular benchmarking framework for settings. LEAF includes suite open-source datasets, rigorous evaluation framework, set reference implementations, all geared towards capturing obstacles intricacies practical environments.