A5041080860- Stochastic Gradient Optimization Techniques
- Privacy-Preserving Technologies in Data
- Sparse and Compressive Sensing Techniques
- Machine Learning and Algorithms
- Neural Networks and Applications
- Statistical Methods and Inference
- Mobile Crowdsensing and Crowdsourcing
- Age of Information Optimization
- Advanced Graph Neural Networks
- Recommender Systems and Techniques
- Cryptography and Data Security
- Markov Chains and Monte Carlo Methods
- Advanced Multi-Objective Optimization Algorithms
- Advanced Bandit Algorithms Research
- Advanced Neural Network Applications
- Machine Learning and Data Classification
- Advanced Optimization Algorithms Research
- Gaussian Processes and Bayesian Inference
- Distributed Sensor Networks and Detection Algorithms
- Parallel Computing and Optimization Techniques
- Natural Language Processing Techniques
- Smart Grid Energy Management
- Topic Modeling
- Advanced Causal Inference Techniques
- Embedded Systems Design Techniques
Mohamed bin Zayed University of Artificial Intelligence
2022-2025
King Abdullah University of Science and Technology
2018-2022
Computing Center
2022
Kootenay Association for Science & Technology
2020-2021
Federated learning and analytics are a distributed approach for collaboratively models (or statistics) from decentralized data, motivated by designed privacy protection. The process can be formulated as solving federated optimization problems, which emphasize communication efficiency, data heterogeneity, compatibility with system requirements, other constraints that not primary considerations in problem settings. This paper provides recommendations guidelines on formulating, designing,...
It is well understood that client-master communication can be a primary bottleneck in Federated Learning. In this work, we address issue with novel client subsampling scheme, where restrict the number of clients allowed to communicate their updates back master node. each round, all participating compute updates, but only ones "important" master. We show importance measured using norm update and give formula for optimal participation. This minimizes distance between full update, participate,...
We consider distributed optimization where the objective function is spread among different devices, each sending incremental model updates to a central server. To alleviate communication bottleneck, recent work proposed various schemes compress (e.g.\ quantize or sparsify) gradients, thereby introducing additional variance $ω\geq 1$ that might slow down convergence. For strongly convex functions with condition number $κ$ $n$ machines, we (i) give scheme converges in $\mathcal{O}((κ+...
Federated Learning (FL) has been gaining significant traction across different ML tasks, ranging from vision to keyboard predictions. In large-scale deployments, client heterogeneity is a fact and constitutes primary problem for fairness, training performance accuracy. Although efforts have made into tackling statistical data heterogeneity, the diversity in processing capabilities network bandwidth of clients, termed as system remained largely unexplored. Current solutions either disregard...
In this work, we consider the optimization formulation of personalized federated learning recently introduced by Hanzely and Richtárik (2020) which was shown to give an alternative explanation workings local {\tt SGD} methods. Our first contribution is establishing lower bounds for formulation, both communication complexity oracle complexity. second design several optimal methods matching these in almost all regimes. These are provably learning. include accelerated variant FedProx},...
We consider distributed optimization over several devices, each sending incremental model updates to a central server. This setting is considered, for instance, in federated learning. Various schemes have been designed compress the order reduce overall communication cost. However, existing methods suffer from significant slowdown due additional variance ω>0 coming compression operator and as result, only converge sublinearly. What needed reduction technique taming introduced by compression....
High-performance fuel design is imperative to achieve cleaner burning and high-efficiency engine systems. We introduce a data-driven artificial intelligence (AI) framework liquid fuels exhibiting tailor-made properties for combustion applications improve efficiency lower carbon emissions. The approach constrained optimization task integrating two parts: (i) deep learning (DL) model predict the of pure components mixtures (ii) search algorithms efficiently navigate in chemical space. Our...
In the last few years, various communication compression techniques have emerged as an indispensable tool helping to alleviate bottleneck in distributed learning. However, despite fact biased compressors often show superior performance practice when compared much more studied and understood unbiased compressors, very little is known about them. this work we study three classes of operators, two which are new, their applied (stochastic) gradient descent descent. We for first time that can...
Adaptivity is an important yet under-studied property in modern optimization theory. The gap between the state-of-the-art theory and current practice striking that algorithms with desirable theoretical guarantees typically involve drastically different settings of hyperparameters, such as step size schemes batch sizes, regimes. Despite appealing results, divisive strategies provide little, if any, insight to practitioners select work broadly without tweaking hyperparameters. In this work,...
LocalSGD and SCAFFOLD are widely used methods in distributed stochastic optimization, with numerous applications machine learning, large-scale data processing, federated learning. However, rigorously establishing their theoretical advantages over simpler methods, such as minibatch SGD (MbSGD), has proven challenging, existing analyses often rely on strong assumptions, unrealistic premises, or overly restrictive scenarios. In this work, we revisit the convergence properties of under a variety...
Collaborative learning enables multiple participants to learn a single global model by exchanging focused updates instead of sharing data. One the core challenges in collaborative is ensuring that are rewarded fairly for their contributions, which entails two key sub-problems: contribution assessment and reward allocation. This work focuses on fair allocation, where incentivized through rewards - differentiated final models whose performance commensurate with contribution. In this work, we...
Pruning offers a promising solution to mitigate the associated costs and environmental impact of deploying large deep neural networks (DNNs). Traditional approaches rely on computationally expensive trained models or time-consuming iterative prune-retrain cycles, undermining their utility in resource-constrained settings. To address this issue, we build upon established principles saliency (LeCun et al., 1989) connection sensitivity (Lee 2018) tackle challenging problem one-shot pruning...
Strong Differential Privacy (DP) and Optimization guarantees are two desirable properties for a method in Federated Learning (FL). However, existing algorithms do not achieve both at once: they either have optimal DP but rely on restrictive assumptions such as bounded gradients/bounded data heterogeneity, or ensure strong optimization performance lack guarantees. To address this gap the literature, we propose analyze new called Clip21-SGD2M based novel combination of clipping, heavy-ball...
Collaborative learning (CL) enables multiple participants to jointly train machine (ML) models on decentralized data sources without raw sharing. While the primary goal of CL is maximize expected accuracy gain for each participant, it also important ensure that gains are fairly distributed. Specifically, no client should be negatively impacted by collaboration, and individual must ideally commensurate with contributions. Most existing algorithms require central coordination focus...
Modern deep learning models are often trained in parallel over a collection of distributed machines to reduce training time. In such settings, communication model updates among becomes significant performance bottleneck and various lossy update compression techniques have been proposed alleviate this problem. work, we introduce new, simple yet theoretically practically effective technique: natural (NC). Our technique is applied individually all entries the to-be-compressed vector works by...
In modern federated learning, one of the main challenges is to account for inherent heterogeneity and diverse nature data distributions different clients. This problem often addressed by introducing personalization models towards distribution particular client. However, a personalized model might be unreliable when applied that not typical this Eventually, it may perform worse these than non-personalized global trained in way on from all paper presents new approach learning allows selecting...
In Federated Learning (FL), forgetting, or the loss of knowledge across rounds, hampers algorithm convergence, particularly in presence severe data heterogeneity among clients. This study explores nuances this issue, emphasizing critical role forgetting FL's inefficient learning within heterogeneous contexts. Knowledge occurs both client-local updates and server-side aggregation steps; addressing one without other fails to mitigate forgetting. We introduce a metric measure granularly,...