FedMDS: An Efficient Model Discrepancy-Aware Semi-asynchronous Clustered Federated Learning Framework

Federated learning (FL) is an emerging distributed machine learning paradigm that protects privacy and tackles the problem of isolated data islands. At present, there are two main communication strategies of FL: synchronous FL and asynchronous FL. The advantages of synchronous FL are that the model...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:IEEE transactions on parallel and distributed systems Ročník 34; číslo 3; s. 1 - 14
Hlavní autori: Zhang, Yu, Liu, Duo, Duan, Moming, Li, Li, Chen, Xianzhang, Ren, Ao, Tan, Yujuan, Wang, Chengliang
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: New York IEEE 01.03.2023
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:
Accuracy
Algorithms
Clients
Data models
Datasets
Distributed Machine Learning
Federated learning
Machine learning
Neural Network
Optimization
Risk communication
Servers
Strategy
Synchronism
Task analysis
Time synchronization
Training
Training data
ISSN:1045-9219, 1558-2183
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Federated learning (FL) is an emerging distributed machine learning paradigm that protects privacy and tackles the problem of isolated data islands. At present, there are two main communication strategies of FL: synchronous FL and asynchronous FL. The advantages of synchronous FL are that the model has high precision and converges easily. However, this synchronous communication strategy has the risk that the central server waits too long for the devices, namely, the straggler effect. Asynchronous FL has a natural advantage in mitigating the straggler effect, but there are threats of model quality degradation and server crash. In this paper, we demonstrate the negative impact of the asynchronous update mechanism on the accuracy of the model and the phenomenon that the client waits too long due to the existence of stragglers in the synchronous update mechanism. Therefore, we propose a model discrepancy-aware semi-asynchronous clustered federated learning framework ( FedMDS ), which alleviates the straggler effect by 1) a clustered strategy based on the delay and direction of the model update. 2) a synchronous trigger mechanism that limits the model staleness. FedMDS leverages the clustered algorithm to reschedule the clients, and each group of clients performs asynchronous updates until the synchronous update mechanism based on the model discrepancy is triggered. We evaluate FedMDS based on four federated datasets in a non-IID setting and compare FedMDS to the baseline methods. The experimental results show that FedMDS significantly improves average test accuracy by more than <inline-formula><tex-math notation="LaTeX">+9.2\%</tex-math></inline-formula> on the four datasets compared to TA-FedAvg . In particular, FedMDS improves absolute Top-1 test accuracy by <inline-formula><tex-math notation="LaTeX">+37.6\%</tex-math></inline-formula> on FEMNIST compared to TA-FedAvg . The frequency of the average synchronization waiting time (ASWT) of FedMDS is significantly lower than that of T-FedAvg on four datasets. The experiment results show that FedMDS can improve the accuracy and alleviate the straggler effect.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1045-9219
1558-2183
DOI:10.1109/TPDS.2023.3237752