FedGroup: Efficient Federated Learning via Decomposed Similarity-Based Clustering

Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID and imbalanced (statistical heterogeneity) training data of FL is distribute...

Full description

Saved in:
Bibliographic Details
Published in:2021 IEEE Intl Conf on Parallel & Distributed Processing with Applications, Big Data & Cloud Computing, Sustainable Computing & Communications, Social Computing & Networking (ISPA/BDCloud/SocialCom/SustainCom) pp. 228 - 237
Main Authors: Duan, Moming, Liu, Duo, Ji, Xinyuan, Liu, Renping, Liang, Liang, Chen, Xianzhang, Tan, Yujuan
Format: Conference Proceeding
Language:English
Published: IEEE 01.09.2021
Subjects:
Collaborative work
Distributed Machine Learning
Federated Learning
Linear approximation
Neural networks
Performance evaluation
Scalability
Training
Training data
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Federated Learning (FL) enables the multiple participating devices to collaboratively contribute to a global neural network model while keeping the training data locally. Unlike the centralized training setting, the non-IID and imbalanced (statistical heterogeneity) training data of FL is distributed in the federated network, which will increase the divergences between the local models and the global model, further degrading performance. In this paper, we propose a novel clustered federated learning (CFL) framework FedGroup, in which we 1) group the training of clients based on the similarities between the clients' optimization directions for high training performance; 2) construct a new data-driven distance measure to improve the efficiency of the client clustering procedure. 3) implement a newcomer device cold start mechanism based on the auxiliary global model for framework scalability and practicality.FedGroup can achieve improvements by dividing joint optimization into groups of sub-optimization and can be combined with FL optimizer FedProx. The convergence and complexity are analyzed to demonstrate the efficiency of our proposed framework. We also evaluate FedGroup and FedGrouProx (combined with FedProx) on several open datasets and made comparisons with related CFL frameworks. The results show that FedGroup can significantly improve absolute test accuracy by +14.1% on FEMNIST compared to FedAvg, +3.4% on Sentiment140 compared to FedProx, +6.9% on MNIST compared to FeSEM.
DOI:10.1109/ISPA-BDCloud-SocialCom-SustainCom52081.2021.00042