Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration

As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:2021 58th ACM/IEEE Design Automation Conference (DAC) s. 997 - 1002
Hlavní autori: Xu, Zirui, Yu, Fuxun, Xiong, Jinjun, Chen, Xiang
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 05.12.2021
Predmet:
Collaboration
Collaborative work
Object recognition
Performance evaluation
Resistance
Solid modeling
Training
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings.
DOI:10.1109/DAC18074.2021.9586241