Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration

As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2021 58th ACM/IEEE Design Automation Conference (DAC) s. 997 - 1002
Hlavní autoři: Xu, Zirui, Yu, Fuxun, Xiong, Jinjun, Chen, Xiang
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 05.12.2021
Témata:
Collaboration
Collaborative work
Object recognition
Performance evaluation
Resistance
Solid modeling
Training
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings.
DOI:10.1109/DAC18074.2021.9586241