Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration

As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay...

Full description

Saved in:
Bibliographic Details
Published in:2021 58th ACM/IEEE Design Automation Conference (DAC) pp. 997 - 1002
Main Authors: Xu, Zirui, Yu, Fuxun, Xiong, Jinjun, Chen, Xiang
Format: Conference Proceeding
Language:English
Published: IEEE 05.12.2021
Subjects:
Collaboration
Collaborative work
Object recognition
Performance evaluation
Resistance
Solid modeling
Training
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings.
AbstractList As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings.
Author Yu, Fuxun
Xiong, Jinjun
Xu, Zirui
Chen, Xiang
Author_xml – sequence: 1
  givenname: Zirui
  surname: Xu
  fullname: Xu, Zirui
  email: zxu21@gmu.edu
  organization: George Mason University
– sequence: 2
  givenname: Fuxun
  surname: Yu
  fullname: Yu, Fuxun
  email: fyu2@gmu.edu
  organization: George Mason University
– sequence: 3
  givenname: Jinjun
  surname: Xiong
  fullname: Xiong, Jinjun
  email: jinjun@us.ibm.com
  organization: IBM Thomas J. Watson Research Center
– sequence: 4
  givenname: Xiang
  surname: Chen
  fullname: Chen, Xiang
  email: xchen26@gmu.edu
  organization: George Mason University
BookMark eNotz81Kw0AUBeARFNSaJxAhL5A4_5lxF9PWCAE36rbcSW7qwHQiSaDk7Q3YzTmbjwPnnlzHISIhT4zmjFH7vC0rZmghc045y60ymkt2RRJbGKa1koIXkt6SZJq8o5oqI9e8I981Bj9ML2mNM47DESP6ecnKM4yY7rHDEWbs0gZhjD4e07Off9LtEuHkWwhhSV8hQGxXUg0hgBtW74f4QG56CBMml96Qr_3us6qz5uPtvSqbDLgp5oxxhZIi7W0vrZKt4Fq0HVrpdEcVdbQtNIJxIPgK0GrmtOCMW4097x0TG_L4v-sR8fA7-hOMy-HyXvwBDuJS4A
ContentType Conference Proceeding
DBID 6IE
6IH
CBEJK
RIE
RIO
DOI 10.1109/DAC18074.2021.9586241
DatabaseName IEEE Electronic Library (IEL) Conference Proceedings
IEEE Proceedings Order Plan (POP) 1998-present by volume
IEEE Xplore All Conference Proceedings
IEEE Electronic Library (IEL)
IEEE Proceedings Order Plans (POP) 1998-present
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISBN 9781665432740
1665432748
EndPage 1002
ExternalDocumentID 9586241
Genre orig-research
GroupedDBID 6IE
6IH
ACM
ALMA_UNASSIGNED_HOLDINGS
CBEJK
RIE
RIO
ID FETCH-LOGICAL-a287t-125e40e0f9f4954c3263cde94b6d050b0c76ea8ba32f49e961b6321296ef2fb13
IEDL.DBID RIE
ISICitedReferencesCount 37
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000766079700167&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
IngestDate Wed Aug 27 02:28:29 EDT 2025
IsPeerReviewed false
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-a287t-125e40e0f9f4954c3263cde94b6d050b0c76ea8ba32f49e961b6321296ef2fb13
PageCount 6
ParticipantIDs ieee_primary_9586241
PublicationCentury 2000
PublicationDate 2021-Dec.-5
PublicationDateYYYYMMDD 2021-12-05
PublicationDate_xml – month: 12
  year: 2021
  text: 2021-Dec.-5
  day: 05
PublicationDecade 2020
PublicationTitle 2021 58th ACM/IEEE Design Automation Conference (DAC)
PublicationTitleAbbrev DAC
PublicationYear 2021
Publisher IEEE
Publisher_xml – name: IEEE
SSID ssib060584060
Score 2.4487944
Snippet As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical...
SourceID ieee
SourceType Publisher
StartPage 997
SubjectTerms Collaboration
Collaborative work
Object recognition
Performance evaluation
Resistance
Solid modeling
Training
Title Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
URI https://ieeexplore.ieee.org/document/9586241
WOSCitedRecordID wos000766079700167&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePCk0opvcvBo2mSzr3irW5cepPSgpbeSx6wUSlf6UPz3TrZrVfDiLSwJC5OBb77MzDeE3CCoSK21Y6AFEhThCpZGRjBwAlKLPmYrnYLxYzIcppOJGjXI7a4XBgCq4jPo-GWVy3el3finsq6KfDsDcp29JIm3vVpfvuOze4hNvG7SEVx1-71MeKkXJIGB6NRnfw1RqTAkP_zf349I-7sZj452MHNMGrBokTHCxaxc3dGBL2cp0QsAw2nWe9dLoLkXiMAY0tFaPfWF-udW2t9On9fz-Qe99yWNFrdkPx2hTZ7zh6dswOoRCUwj1VkzDE8g5MALVSDTCS0GY9I6UKGJHY-44TaJQadGywA3gIqFiSWilYqhCAoj5AlpLsoFnBIqlQOuUp-6RYpkpAqds7rQUaQhDYQ-Iy1vk-nrVgVjWpvj_O_PF-TAm70q_IguSXO93MAV2bdv69lqeV1d3SdOXpsz
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NSwMxEA2lCnpSacVvc_Bo2mSz2W681dZSsZYeaumtZJNZKZSu9EPx3zvZrl_gxduyZFlIBt68zLw3hFwhqEhjjGNgBBIU4VIWq0QwcAJiizFmc5-CUa_R78fjsR6UyPWXFgYA8uYzqPnHvJbvMrv2V2V1rbycAbnOlgrDgG_UWp_R4-t7iE68kOkIruvtZkt4sxekgYGoFV__GqOSo0hn73__3yfVbzkeHXwBzQEpwbxCRggY02x5Q7u-oSXDOABMqFnzzSyAdrxFBGaRjhb-qc_UX7jS9mb-vJnN3umtb2q0uKT1MxSq5KlzN2x1WTEkgRkkOyuGCQqEHHiqU-Q6ocV0TFoHOkwixxVPuG1EYOLEyAAXgI5EEknEKx1BGqSJkIekPM_mcESo1A64jn3xFklSInXonDWpUcpAHAhzTCp-TyYvGx-MSbEdJ3-_viQ73eFjb9K77z-ckl1_BHkbiDoj5dViDedk276upsvFRX6MH5ELnno
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+58th+ACM%2FIEEE+Design+Automation+Conference+%28DAC%29&rft.atitle=Helios%3A+Heterogeneity-Aware+Federated+Learning+with+Dynamically+Balanced+Collaboration&rft.au=Xu%2C+Zirui&rft.au=Yu%2C+Fuxun&rft.au=Xiong%2C+Jinjun&rft.au=Chen%2C+Xiang&rft.date=2021-12-05&rft.pub=IEEE&rft.spage=997&rft.epage=1002&rft_id=info:doi/10.1109%2FDAC18074.2021.9586241&rft.externalDocID=9586241