Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration
As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay...
Uloženo v:
| Vydáno v: | 2021 58th ACM/IEEE Design Automation Conference (DAC) s. 997 - 1002 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Konferenční příspěvek |
| Jazyk: | angličtina |
| Vydáno: |
IEEE
05.12.2021
|
| Témata: | |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings. |
|---|---|
| AbstractList | As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical neural network models to heterogeneous devices, the ones with weak computational capacities, referred to as stragglers, may significantly delay the synchronous parameter aggregation. Although discarding stragglers from the collaboration can relieve this issue to a certain extent, stragglers may keep unique and critical information learned from the non-identical dataset, and directly discarding will harm the overall collaboration performance. Therefore, in this paper, we propose Helios - a heterogeneity-aware FL framework to tackle the straggler issue. Helios identifies individual devices' heterogeneous training capability, and therefore the expected neural network model training volumes regarding the collaborative training pace. For straggling devices, a "softtraining" method is proposed to dynamically compress the original identical training model into the expected volume through a rotated neuron training approach. With extensive algorithm analysis and optimization schemes, stragglers can be accelerated while retaining the convergence for local training as well as federated collaboration. Experiments show that Helios can provide up to 2.5\times training acceleration and maximum 4.64% convergence accuracy improvement in various collaboration settings. |
| Author | Yu, Fuxun Xiong, Jinjun Xu, Zirui Chen, Xiang |
| Author_xml | – sequence: 1 givenname: Zirui surname: Xu fullname: Xu, Zirui email: zxu21@gmu.edu organization: George Mason University – sequence: 2 givenname: Fuxun surname: Yu fullname: Yu, Fuxun email: fyu2@gmu.edu organization: George Mason University – sequence: 3 givenname: Jinjun surname: Xiong fullname: Xiong, Jinjun email: jinjun@us.ibm.com organization: IBM Thomas J. Watson Research Center – sequence: 4 givenname: Xiang surname: Chen fullname: Chen, Xiang email: xchen26@gmu.edu organization: George Mason University |
| BookMark | eNotz81Kw0AUBeARFNSaJxAhL5A4_5lxF9PWCAE36rbcSW7qwHQiSaDk7Q3YzTmbjwPnnlzHISIhT4zmjFH7vC0rZmghc045y60ymkt2RRJbGKa1koIXkt6SZJq8o5oqI9e8I981Bj9ML2mNM47DESP6ecnKM4yY7rHDEWbs0gZhjD4e07Off9LtEuHkWwhhSV8hQGxXUg0hgBtW74f4QG56CBMml96Qr_3us6qz5uPtvSqbDLgp5oxxhZIi7W0vrZKt4Fq0HVrpdEcVdbQtNIJxIPgK0GrmtOCMW4097x0TG_L4v-sR8fA7-hOMy-HyXvwBDuJS4A |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/DAC18074.2021.9586241 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Xplore url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| EISBN | 9781665432740 1665432748 |
| EndPage | 1002 |
| ExternalDocumentID | 9586241 |
| Genre | orig-research |
| GroupedDBID | 6IE 6IH ACM ALMA_UNASSIGNED_HOLDINGS CBEJK RIE RIO |
| ID | FETCH-LOGICAL-a287t-125e40e0f9f4954c3263cde94b6d050b0c76ea8ba32f49e961b6321296ef2fb13 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 37 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000766079700167&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:28:29 EDT 2025 |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a287t-125e40e0f9f4954c3263cde94b6d050b0c76ea8ba32f49e961b6321296ef2fb13 |
| PageCount | 6 |
| ParticipantIDs | ieee_primary_9586241 |
| PublicationCentury | 2000 |
| PublicationDate | 2021-Dec.-5 |
| PublicationDateYYYYMMDD | 2021-12-05 |
| PublicationDate_xml | – month: 12 year: 2021 text: 2021-Dec.-5 day: 05 |
| PublicationDecade | 2020 |
| PublicationTitle | 2021 58th ACM/IEEE Design Automation Conference (DAC) |
| PublicationTitleAbbrev | DAC |
| PublicationYear | 2021 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssib060584060 |
| Score | 2.4487944 |
| Snippet | As Federated Learning (FL) has been widely used for collaborative training, a considerable computational straggler issue emerged: when FL deploys identical... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 997 |
| SubjectTerms | Collaboration Collaborative work Object recognition Performance evaluation Resistance Solid modeling Training |
| Title | Helios: Heterogeneity-Aware Federated Learning with Dynamically Balanced Collaboration |
| URI | https://ieeexplore.ieee.org/document/9586241 |
| WOSCitedRecordID | wos000766079700167&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEA61ePCk0opvcvDotsk-ko23Wi09SOlBpbeSZGelULrSh-K_d2Z3rQpe3NOyJCxMhnwzyXzfMHYlXaQh1xDg3pcFsQTcB61NA2pnHZMOKlRdSx70aJROJmbcYNdbLgwAlMVn0KHX8i4_K_yGjsq6JiE6A-Y6O1qriqv15Tt0u4fYJGqSjhSme9frS5J6wSQwlJ167q8mKiWGDPb_9_cD1v4m4_HxFmYOWQMWLfaMcDErVjd8SOUsBXoBYDgd9N7tEviABCIwhsx4rZ76wum4ld9V3eftfP7Bb6mk0eOQ_k9HaLOnwf1jfxjULRICi6nOGi2cQCxA5CbHTCf2GIxFPgMTO5WJRDjhtQKbOhuFOACMkk5FiFZGQR7mTkZHrLkoFnBM5G2n8fEitC7WeZLqzGHwKK3GSdqoE9Yim0xfKxWMaW2O078_n7E9MntZ-JGcs-Z6uYELtuvf1rPV8rJcuk-KK5oR |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3PS8MwFA5jCnpS2cTf5uDRbkmbNo23uTkmzrHDlN1Gkr7KYKyyH4r_vS9tnQpe7KmUhMLLI997yfu-R8gVN4GEVIKHe1_iCQ64D2ode66dtXA6qFB0LenLwSAej9WwQq43XBgAyIvPoOFe87v8JLNrd1TWVKGjM2CusxUK4bOCrfXlPe5-D9GJlTQdzlSz02pzJ_aCaaDPG-XsX21UchTp7v3v__uk_k3Ho8MN0ByQCsxr5BkBY5otb2jPFbRk6AeAAbXXetcLoF0nEYFRZEJL_dQX6g5caafoP69nsw9664oaLQ5p_3SFOnnq3o3aPa9skuBpTHZWaOMQBAOWqhRzHWExHAtsAkqYKGEhM8zKCHRsdODjAFARN1GAeKUiSP3U8OCQVOfZHI4cfdtIfCzztREyDWOZGAwfuZY4SaromNScTSavhQ7GpDTHyd-fL8lOb_TYn_TvBw-nZNctQV4GEp6R6mqxhnOybd9W0-XiIl_GT72cnVg |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=2021+58th+ACM%2FIEEE+Design+Automation+Conference+%28DAC%29&rft.atitle=Helios%3A+Heterogeneity-Aware+Federated+Learning+with+Dynamically+Balanced+Collaboration&rft.au=Xu%2C+Zirui&rft.au=Yu%2C+Fuxun&rft.au=Xiong%2C+Jinjun&rft.au=Chen%2C+Xiang&rft.date=2021-12-05&rft.pub=IEEE&rft.spage=997&rft.epage=1002&rft_id=info:doi/10.1109%2FDAC18074.2021.9586241&rft.externalDocID=9586241 |