Adaptive Weighting Push-SUM for Decentralized Optimization With Statistical Diversity
Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of the Push-SUM protocol reduce the performance in handling the statistical diversity of optimization algorithms based on it. In this article, we...
Uloženo v:
| Vydáno v: | IEEE transactions on control of network systems Ročník 12; číslo 3; s. 2337 - 2349 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
Piscataway
IEEE
01.09.2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Témata: | |
| ISSN: | 2325-5870, 2372-2533 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of the Push-SUM protocol reduce the performance in handling the statistical diversity of optimization algorithms based on it. In this article, we theoretically and empirically reduce the negative impact of statistical diversity on decentralized optimization using the Push-SUM protocol. Specifically, we propose the adaptive weighting Push-SUM protocol, a theoretical generalization of the original Push-SUM protocol, where the latter is a special case of the former. Our theoretical analysis shows that, with sufficient communication, the upper bound on the consensus distance for the new protocol reduces to <inline-formula><tex-math notation="LaTeX">O(1/N)</tex-math></inline-formula>, whereas it remains at <inline-formula><tex-math notation="LaTeX">O(1)</tex-math></inline-formula> for the Push-SUM protocol. We adopt stochastic gradient descent (SGD) and momentum SGD on the new protocol and prove that the convergence rate of these two algorithms to statistical diversity is <inline-formula><tex-math notation="LaTeX">O(N/T)</tex-math></inline-formula> on the new protocol, while it is <inline-formula><tex-math notation="LaTeX">O(Nd/T)</tex-math></inline-formula> on the Push-SUM protocol, where <inline-formula><tex-math notation="LaTeX">d</tex-math></inline-formula> is the parameter size of the training model. To address statistical diversity in practical applications of the new protocol, we develop the Moreau weighting method for its generalized weight matrix definition. This method, derived from the Moreau envelope, is an approximate optimization of the distance penalty of the Moreau envelope. We verify that the adaptive weighting Push-SUM protocol is practically more efficient than the Push-SUM protocol via deep learning experiments. |
|---|---|
| AbstractList | Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of the Push-SUM protocol reduce the performance in handling the statistical diversity of optimization algorithms based on it. In this article, we theoretically and empirically reduce the negative impact of statistical diversity on decentralized optimization using the Push-SUM protocol. Specifically, we propose the adaptive weighting Push-SUM protocol, a theoretical generalization of the original Push-SUM protocol, where the latter is a special case of the former. Our theoretical analysis shows that, with sufficient communication, the upper bound on the consensus distance for the new protocol reduces to [Formula Omitted], whereas it remains at [Formula Omitted] for the Push-SUM protocol. We adopt stochastic gradient descent (SGD) and momentum SGD on the new protocol and prove that the convergence rate of these two algorithms to statistical diversity is [Formula Omitted] on the new protocol, while it is [Formula Omitted] on the Push-SUM protocol, where [Formula Omitted] is the parameter size of the training model. To address statistical diversity in practical applications of the new protocol, we develop the Moreau weighting method for its generalized weight matrix definition. This method, derived from the Moreau envelope, is an approximate optimization of the distance penalty of the Moreau envelope. We verify that the adaptive weighting Push-SUM protocol is practically more efficient than the Push-SUM protocol via deep learning experiments. Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of the Push-SUM protocol reduce the performance in handling the statistical diversity of optimization algorithms based on it. In this article, we theoretically and empirically reduce the negative impact of statistical diversity on decentralized optimization using the Push-SUM protocol. Specifically, we propose the adaptive weighting Push-SUM protocol, a theoretical generalization of the original Push-SUM protocol, where the latter is a special case of the former. Our theoretical analysis shows that, with sufficient communication, the upper bound on the consensus distance for the new protocol reduces to <inline-formula><tex-math notation="LaTeX">O(1/N)</tex-math></inline-formula>, whereas it remains at <inline-formula><tex-math notation="LaTeX">O(1)</tex-math></inline-formula> for the Push-SUM protocol. We adopt stochastic gradient descent (SGD) and momentum SGD on the new protocol and prove that the convergence rate of these two algorithms to statistical diversity is <inline-formula><tex-math notation="LaTeX">O(N/T)</tex-math></inline-formula> on the new protocol, while it is <inline-formula><tex-math notation="LaTeX">O(Nd/T)</tex-math></inline-formula> on the Push-SUM protocol, where <inline-formula><tex-math notation="LaTeX">d</tex-math></inline-formula> is the parameter size of the training model. To address statistical diversity in practical applications of the new protocol, we develop the Moreau weighting method for its generalized weight matrix definition. This method, derived from the Moreau envelope, is an approximate optimization of the distance penalty of the Moreau envelope. We verify that the adaptive weighting Push-SUM protocol is practically more efficient than the Push-SUM protocol via deep learning experiments. |
| Author | Xu, Linli Cheng, Yifei Chen, Enhong Zhou, Yiming |
| Author_xml | – sequence: 1 givenname: Yiming orcidid: 0009-0000-3151-7406 surname: Zhou fullname: Zhou, Yiming email: zym2019@mail.ustc.edu.cn organization: Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China, Hefei, China – sequence: 2 givenname: Yifei orcidid: 0000-0003-3859-2921 surname: Cheng fullname: Cheng, Yifei email: yfcheng.ifc@gmail.com organization: Shenzhen Campus of Sun Yat-sen University, Shenzhen, China – sequence: 3 givenname: Linli orcidid: 0000-0003-0227-3793 surname: Xu fullname: Xu, Linli email: linlixu@ustc.edu.cn organization: Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China, Hefei, China – sequence: 4 givenname: Enhong orcidid: 0000-0002-4835-4102 surname: Chen fullname: Chen, Enhong email: cheneh@ustc.edu.cn organization: Anhui Province Key Laboratory of Big Data Analysis and Application, University of Science and Technology of China, Hefei, China |
| BookMark | eNp9kEtrAjEUhUOxUGv9AYUuAl2PzcPMJEvRvsDWgorLEDMZjYwzNokF_fXNVBeli67uvXDOuZzvGrSqujIA3GLUwxiJh9nwfdojiLAeZWlKibgAbUIzkhBGaavZCUsYz9AV6Hq_QQhhwuJN22A-yNUu2C8DF8au1sFWK_ix9-tkOn-DRe3gyGhTBadKezQ5nETt1h5VsHUFFzas4TTEwwerVQlHMcd5Gw434LJQpTfd8-yA-dPjbPiSjCfPr8PBONFEpCHRuVAoW_YpKxTBSpAc5whrwQpiMlTkhqMlLYRWGukl1YRzhlWa9VMiOGJa0w64P-XuXP25Nz7ITb13VXwpY2XKBeN9GlXZSaVd7b0zhdQ2_FSIvWwpMZINRtlglA1GecYYnfiPc-fsVrnDv567k8caY37pBccpR_Qbl6yAXQ |
| CODEN | ITCNAY |
| CitedBy_id | crossref_primary_10_15622_ia_24_4_2 |
| Cites_doi | 10.1109/TCNS.2023.3338236 10.1287/moor.17.3.670 10.1109/SFCS.2003.1238221 10.1109/TCNS.2024.3354875 10.1016/0041-5553(64)90137-5 10.1109/TCNS.2023.3281561 10.1109/TAC.2014.2364096 10.1109/TNSE.2022.3225229 10.1186/s13634-018-0596-y 10.1137/16M1084316 10.1007/s10107-018-01357-w 10.1016/s0294-1449(17)30036-7 10.1109/TAC.2017.2737582 10.1109/TAC.2017.2648041 10.1109/CVPR.2016.90 10.1109/TAC.2020.2981035 10.1137/14096668X 10.1109/TAC.2020.2972824 10.1109/TAC.2016.2529285 10.1109/LCSYS.2020.3006420 10.1109/LCSYS.2018.2834316 10.1109/5.726791 10.1109/TCNS.2023.3272220 10.1109/JPROC.2020.3024266 10.1109/LCSYS.2021.3090652 10.1109/CDC.2015.7402509 10.1109/TAC.2017.2672698 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TCNS.2025.3566329 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Computer and Information Systems Abstracts |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 2372-2533 |
| EndPage | 2349 |
| ExternalDocumentID | 10_1109_TCNS_2025_3566329 10981680 |
| Genre | orig-research |
| GroupedDBID | 0R~ 4.4 6IK 97E AAJGR AASAJ AAWTH ABAZT ABJNI ABQJQ ABVLG AGQYO AGSQL AHBIQ AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ EBS EJD IFIPE IPLJI JAVBF M43 OCL PQQKQ RIA RIE AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c296t-cd9a07b435fa21a92d1d01c95f2e70fde80b3f9cac0cb3c28851a674629805cc3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001579032800010&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2325-5870 |
| IngestDate | Mon Oct 13 19:10:35 EDT 2025 Tue Nov 18 21:32:52 EST 2025 Sat Nov 29 07:23:43 EST 2025 Wed Oct 01 07:05:10 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c296t-cd9a07b435fa21a92d1d01c95f2e70fde80b3f9cac0cb3c28851a674629805cc3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0003-0227-3793 0000-0002-4835-4102 0000-0003-3859-2921 0009-0000-3151-7406 |
| PQID | 3253895843 |
| PQPubID | 2040410 |
| PageCount | 13 |
| ParticipantIDs | crossref_citationtrail_10_1109_TCNS_2025_3566329 crossref_primary_10_1109_TCNS_2025_3566329 proquest_journals_3253895843 ieee_primary_10981680 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-09-01 |
| PublicationDateYYYYMMDD | 2025-09-01 |
| PublicationDate_xml | – month: 09 year: 2025 text: 2025-09-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationPlace | Piscataway |
| PublicationPlace_xml | – name: Piscataway |
| PublicationTitle | IEEE transactions on control of network systems |
| PublicationTitleAbbrev | TCNS |
| PublicationYear | 2025 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref35 ref12 ref34 ref15 ref37 ref14 ref11 ref33 ref10 ref2 ref1 ref17 ref16 ref19 ref18 Li (ref31) 2021 Yu (ref28) 2019 Dinh (ref29) 2020; 33 ref23 ref26 ref25 ref22 Krizhevsky (ref36) 2009 Lin (ref30) 2018; 18 Yosida (ref32) 1980; 123 ref27 ref8 ref7 Jiang (ref20) 2021; 34 ref9 Assran (ref13) 2019 ref4 ref3 ref6 ref5 Moreau (ref21) 1962; 255 Koloskova (ref24) 2021; 34 |
| References_xml | – ident: ref1 doi: 10.1109/TCNS.2023.3338236 – volume: 255 start-page: 2897 year: 1962 ident: ref21 article-title: Fonctions convexes duales et points proximaux dans un espace hilbertien publication-title: C. R. Acad. Sci. Paris – ident: ref33 doi: 10.1287/moor.17.3.670 – ident: ref5 doi: 10.1109/SFCS.2003.1238221 – ident: ref4 doi: 10.1109/TCNS.2024.3354875 – ident: ref27 doi: 10.1016/0041-5553(64)90137-5 – ident: ref3 doi: 10.1109/TCNS.2023.3281561 – ident: ref6 doi: 10.1109/TAC.2014.2364096 – ident: ref16 doi: 10.1109/TNSE.2022.3225229 – start-page: 6357 volume-title: Proc. Int. Conf. Mach. Learn. year: 2021 ident: ref31 article-title: Ditto: Fair and robust federated learning through personalization – ident: ref11 doi: 10.1186/s13634-018-0596-y – start-page: 344 volume-title: Proc. Int. Conf. Mach. Learn. year: 2019 ident: ref13 article-title: Stochastic gradient push for distributed deep learning – ident: ref23 doi: 10.1137/16M1084316 – volume: 34 start-page: 11422 volume-title: Proc. Int. Conf. Neural Inf. Process. Syst. year: 2021 ident: ref24 article-title: An improved analysis of gradient tracking for decentralized machine learning – ident: ref18 doi: 10.1007/s10107-018-01357-w – volume: 34 start-page: 20185 volume-title: Proc. Int. Conf. Neural Inf. Process. Syst. year: 2021 ident: ref20 article-title: Asynchronous decentralized online learning – ident: ref34 doi: 10.1016/s0294-1449(17)30036-7 – ident: ref25 doi: 10.1109/TAC.2017.2737582 – ident: ref7 doi: 10.1109/TAC.2017.2648041 – ident: ref37 doi: 10.1109/CVPR.2016.90 – ident: ref19 doi: 10.1109/TAC.2020.2981035 – ident: ref15 doi: 10.1137/14096668X – ident: ref9 doi: 10.1109/TAC.2020.2972824 – ident: ref12 doi: 10.1109/TAC.2016.2529285 – ident: ref26 doi: 10.1109/LCSYS.2020.3006420 – ident: ref8 doi: 10.1109/LCSYS.2018.2834316 – ident: ref35 doi: 10.1109/5.726791 – ident: ref2 doi: 10.1109/TCNS.2023.3272220 – volume: 123 volume-title: Functional Analysis year: 1980 ident: ref32 – year: 2009 ident: ref36 article-title: Learning multiple layers of features from tiny images – start-page: 7184 volume-title: Proc. Int. Conf. Mach. Learn. year: 2019 ident: ref28 article-title: On the linear speedup analysis of communication efficient momentum SGD for distributed non-convex optimization – ident: ref10 doi: 10.1109/JPROC.2020.3024266 – ident: ref17 doi: 10.1109/LCSYS.2021.3090652 – ident: ref22 doi: 10.1109/CDC.2015.7402509 – ident: ref14 doi: 10.1109/TAC.2017.2672698 – volume: 33 start-page: 21394 volume-title: Proc. Int. Conf. Neural Inf. Process. Syst. year: 2020 ident: ref29 article-title: Personalized federated learning with Moreau envelopes – volume: 18 start-page: 7854 issue: 1 year: 2018 ident: ref30 article-title: Catalyst acceleration for first-order convex optimization: From theory to practice publication-title: J. Mach. Learn. Res. |
| SSID | ssj0001255873 |
| Score | 2.3442137 |
| Snippet | Statistical diversity is a property of data distribution and can hinder the optimization of a decentralized network. However, the theoretical limitations of... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 2337 |
| SubjectTerms | Algorithms Analytical models Communication networks Control systems Convergence directed decentralized optimization distributed algorithms/control learning Network systems Optimization Protocols Robustness Training Upper bound Upper bounds Vectors Weighting methods |
| Title | Adaptive Weighting Push-SUM for Decentralized Optimization With Statistical Diversity |
| URI | https://ieeexplore.ieee.org/document/10981680 https://www.proquest.com/docview/3253895843 |
| Volume | 12 |
| WOSCitedRecordID | wos001579032800010&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2372-2533 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001255873 issn: 2325-5870 databaseCode: RIE dateStart: 20140101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3NS8MwFA86POjBz4nTKTl4EjrTpG2S49gcHmQO3NhupU1SNtBt7MODf70vaaYDUfBWyktb3q95-eXlfSB0C2tcYoQWgVLcBBEXOshZYd0cgpvEJg7kDukn3u2K0Uj2fLK6y4UxxrjgM9Owl-4sX8_U2rrKYIZL2yYCdui7nCdlstaWQyWOBWf-5BJE7_ut7gvsAGncYEBamGOR32uPa6bywwK7ZaVz9M8POkaHnj_iZgn4Cdox01N0sFVV8AwNmjqbWyuGh87vCTdxb70cB2BBMXBU3DY-JHPyYTR-Btk3n42Jh5PVGFsC6uo3w4vam7iNKhp0Hvqtx8B3TwgUlckqUFpmhOdAh4qMhpmkOtQkVDIuqOGk0EYQQEWqTBGVM0UFcK8s4VFCpSCxUuwcVaazqblAuKA5TQRlmnEThVGYF5xoGcXC5ElR0LiGyEavqfKlxW2Hi9fUbTGITC0UqYUi9VDU0N3XkHlZV-Mv4arV_ZZgqfYaqm_QS_3UW6aMgg2XwKvY5S_DrtC-fXoZKVZHldViba7RnnoH3S5u3F_1CeY7y4s |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFA6igvrgXZzXPPgkdKZJ2ySP4gXFWQU3trfSJikb6DZ28cFf70ma6UAUfCvlhJTzNSdfTs4FoTPY4xIjtAiU4iaIuNBBwUrr5hDcJDZxoHBIN3iaik5HPvtkdZcLY4xxwWembh_dXb4eqKl1lcEKl7ZNBJzQl-IooqRK15pzqcSx4MzfXYLwRfMqfYEzII3rDGgLczzye_dx7VR-2GC3sdxu_POTNtG6Z5D4soJ8Cy2Y_jZam6sruINalzofWjuG287zCS_x83TcDcCGYmCp-Nr4oMzeh9H4CWTffD4mbvcmXWwpqKvgDBNdzyI3dlHr9qZ5dRf4_gmBojKZBErLnPACCFGZ0zCXVIeahErGJTWclNoIArhIlSuiCqaoAPaVJzxKqBQkVortocX-oG_2ES5pQRNBmWbcRGEUFiUnWkaxMEVSljSuITLTa6Z8cXHb4-I1c4cMIjMLRWahyDwUNXT-NWRYVdb4S3jX6n5OsFJ7DR3N0Mv84htnjIIVl8Cs2MEvw07Ryl3zsZE17tOHQ7RqZ6rixo7Q4mQ0NcdoWb2Dnkcn7g_7BKR-ztI |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Adaptive+Weighting+Push-SUM+for+Decentralized+Optimization+With+Statistical+Diversity&rft.jtitle=IEEE+transactions+on+control+of+network+systems&rft.au=Zhou%2C+Yiming&rft.au=Cheng%2C+Yifei&rft.au=Xu%2C+Linli&rft.au=Chen%2C+Enhong&rft.date=2025-09-01&rft.issn=2325-5870&rft.eissn=2372-2533&rft.volume=12&rft.issue=3&rft.spage=2337&rft.epage=2349&rft_id=info:doi/10.1109%2FTCNS.2025.3566329&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TCNS_2025_3566329 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2325-5870&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2325-5870&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2325-5870&client=summon |