Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers
Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to p...
Saved in:
| Published in: | IEEE transactions on signal processing Vol. 67; no. 22; pp. 5850 - 5864 |
|---|---|
| Main Authors: | , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
15.11.2019
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 1053-587X, 1941-0476 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms. |
|---|---|
| AbstractList | Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of distributed machine learning algorithms. However, distributed algorithms are sensitive to Byzantine attackers who can send falsified data to prevent the convergence of algorithms or lead the algorithms to converge to value of the attackers' choice. Some recent work proposed interesting algorithms that can deal with the scenario when up to half of the workers are compromised. In this paper, we propose a novel algorithm that can deal with an arbitrary number of Byzantine attackers. The main idea is to ask the parameter server to randomly select a small clean dataset and compute noisy gradient using this small dataset. This noisy gradient will then be used as a ground truth to filter out information sent by compromised workers. We show that the proposed algorithm converges to the neighborhood of the population minimizer regardless the number of Byzantine attackers. We further provide numerical examples to show that the proposed algorithm can benefit from the presence of good workers and achieve better performance than existing algorithms. |
| Author | Lai, Lifeng Cao, Xinyang |
| Author_xml | – sequence: 1 givenname: Xinyang surname: Cao fullname: Cao, Xinyang email: xycao@ucdavis.edu organization: Department of Electrical and Computer Engineering, University of California, Davis, CA, USA – sequence: 2 givenname: Lifeng orcidid: 0000-0002-9493-8248 surname: Lai fullname: Lai, Lifeng email: lflai@ucdavis.edu organization: Department of Electrical and Computer Engineering, University of California, Davis, CA, USA |
| BookMark | eNp9kL1PwzAUxC1UJNrCjsRiiTnFH4kbj6GFglQBgiIhMUR2_AIubVxsZyh_PamKGBiY7g33e6e7Aeo1rgGETikZUUrkxeLpYcQIlSMmU0EYOUB9KlOakHQset1NMp5k-fjlCA1CWBJC01SKPnqd2hC91W0Eg2deGQtNxFMI1U6L1ZvzNr6v8aPTbYg4OqwaXHhto1d-i-_atQaPXY0vt1-qibYBXMSoqg_w4Rgd1moV4ORHh-j5-moxuUnm97PbSTFPKiZpTEyeCaMBUiC5qQ2hDDRQUUtaGZ3JnOdiDCBrykRFqFbM1AIyYjQ3nCum-BCd7_9uvPtsIcRy6VrfdJEl45QIkTGedS6yd1XeheChLjferrsSJSXlbsKym7DcTVj-TNgh4g9S2aiidU3X3q7-A8_2oAWA35w8F5RLzr8Bh1SBiQ |
| CODEN | ITPRED |
| CitedBy_id | crossref_primary_10_1109_JIOT_2023_3272334 crossref_primary_10_1145_3616537 crossref_primary_10_1109_TIT_2024_3442211 crossref_primary_10_1080_0952813X_2024_2391778 crossref_primary_10_1109_JSAC_2025_3560046 crossref_primary_10_1007_s11425_023_2217_2 crossref_primary_10_1016_j_sigpro_2024_109419 crossref_primary_10_1016_j_comnet_2023_110081 crossref_primary_10_3390_app14010351 crossref_primary_10_1109_TIFS_2024_3360869 crossref_primary_10_1109_TCE_2024_3376561 crossref_primary_10_1016_j_comnet_2025_111234 crossref_primary_10_1109_TNSE_2025_3551923 crossref_primary_10_3390_electronics10141687 crossref_primary_10_1016_j_ijepes_2020_106662 crossref_primary_10_1109_TMC_2025_3571058 crossref_primary_10_1109_ACCESS_2025_3602217 crossref_primary_10_1109_TSP_2025_3564842 crossref_primary_10_1109_TNNLS_2025_3563537 crossref_primary_10_1007_s10462_024_10797_0 crossref_primary_10_1109_TIFS_2025_3564879 crossref_primary_10_1109_TSP_2025_3539883 crossref_primary_10_1109_TSP_2020_3029461 crossref_primary_10_3390_sym17071159 crossref_primary_10_1109_ACCESS_2023_3297112 crossref_primary_10_1002_cpe_8084 crossref_primary_10_3390_electronics9061038 crossref_primary_10_1109_TNNLS_2024_3436149 crossref_primary_10_1109_TSIPN_2023_3265892 crossref_primary_10_1016_j_ins_2022_10_120 crossref_primary_10_1007_s11277_020_07354_7 crossref_primary_10_1016_j_adhoc_2025_103934 crossref_primary_10_1016_j_sigpro_2023_109222 crossref_primary_10_1109_JIOT_2024_3523293 crossref_primary_10_1109_JSTSP_2021_3137028 crossref_primary_10_1109_TSP_2021_3078624 crossref_primary_10_1109_TIT_2025_3542896 crossref_primary_10_3390_electronics13081540 crossref_primary_10_1109_TSP_2022_3153135 |
| Cites_doi | 10.1109/TSP.2015.2393839 10.1007/s10107-015-0901-6 10.1016/j.radonc.2016.10.002 10.1109/TSP.2015.2430837 10.1109/TSP.2017.2656847 10.1080/01621459.2018.1429274 10.1145/2046684.2046692 10.1002/9780470515143.ch9 10.1109/TSP.2017.2695451 10.1561/2200000016 10.1145/3219617.3219655 10.1109/ICASSP.2018.8461691 10.1109/TSP.2015.2434327 10.1109/TSP.2013.2273887 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2019 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TSP.2019.2946020 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Statistics |
| EISSN | 1941-0476 |
| EndPage | 5864 |
| ExternalDocumentID | 10_1109_TSP_2019_2946020 8861393 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: National Science Foundation grantid: ECCS-17-11468; CCF-17-17943; CNS-1824553; CCF-1908258 funderid: 10.13039/100000001 |
| GroupedDBID | -~X .DC 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AGQYO AHBIQ AJQPL AKJIK AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 EBS EJD F5P HZ~ IFIPE IPLJI JAVBF LAI MS~ O9- OCL P2P RIA RIE RNS TAE TN5 AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D RIG |
| ID | FETCH-LOGICAL-c291t-d856dbee4e08dfd012ebe16f91cdb5983867ee9f126c01ba2df6e50db3d33a2a3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 58 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000494309800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1053-587X |
| IngestDate | Mon Jun 30 10:11:16 EDT 2025 Tue Nov 18 22:04:51 EST 2025 Sat Nov 29 04:10:50 EST 2025 Wed Aug 27 02:40:42 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 22 |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c291t-d856dbee4e08dfd012ebe16f91cdb5983867ee9f126c01ba2df6e50db3d33a2a3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0000-0002-9493-8248 |
| PQID | 2310665235 |
| PQPubID | 85478 |
| PageCount | 15 |
| ParticipantIDs | ieee_primary_8861393 proquest_journals_2310665235 crossref_primary_10_1109_TSP_2019_2946020 crossref_citationtrail_10_1109_TSP_2019_2946020 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-11-15 |
| PublicationDateYYYYMMDD | 2019-11-15 |
| PublicationDate_xml | – month: 11 year: 2019 text: 2019-11-15 day: 15 |
| PublicationDecade | 2010 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on signal processing |
| PublicationTitleAbbrev | TSP |
| PublicationYear | 2019 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | ref13 ref12 ref15 richtárik (ref17) 2016; 17 su (ref27) 2018 moritz (ref6) 2016 ref14 chen (ref25) 0 ref30 ref11 ref10 zhang (ref16) 0 wainwright (ref35) 2015 ref1 lecun (ref32) 2010; 2 ref19 bottou (ref7) 0 blanchard (ref21) 2017 ref18 gordon (ref31) 2006; 22 crotty (ref2) 2014; 37 alistarh (ref22) 0 xie (ref29) 2018 ref20 krizhevsky (ref33) 2014 vershynin (ref34) 2010 damaskinos (ref26) 2018 yin (ref24) 2018 yin (ref28) 2018 lee (ref8) 2017; 18 ref9 xie (ref23) 2018 ref3 ref5 provost (ref4) 0; 1 |
| References_xml | – ident: ref13 doi: 10.1109/TSP.2015.2393839 – ident: ref18 doi: 10.1007/s10107-015-0901-6 – ident: ref19 doi: 10.1016/j.radonc.2016.10.002 – year: 2010 ident: ref34 article-title: Introduction to the non-asymptotic analysis of random matrices publication-title: arXiv 1011 3027 – start-page: 1 year: 0 ident: ref25 article-title: DRACO: Byzantine-resilient Distributed Training via Redundant Gradients publication-title: Proc Intl Conf Mach Learning – ident: ref12 doi: 10.1109/TSP.2015.2430837 – year: 2018 ident: ref29 article-title: Zeno: Byzantine-suspicious stochastic gradient descent publication-title: arXiv 1805 10032 – start-page: 4613 year: 0 ident: ref22 article-title: Byzantine stochastic gradient descent publication-title: Proc Advances Neural Inform Process Syst – volume: 18 start-page: 4404 year: 2017 ident: ref8 article-title: Distributed stochastic variance reduced gradient methods by sampling extra data with replacement publication-title: J Mach Learn Res – year: 2018 ident: ref24 article-title: Byzantine-robust distributed learning: Towards optimal statistical rates publication-title: arxiv 1803 01498 – year: 2016 ident: ref6 article-title: Sparknet: Training deep networks in spark publication-title: arXiv 1511 06051 – ident: ref11 doi: 10.1109/TSP.2017.2656847 – volume: 17 start-page: 2657 year: 2016 ident: ref17 article-title: Distributed coordinate descent method for learning with big data publication-title: J Mach Learn Res – ident: ref3 doi: 10.1080/01621459.2018.1429274 – start-page: 177 year: 0 ident: ref7 article-title: Large-scale machine learning with stochastic gradient descent publication-title: Proc Int Conf Comput Statist – volume: 37 start-page: 63 year: 2014 ident: ref2 article-title: Tupleware: Distributed machine learning on small clusters publication-title: IEEE Data Eng Bull – year: 2018 ident: ref26 article-title: Asynchronous byzantine machine learning publication-title: arXiv 1802 07928 – ident: ref30 doi: 10.1145/2046684.2046692 – ident: ref9 doi: 10.1002/9780470515143.ch9 – year: 2018 ident: ref27 article-title: Securing distributed machine learning in high dimensions publication-title: arXiv 1804 10140 – volume: 1 start-page: 74 year: 0 ident: ref4 article-title: Scaling up: Distributed machine learning with cooperation publication-title: Proc Nat Conf Artif Intell – year: 2015 ident: ref35 publication-title: High-dimensional statistics A non-asymptotic viewpoint – volume: 2 year: 2010 ident: ref32 article-title: Mnist handwritten digit database – ident: ref10 doi: 10.1109/TSP.2017.2695451 – ident: ref5 doi: 10.1561/2200000016 – volume: 22 start-page: 1 year: 2006 ident: ref31 article-title: 2006 CSI/FBI computer crime and security survey publication-title: Comput Secur – start-page: 362 year: 0 ident: ref16 article-title: Disco: Distributed optimization for self-concordant empirical loss publication-title: Proc Int Conf Mach Learn – ident: ref20 doi: 10.1145/3219617.3219655 – year: 2018 ident: ref28 article-title: Defending against saddle point attack in byzantine-robust distributed learning publication-title: arXiv 1806 05358 – year: 2018 ident: ref23 article-title: Phocas: Dimensional byzantine-resilient stochastic gradient descent publication-title: arXiv 1805 09682 – ident: ref1 doi: 10.1109/ICASSP.2018.8461691 – ident: ref14 doi: 10.1109/TSP.2015.2434327 – ident: ref15 doi: 10.1109/TSP.2013.2273887 – year: 2017 ident: ref21 article-title: Byzantine-tolerant machine learning publication-title: arXiv 1703 02757 – year: 2014 ident: ref33 article-title: The CIFAR-10 dataset |
| SSID | ssj0014496 |
| Score | 2.597748 |
| Snippet | Due to the growth of modern dataset size and the desire to harness computing power of multiple machines, there is a recent surge of interest in the design of... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 5850 |
| SubjectTerms | Algorithms Byzantine attacker Convergence Datasets distributed gradient descent Ground truth Machine learning Optimization Radio frequency Robustness (mathematics) Servers Signal processing algorithms Sociology Statistics |
| Title | Distributed Gradient Descent Algorithm Robust to an Arbitrary Number of Byzantine Attackers |
| URI | https://ieeexplore.ieee.org/document/8861393 https://www.proquest.com/docview/2310665235 |
| Volume | 67 |
| WOSCitedRecordID | wos000494309800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1941-0476 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014496 issn: 1053-587X databaseCode: RIE dateStart: 19910101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8MwGH7R4UEPfovTKTl4EaxLkzZNjvP7NMQPGHgoafNWBV1l6wT99SZpNwRF8NQekhD6pHne5P14AA4KS5GUsyQwuXMzJgULNGUYJJZ67IJiTGrjxSaSfl8OBup6Do5muTCI6IPP8Ni9el--KfOJuyrrSmnJR_F5mE8SUedqzTwGUeS1uKy5wINYJoOpS5Kq7t3ttYvhUsdMRYI6Ze9vFOQ1VX5sxJ5dLlb-N69VWG6sSNKrYV-DORyuw9K32oLrsOjMyLoK8wY8nLn6uE7aCg25HPkwr4qc1aWcSO_lsRw9V0-v5KbMJuOKVCXRQzt69uyT8knfy4aQsiAnH5_aaUsg6VWVdiEZ4024vzi_O70KGlmFIGcqrAIjY2EyxAipNIWxDGWBDEWhwtxksZJcigRRFSETOQ0zzUwhMKYm44ZzzTTfgtawHOI2kDhElWkqCvuItKIqjzjaPRZlxBmPWBu60y-d5k3NcSd98ZL6swdVqcUmddikDTZtOJz1eKvrbfzRdsNhMWvXwNCGzhTMtPkhx6kzY4Wwp-545_deu7DoxnZphmHcgVY1muAeLOTvFqrRvl9rX62w0Yo |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bT9swFD4ChgQ8MG4THWzzAy-TCHVsx7Efy21MQIW2IlXiIXLikw2JNahNkbZfP9tJK6QhpD0lD7YT5bP9HedcPoCD0lEk5SyNbOHdjGnJIkMZRqmjHjehGFPGBrGJtN9Xw6G-WYDDeS4MIobgMzzyt8GXb6ti6n-VdZVy5KP5IrxJhGC0ydaa-wyECGpczmDgUaLS4cwpSXV38P3GR3HpI6aFpF7b-xkJBVWVf7biwC_nb__vzTZgvbUjSa8BfhMWcLQFa8-qC27BqjckmzrM23B36ivkenErtOTLOAR61eS0KeZEeg8_qvF9_fMX-Vbl00lN6oqYkRs9vw9p-aQfhENIVZLj33-MV5dA0qtr44MyJjtwe342OLmIWmGFqGA6riOrEmlzRIFU2dI6jnJQxrLUcWHzRCuuZIqoy5jJgsa5YbaUmFCbc8u5YYa_g6VRNcJdIEmMOjdUlu4ijKa6EBzdLotKcMYF60B39qWzoq067sUvHrJw-qA6c9hkHpusxaYDn-c9HpuKG6-03fZYzNu1MHRgfwZm1i7JSeYNWSnduTt5_3KvT7ByMbi-yq6-9i_3YNU_xycdxsk-LNXjKX6A5eLJwTb-GObdX7pt1NE |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Distributed+Gradient+Descent+Algorithm+Robust+to+an+Arbitrary+Number+of+Byzantine+Attackers&rft.jtitle=IEEE+transactions+on+signal+processing&rft.au=Cao%2C+Xinyang&rft.au=Lai%2C+Lifeng&rft.date=2019-11-15&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=1053-587X&rft.eissn=1941-0476&rft.volume=67&rft.issue=22&rft.spage=5850&rft_id=info:doi/10.1109%2FTSP.2019.2946020&rft.externalDBID=NO_FULL_TEXT |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1053-587X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1053-587X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1053-587X&client=summon |