A(DP)^2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy
As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentr...
Gespeichert in:
| Veröffentlicht in: | IEEE transactions on pattern analysis and machine intelligence Jg. 44; H. 11; S. 1 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
01.11.2022
|
| ISSN: | 0162-8828, 1939-3539, 2160-9292, 1939-3539 |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentralized parallel stochastic gradient descent (ADPSGD) algorithm has been proposed and demonstrated to be an efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD framework, or A(DP) 2SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) 2SGD also converges at the optimal O(1/√T) rate as SGD. Empirically, A(DP) 2SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments.As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentralized parallel stochastic gradient descent (ADPSGD) algorithm has been proposed and demonstrated to be an efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD framework, or A(DP) 2SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) 2SGD also converges at the optimal O(1/√T) rate as SGD. Empirically, A(DP) 2SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments. |
|---|---|
| AbstractList | As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentralized parallel stochastic gradient descent (ADPSGD) algorithm has been proposed and demonstrated to be an efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD framework, or A(DP) 2SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) 2SGD also converges at the optimal O(1/√T) rate as SGD. Empirically, A(DP) 2SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments.As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world application scenarios like healthcare, distributed learning can also keep the data local and protect privacy. Recently, the asynchronous decentralized parallel stochastic gradient descent (ADPSGD) algorithm has been proposed and demonstrated to be an efficient and practical strategy where there is no central server, so that each computing node only communicates with its neighbors. Although no raw data will be transmitted across different local nodes, there is still a risk of information leak during the communication process for malicious participants to make attacks. In this paper, we present a differentially private version of asynchronous decentralized parallel SGD framework, or A(DP) 2SGD for short, which maintains communication efficiency of ADPSGD and prevents the inference from malicious participants. Specifically, Rényi differential privacy is used to provide tighter privacy analysis for our composite Gaussian mechanisms while the convergence rate is consistent with the non-private version. Theoretical analysis shows A(DP) 2SGD also converges at the optimal O(1/√T) rate as SGD. Empirically, A(DP) 2SGD achieves comparable model accuracy as the differentially private version of Synchronous SGD (SSGD) but runs much faster than SSGD in heterogeneous computing environments. |
| Author | Xu, Jie Zhang, Wei Wang, Fei |
| Author_xml | – sequence: 1 givenname: Jie surname: Xu fullname: Xu, Jie – sequence: 2 givenname: Wei surname: Zhang fullname: Zhang, Wei – sequence: 3 givenname: Fei surname: Wang fullname: Wang, Fei |
| BookMark | eNp9kE9PwjAYhxuDiYB-AU874mHYP2u3eltAkQTjEjjbdF0XasqG7dDgp7cTTh489W3y_N68v2cEBk3baABuEZwiBPn9pshfllMMMZoSBNOUswswxIjBmGOOB2AIEcNxluHsCoy8f4cQJRSSIdD5ZF7cveH1Yv4Q5f7YqK1rm_bgo7lWuumctOZbV1Ehw2S1jdZdq7bSd0ZFCycrE5iA-p6Nvky3jeamrrULXyNtVDjzKdXxGlzW0np9c37HYPP0uJk9x6vXxXKWr2KFM9jFnPCUVjRRWSkzqnCdVhynrNSlSkpUIk4orZAmZaqx5iRUUgHGtU4ZY5KQMZic1u5d-3HQvhM7Ey6zVjY6VBKYMgYJThgNaHZClWu9d7oWynSyM21f2ViBoOjFil-xohcrzmJDFP-J7p3ZSXf8L_QDYuV9xA |
| CitedBy_id | crossref_primary_10_1145_3650028 crossref_primary_10_1016_j_automatica_2024_111514 crossref_primary_10_1109_TPAMI_2023_3332428 crossref_primary_10_1155_2021_6679453 |
| Cites_doi | 10.1145/2810103.2813687 10.1109/CVPR.2018.00716 10.21437/Interspeech.2019-2700 10.1109/CSF.2017.11 10.1109/CVPR.2018.00474 10.1109/CVPR.2018.00745 10.1109/TNNLS.2019.2944481 10.1007/978-3-030-23502-4_10 10.1109/ICASSP.2019.8682888 10.1109/ICASSP40776.2020.9054065 10.29012/jpc.v4i1.612 10.1109/CVPR.2016.90 10.1109/CVPR.2017.634 10.4324/9781410605337-29 10.1007/s00287-019-01205-x 10.1007/11761679_29 10.1109/TII.2019.2942179 10.1145/2976749.2978318 10.29012/jpc.723 |
| ContentType | Journal Article |
| DBID | AAYXX CITATION 7X8 |
| DOI | 10.1109/TPAMI.2021.3107796 |
| DatabaseName | CrossRef MEDLINE - Academic |
| DatabaseTitle | CrossRef MEDLINE - Academic |
| DatabaseTitleList | MEDLINE - Academic |
| Database_xml | – sequence: 1 dbid: 7X8 name: MEDLINE - Academic url: https://search.proquest.com/medline sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Computer Science |
| EISSN | 2160-9292 1939-3539 |
| EndPage | 1 |
| ExternalDocumentID | 10_1109_TPAMI_2021_3107796 |
| GroupedDBID | --- -DZ -~X .DC 0R~ 29I 4.4 53G 5GY 5VS 6IK 97E 9M8 AAJGR AASAJ AAWTH AAYXX ABFSI ABQJQ ABVLG ACGFO ACGFS ACIWK ACNCT ADRHT AENEX AETEA AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CITATION CS3 DU5 E.L EBS EJD F5P FA8 HZ~ H~9 IBMZZ ICLAB IEDLZ IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNI RNS RXW RZB TAE TN5 UHB VH1 ~02 7X8 ABAZT |
| ID | FETCH-LOGICAL-c280t-93975d54c8ba85c2f7d9276bebc4b1b19355d1e3b7e2e93162c54c2fe7666a33 |
| ISICitedReferencesCount | 22 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000864325900055&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0162-8828 1939-3539 |
| IngestDate | Sun Sep 28 09:54:08 EDT 2025 Tue Nov 18 22:43:54 EST 2025 Sat Nov 29 05:16:01 EST 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 11 |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c280t-93975d54c8ba85c2f7d9276bebc4b1b19355d1e3b7e2e93162c54c2fe7666a33 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23 |
| PQID | 2566032465 |
| PQPubID | 23479 |
| PageCount | 1 |
| ParticipantIDs | proquest_miscellaneous_2566032465 crossref_citationtrail_10_1109_TPAMI_2021_3107796 crossref_primary_10_1109_TPAMI_2021_3107796 |
| PublicationCentury | 2000 |
| PublicationDate | 2022-11-01 |
| PublicationDateYYYYMMDD | 2022-11-01 |
| PublicationDate_xml | – month: 11 year: 2022 text: 2022-11-01 day: 01 |
| PublicationDecade | 2020 |
| PublicationTitle | IEEE transactions on pattern analysis and machine intelligence |
| PublicationYear | 2022 |
| References | ref13 Roy (ref7) 2019 ref12 ref15 Lian (ref11) Cheng (ref20) 2018 ref36 ref31 Liu (ref34) Zhang (ref2) Li (ref25) 2019 ref32 Smith (ref3) ref16 ref38 ref19 Bellet (ref22) 2017 ref18 Akiba (ref1) Lalitha (ref9) 2019 Goyal (ref28) 2017 Bellet (ref23) 2017 Tan (ref35) 2019 ref24 ref45 ref26 ref42 ref41 ref44 ref21 ref43 Chen (ref29) Lalitha (ref8) 2019 Chaudhuri (ref14) 2013; 14 Zhang (ref30) Zinkevich (ref10) Konevcny (ref4) 2016 ref5 Wang (ref27) 2019 ref40 McMahan (ref17) 2017 Shayan (ref6) 2018 Howard (ref37) 2017 Krizhevsky (ref33) 2009 Simonyan (ref39) |
| References_xml | – ident: ref19 doi: 10.1145/2810103.2813687 – ident: ref41 doi: 10.1109/CVPR.2018.00716 – ident: ref45 doi: 10.21437/Interspeech.2019-2700 – ident: ref39 article-title: Very deep convolutional networks for large-scale image recognition – year: 2017 ident: ref28 article-title: Accurate, large minibatch SGD: Training imagenet in 1 hour – year: 2017 ident: ref37 article-title: Mobilenets: Efficient convolutional neural networks for mobile vision applications – year: 2019 ident: ref7 article-title: Braintorrent: A peer-to-peer environment for decentralized federated learning – start-page: 2350 volume-title: Proc. 25th Int. Joint Conf. Artif. Intell. ident: ref2 article-title: Staleness-aware async-SGD for distributed deep learning – start-page: 4424 volume-title: Proc. 31st Conf. Neural Inf. Process. Syst. ident: ref3 article-title: Federated multi-task learning – start-page: 3043 volume-title: in Proc. 35th Int. Conf. Mach. Learn. ident: ref11 article-title: Asynchronous decentralized parallel stochastic gradient descent – ident: ref18 doi: 10.1109/CSF.2017.11 – ident: ref38 doi: 10.1109/CVPR.2018.00474 – ident: ref43 doi: 10.1109/CVPR.2018.00745 – ident: ref5 doi: 10.1109/TNNLS.2019.2944481 – ident: ref1 article-title: ChainerMN: Scalable distributed deep learning framework – year: 2018 ident: ref6 article-title: Biscotti: A ledger for private and secure peer-to-peer machine learning – ident: ref21 doi: 10.1007/978-3-030-23502-4_10 – ident: ref44 doi: 10.1109/ICASSP.2019.8682888 – ident: ref31 article-title: NCCL: Optimized primitives for collective multi-GPU communication. – start-page: 171 volume-title: Proc. IEEE Int. Conf. Data Mining ident: ref30 article-title: Model accuracy and runtime tradeoff in distributed deep learning: A systematic study – volume: 14 start-page: 2905 issue: 1 year: 2013 ident: ref14 article-title: A near-optimal algorithm for differentially-private principal components publication-title: J. Mach. Learn. Res. – year: 2019 ident: ref25 article-title: Asynchronous federated learning with differential privacy for edge intelligence – year: 2016 ident: ref4 article-title: Federated learning: Strategies for improving communication efficiency – ident: ref32 doi: 10.1109/ICASSP40776.2020.9054065 – year: 2018 ident: ref20 article-title: LEASGD: An efficient and privacy-preserving decentralized algorithm for distributed learning – ident: ref15 doi: 10.29012/jpc.v4i1.612 – ident: ref29 article-title: Revisiting distributed synchronous SGD – ident: ref34 article-title: Train CIFAR10 with PyTorch – year: 2019 ident: ref9 article-title: Peer-to-peer federated learning on graphs – year: 2019 ident: ref27 article-title: Efficient privacy-preserving nonconvex optimization – ident: ref40 doi: 10.1109/CVPR.2016.90 – ident: ref36 doi: 10.1109/CVPR.2017.634 – year: 2017 ident: ref17 article-title: Learning differentially private recurrent language models – year: 2017 ident: ref22 article-title: Fast and differentially private algorithms for decentralized collaborative machine learning – ident: ref42 doi: 10.4324/9781410605337-29 – year: 2009 ident: ref33 article-title: Learning multiple layers of features from tiny images – year: 2017 ident: ref23 article-title: Personalized and private peer-to-peer machine learning – ident: ref12 doi: 10.1007/s00287-019-01205-x – ident: ref13 doi: 10.1007/11761679_29 – year: 2019 ident: ref8 article-title: Decentralized Bayesian learning over graphs – ident: ref24 doi: 10.1109/TII.2019.2942179 – year: 2019 ident: ref35 article-title: EfficientNet: Rethinking model scaling for convolutional neural networks – ident: ref16 doi: 10.1145/2976749.2978318 – start-page: 2595 volume-title: Proc. 24th Annu. Conf. Neural Inf. Process. Syst. ident: ref10 article-title: Parallelized stochastic gradient descent – ident: ref26 doi: 10.29012/jpc.723 |
| SSID | ssj0014503 |
| Score | 2.5588422 |
| Snippet | As deep learning models are usually massive and complex, distributed learning is essential for increasing training efficiency. Moreover, in many real-world... |
| SourceID | proquest crossref |
| SourceType | Aggregation Database Enrichment Source Index Database |
| StartPage | 1 |
| Title | A(DP)^2SGD: Asynchronous Decentralized Parallel Stochastic Gradient Descent with Differential Privacy |
| URI | https://www.proquest.com/docview/2566032465 |
| Volume | 44 |
| WOSCitedRecordID | wos000864325900055&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 2160-9292 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014503 issn: 0162-8828 databaseCode: RIE dateStart: 19790101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV1Lb9QwELa2hQM98CiglkdlJJBAVUrsxLHDbUXawoEqEiuxJ6LYcUSkkq7SdFX4I_xdxo6TzUJB9MAlWo0ca-P5NJ4Zj79B6LkUsQq4L7yyFIUHBo94AvY1r9Cw94WUwFvKNpvgJydiPo_TyeRHfxdmecrrWlxexov_qmqQgbLN1dlrqHuYFATwG5QOT1A7PP9J8VPwGZMUIvwX7JB-PE5s6u_8W60MDa4peE20q8isvoO3meaNaadizqfP1Jfc0DbvHze2EKyFoZbsqcvWJq6XSmuS7GlTLXO1diZs4kbTcqLvP24PIhaWv9NUPDvyE5Oo_2orOLXlquj5QHvNzy8ssKpBMKS0P-lqlf7vREdO5NIWEPGSIW3hMpkRmGLhboZrK6Mk8j3w1-jIopLR1kyutvmWMnWWTj-8h3ifkgPwWDmPryDY_mXjG8oRbSDkx5mdIzNzZG6ODXSDchaL7mLgcDwVMttye_iG_jaWH7_-_X-sezzrG771YmZ30W0XfuBpB5t7aKLrbXSnb-2BnaXfRlsjnsr7SE9fJumrzwZNb_AYS3gNS7jHEl5hCfdYwg5L2GAJj7GEHZYeoNnR4eztO8_15_AUFX7rxeDLsoKFSshcMEVLXsSUR1JLFUoiiaHuL4gOJNdUxwEsloLBtNQcYuY8CB6izfqs1jsIizwC01DIgkdlyGkhWSGlH4J1YSVELGwXkX4JM-W4600LldPsz6rbRfvDO4uOueWvo5_1msnAwJpTs7zWsJAZxASRD2FHxB5da8bH6NYK90_QZttc6Kfoplq21Xmzhzb4XOxZVP0Es7eaoQ |
| linkProvider | IEEE |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=A%28DP%29%5E2SGD%3A+Asynchronous+Decentralized+Parallel+Stochastic+Gradient+Descent+with+Differential+Privacy&rft.jtitle=IEEE+transactions+on+pattern+analysis+and+machine+intelligence&rft.au=Xu%2C+Jie&rft.au=Zhang%2C+Wei&rft.au=Wang%2C+Fei&rft.date=2022-11-01&rft.issn=0162-8828&rft.eissn=2160-9292&rft.spage=1&rft.epage=1&rft_id=info:doi/10.1109%2FTPAMI.2021.3107796&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_TPAMI_2021_3107796 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0162-8828&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0162-8828&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0162-8828&client=summon |