Asynchronous Parallel Nonconvex Optimization Under the Polyak-Łojasiewicz Condition

Communication delays and synchronization are major bottlenecks for parallel computing, and tolerating asynchrony is therefore crucial for accelerating parallel computation. Motivated by optimization problems that do not satisfy convexity assumptions, we present an asynchronous block coordinate desce...

Full description

Saved in:
Bibliographic Details
Published in:IEEE control systems letters Vol. 6; pp. 524 - 529
Main Authors: Yazdani, Kasra, Hale, Matthew
Format: Journal Article
Language:English
Published: IEEE 2022
Subjects:
asynchronous optimization algorithms
Convergence
Delays
Linear programming
Machine learning algorithms
multi-agent systems
nonconvex optimization
Optimization
Parallel computation
Program processors
Signal processing algorithms
ISSN:2475-1456, 2475-1456
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Communication delays and synchronization are major bottlenecks for parallel computing, and tolerating asynchrony is therefore crucial for accelerating parallel computation. Motivated by optimization problems that do not satisfy convexity assumptions, we present an asynchronous block coordinate descent algorithm for nonconvex optimization problems whose objective functions satisfy the Polyak-Łojasiewicz condition. This condition is a generalization of strong convexity to nonconvex problems and requires neither convexity nor uniqueness of minimizers. Under only assumptions of mild smoothness of objective functions and bounded delays, we prove that a linear convergence rate is obtained. Numerical experiments for logistic regression problems are presented to illustrate the impact of asynchrony upon convergence.
AbstractList Communication delays and synchronization are major bottlenecks for parallel computing, and tolerating asynchrony is therefore crucial for accelerating parallel computation. Motivated by optimization problems that do not satisfy convexity assumptions, we present an asynchronous block coordinate descent algorithm for nonconvex optimization problems whose objective functions satisfy the Polyak-Łojasiewicz condition. This condition is a generalization of strong convexity to nonconvex problems and requires neither convexity nor uniqueness of minimizers. Under only assumptions of mild smoothness of objective functions and bounded delays, we prove that a linear convergence rate is obtained. Numerical experiments for logistic regression problems are presented to illustrate the impact of asynchrony upon convergence.
Author Yazdani, Kasra
Hale, Matthew
Author_xml – sequence: 1
  givenname: Kasra
  orcidid: 0000-0001-9533-8909
  surname: Yazdani
  fullname: Yazdani, Kasra
  email: kasra.yazdani@ufl.edu
  organization: Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL, USA
– sequence: 2
  givenname: Matthew
  orcidid: 0000-0003-3991-1680
  surname: Hale
  fullname: Hale, Matthew
  email: matthewhale@ufl.edu
  organization: Department of Mechanical and Aerospace Engineering, University of Florida, Gainesville, FL, USA
BookMark eNp9kMtOwzAQRS1UJErpD8DGP5BiO07iLKuIlxTRSm0XrCLXnqguqV3Z4dHu-Dj-C_oQQixYzSzuuaM556hjnQWELikZUEry67KYPE0GjDA6iIlggpAT1GU8SyLKk7Tzaz9D_RCWhBAqWEZY3kXTYdhYtfDOupeAx9LLpoEGPzqrnH2Fdzxat2ZltrI1zuKZ1eBxuwA8ds1GPkefH24pg4E3o7a4cFabXe4CndayCdA_zh6a3d5Mi_uoHN09FMMyUizN2ojViQCe1zoWbC5yBSzlma4zRjnQWkg5Z4onRIuYqCSDhDKd7sMcBE81xD3EDr3KuxA81NXam5X0m4qSaqem2qupdmqqo5pvSPyBlGn377VemuZ_9OqAGgD4uZXzWKQ0j78AnsR17Q
CODEN ICSLBO
CitedBy_id crossref_primary_10_1016_j_ifacol_2022_07_269
crossref_primary_10_1109_TAC_2023_3327319
crossref_primary_10_1016_j_ifacol_2023_10_116
crossref_primary_10_1109_TAC_2024_3520677
Cites_doi 10.1109/TAC.1971.1099815
10.1109/TAC.2020.3033490
10.1109/TIT.2015.2399924
10.1137/15M1024950
10.1016/0005-1098(71)90059-8
10.1109/TSP.2019.2937282
10.1137/0801036
10.1007/978-3-319-46128-1_50
10.1007/978-3-030-40344-7
ContentType Journal Article
DBID 97E
RIA
RIE
AAYXX
CITATION
DOI 10.1109/LCSYS.2021.3082800
DatabaseName IEEE Xplore (IEEE)
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE Electronic Library (IEL)
CrossRef
DatabaseTitle CrossRef
DatabaseTitleList
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
EISSN 2475-1456
EndPage 529
ExternalDocumentID 10_1109_LCSYS_2021_3082800
9438619
Genre orig-research
GrantInformation_xml – fundername: Air Force Office of Scientific Research (AFOSR)
  grantid: FA9550-19-1-0169
  funderid: 10.13039/100000181
– fundername: Task Order Contract with the Air Force Research Laboratory, Munitions Directorate, at Eglin AFB
  funderid: 10.13039/100006602
– fundername: Office of Naval Research (ONR)
  grantid: N00014-19-1-2543
GroupedDBID 0R~
6IK
97E
AAJGR
AASAJ
AAWTH
ABAZT
ABJNI
ABQJQ
ABVLG
ACGFS
AGQYO
AHBIQ
AKJIK
ALMA_UNASSIGNED_HOLDINGS
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
EBS
EJD
IFIPE
IPLJI
JAVBF
OCL
RIA
RIE
AAYXX
CITATION
ID FETCH-LOGICAL-c267t-2f58e49fd382b89ce2647df7214e1f8aab2c450d830c57e512d6fd3824e846de3
IEDL.DBID RIE
ISICitedReferencesCount 5
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000668835800005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2475-1456
IngestDate Sat Nov 29 06:10:52 EST 2025
Tue Nov 18 22:15:18 EST 2025
Wed Aug 27 02:26:42 EDT 2025
IsPeerReviewed true
IsScholarly true
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c267t-2f58e49fd382b89ce2647df7214e1f8aab2c450d830c57e512d6fd3824e846de3
ORCID 0000-0001-9533-8909
0000-0003-3991-1680
PageCount 6
ParticipantIDs crossref_citationtrail_10_1109_LCSYS_2021_3082800
ieee_primary_9438619
crossref_primary_10_1109_LCSYS_2021_3082800
PublicationCentury 2000
PublicationDate 20220000
2022-00-00
PublicationDateYYYYMMDD 2022-01-01
PublicationDate_xml – year: 2022
  text: 20220000
PublicationDecade 2020
PublicationTitle IEEE control systems letters
PublicationTitleAbbrev LCSYS
PublicationYear 2022
Publisher IEEE
Publisher_xml – name: IEEE
References stich (ref9) 2018
ref15
ref14
ref20
ref10
åström (ref4) 1971; 7
polyak (ref5) 1963; 3
ref17
zhou (ref18) 2017
bertsekas (ref11) 1989; 23
ref16
lian (ref1) 2015
tang (ref12) 2018; 80
ref3
ref6
yi (ref7) 2020
yu (ref13) 2019
bonawitz (ref2) 2019; 1
bhojanapalli (ref19) 2016
haddadpour (ref8) 2019; 32
References_xml – ident: ref3
  doi: 10.1109/TAC.1971.1099815
– ident: ref17
  doi: 10.1109/TAC.2020.3033490
– ident: ref14
  doi: 10.1109/TIT.2015.2399924
– year: 2017
  ident: ref18
  publication-title: Characterization of gradient dominance and regularity conditions for neural networks
– year: 2019
  ident: ref13
  publication-title: On the computation and communication complexity of parallel SGD with dynamic batch sizes for stochastic non-convex optimization
– ident: ref10
  doi: 10.1137/15M1024950
– volume: 7
  start-page: 123
  year: 1971
  ident: ref4
  article-title: System identification-A survey
  publication-title: Automatica
  doi: 10.1016/0005-1098(71)90059-8
– volume: 32
  year: 2019
  ident: ref8
  article-title: Local SGD with periodic averaging: Tighter analysis and adaptive synchronization
  publication-title: Advances in neural information processing systems
– year: 2020
  ident: ref7
  publication-title: A Primal-Dual Algorithm for Distributed Optimization
– start-page: 2737
  year: 2015
  ident: ref1
  article-title: Asynchronous parallel stochastic gradient for nonconvex optimization
  publication-title: Advances in neural information processing systems
– ident: ref15
  doi: 10.1109/TSP.2019.2937282
– ident: ref16
  doi: 10.1137/0801036
– start-page: 3873
  year: 2016
  ident: ref19
  article-title: Global optimality of local search for low rank matrix recovery
  publication-title: Advances in neural information processing systems
– volume: 80
  start-page: 4848
  year: 2018
  ident: ref12
  article-title: $d^{2}$ : Decentralized training over decentralized data
  publication-title: Proc 35th Int Conf Mach Learn
– ident: ref6
  doi: 10.1007/978-3-319-46128-1_50
– ident: ref20
  doi: 10.1007/978-3-030-40344-7
– volume: 1
  start-page: 374
  year: 2019
  ident: ref2
  article-title: Towards federated learning at scale: System design
  publication-title: Proceedings of Machine Learning and Systems
– year: 2018
  ident: ref9
  publication-title: Local SGD converges fast and communicates little
– volume: 23
  year: 1989
  ident: ref11
  publication-title: Parallel and Distributed Computation Numerical Methods
– volume: 3
  start-page: 643
  year: 1963
  ident: ref5
  article-title: Gradient methods for minimizing functionals
  publication-title: Zhurnal Vychislitel'noi Matematiki i Matematicheskoi Fiziki
SSID ssj0001827029
Score 2.2417285
Snippet Communication delays and synchronization are major bottlenecks for parallel computing, and tolerating asynchrony is therefore crucial for accelerating parallel...
SourceID crossref
ieee
SourceType Enrichment Source
Index Database
Publisher
StartPage 524
SubjectTerms asynchronous optimization algorithms
Convergence
Delays
Linear programming
Machine learning algorithms
multi-agent systems
nonconvex optimization
Optimization
Parallel computation
Program processors
Signal processing algorithms
Title Asynchronous Parallel Nonconvex Optimization Under the Polyak-Łojasiewicz Condition
URI https://ieeexplore.ieee.org/document/9438619
Volume 6
WOSCitedRecordID wos000668835800005&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE Electronic Library (IEL)
  customDbUrl:
  eissn: 2475-1456
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0001827029
  issn: 2475-1456
  databaseCode: RIE
  dateStart: 20170101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8MwGA5zePDiB1OcX-TgTbu1WZomxzEcHsYcbMI8lTR5C9PZyj7UefPH-b9Msm7uIIK3Ut5AeZrk_Uie50XoEkISSRMoeJwm3KNMEi9JlVnuKjUOxadSSifi2om6XT4cil4JXa-5MADgLp9BzT66s3ydq7ktldUFbXBmNT63oogtuVo_9RRumVVixYvxRb3T6j_0TQZIgprVZOGWxLbhezaaqThf0t7731fso90iZsTN5U8-QCXIKmjQnC4yZaVtTe6Oe3Jim6KMcTfP3EXyd3xnNoPngmWJXXsjbII93MvHC_nkfX3mj3I6greR-sCt3B5cG7tDdN--GbRuvaJFgqcIi2YeSUMOVKS6wUnChQIT30Q6NWkdhSDlUiZE0dDXvOGrMALj3TVzxhRM4KGhcYTKWZ7BMcJhSEJgKuBmblGmIDGpjZKapFozA66oomAFXqwK_XDbxmIcuzzCF7EDPLaAxwXgVXS1HvOyVM_407pi0V5bFkCf_P76FO0QS0Vw5ZAzVJ5N5nCOttXrbDSdXLjJ8Q3Xvrrp
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1dS8MwFA1jCvriB1Ocn3nwTbu1Wdqmj2M4JtY52IT5VNLkFqazlX2o880f5_8yybq5BxF8K-UWymnSe2-Scw5C5-ASn6tCwWI0Zhb1OLHiRKjpLhKVUGzKOTcirqHfbrN-P-gU0OWSCwMA5vAZVPSl2cuXmZjqpbJqQGvM0xqfa9o5K2dr_ayoMM2tChbMGDuoho3uQ1f1gMSpaFUWpmlsK9lnxU7FZJPm9v_eYwdt5VUjrs8_8y4qQFpCvfp4lgotbqu6d9zhI22LMsTtLDVHyd_xnfodPOc8S2wMjrAq93AnG874k_X1mT3y8QDeBuIDNzK9da3i9tB986rXaFm5SYIliOdPLJK4DGiQyBojMQsEqArHl4lq7Cg4CeM8JoK6tmQ1W7g-qPwuPRNMQZUeEmr7qJhmKRwg7LrEBU84TI0u6gmIVXMjuCSJlJ4CNygjZwFeJHIFcW1kMYxMJ2EHkQE80oBHOeBldLF85mWun_FndEmjvYzMgT78_fYZ2mj1bsMovG7fHKFNookJZnHkGBUnoymcoHXxOhmMR6dmoHwDvpa-Mg
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Asynchronous+Parallel+Nonconvex+Optimization+Under+the+Polyak-%C5%81ojasiewicz+Condition&rft.jtitle=IEEE+control+systems+letters&rft.au=Yazdani%2C+Kasra&rft.au=Hale%2C+Matthew&rft.date=2022&rft.issn=2475-1456&rft.eissn=2475-1456&rft.volume=6&rft.spage=524&rft.epage=529&rft_id=info:doi/10.1109%2FLCSYS.2021.3082800&rft.externalDBID=n%2Fa&rft.externalDocID=10_1109_LCSYS_2021_3082800
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2475-1456&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2475-1456&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2475-1456&client=summon