Accelerated first-order optimization under nonlinear constraints
We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank–Wolfe or projected gradients, these algorithms avoid optimization over the entire f...
Saved in:
| Published in: | Mathematical programming |
|---|---|
| Main Authors: | , |
| Format: | Journal Article |
| Language: | English |
| Published: |
21.04.2025
|
| ISSN: | 0025-5610, 1436-4646 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank–Wolfe or projected gradients, these algorithms avoid optimization over the entire feasible set at each iteration. We prove convergence to stationary points even in a nonconvex setting and we derive accelerated rates for the convex setting both in continuous time, as well as in discrete time. An important property of these algorithms is that constraints are expressed in terms of velocities instead of positions, which naturally leads to sparse, local and convex approximations of the feasible set (even if the feasible set is nonconvex). Thus, the complexity tends to grow mildly in the number of decision variables and in the number of constraints, which makes the algorithms suitable for machine learning applications. We apply our algorithms to a compressed sensing and a sparse regression problem, showing that we can treat nonconvex $$\ell ^p$$ ℓ p constraints ( $$p<1$$ p < 1 ) efficiently, while recovering state-of-the-art performance for $$p=1$$ p = 1 . |
|---|---|
| AbstractList | We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated first-order algorithms for constrained optimization. Unlike Frank–Wolfe or projected gradients, these algorithms avoid optimization over the entire feasible set at each iteration. We prove convergence to stationary points even in a nonconvex setting and we derive accelerated rates for the convex setting both in continuous time, as well as in discrete time. An important property of these algorithms is that constraints are expressed in terms of velocities instead of positions, which naturally leads to sparse, local and convex approximations of the feasible set (even if the feasible set is nonconvex). Thus, the complexity tends to grow mildly in the number of decision variables and in the number of constraints, which makes the algorithms suitable for machine learning applications. We apply our algorithms to a compressed sensing and a sparse regression problem, showing that we can treat nonconvex $$\ell ^p$$ ℓ p constraints ( $$p<1$$ p < 1 ) efficiently, while recovering state-of-the-art performance for $$p=1$$ p = 1 . |
| Author | Muehlebach, Michael Jordan, Michael I. |
| Author_xml | – sequence: 1 givenname: Michael orcidid: 0000-0002-7764-3069 surname: Muehlebach fullname: Muehlebach, Michael – sequence: 2 givenname: Michael I. surname: Jordan fullname: Jordan, Michael I. |
| BookMark | eNotkMtOwzAQRS1UJNLCD7DKDxhm_EidHVXFS6rEBtaW40wko9SpbLOAryehLOaOdK80j7NmqzhFYuwW4Q4BtvcZAWHLQei5hFAcL1iFSjZcNapZsQqWSDcIV2yd8ycAoDSmYg8772mk5Ar19RBSLnxKPaV6OpVwDD-uhCnWX3Gx5p1jiORS7aeYS3IhlnzNLgc3Zrr57xv28fT4vn_hh7fn1_3uwL1QqszaS9mScUZT69HT0DuHhlyHmtALIg3QdEZvdTdfSbqR0AkltUBnuraXGybOc32ack402FMKR5e-LYJdGNgzAzs_av8YWJS_xZZSSA |
| Cites_doi | 10.1007/978-1-4939-1037-3 10.1007/BF00933293 10.1137/080716542 10.1007/978-3-540-76975-0 10.1007/s11081-016-9328-z 10.1080/02331930600711448 10.1007/978-3-7091-2624-0_1 10.1007/978-3-642-01100-9 10.1007/s10898-016-0493-6 10.1137/21M1410063 10.1007/978-3-540-44479-4 10.1145/1824777.1824783 10.1051/cocv/2010024 10.1016/S0167-6377(02)00231-6 10.1007/s10107-019-01382-3 10.1007/s10107-016-0992-8 10.1142/S0218127408021099 10.1007/978-1-4419-8853-9 10.1109/9.802938 10.1007/s11228-020-00559-9 10.1016/0041-5553(64)90137-5 10.1093/imanum/23.4.539 10.1007/s10107-018-1311-3 10.1137/20M1322716 10.1088/1742-5468/abcaee 10.1023/A:1011253113155 10.1137/16M1133889 10.1007/BFb0120959 10.1515/9781400873173 10.1073/pnas.1614734113 10.1007/s10957-021-01859-2 10.1017/CBO9780511804458.003 10.1007/978-0-387-84858-7 10.1561/2200000024 10.1103/PhysRevLett.133.057401 |
| ContentType | Journal Article |
| DBID | AAYXX CITATION |
| DOI | 10.1007/s10107-025-02224-1 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | CrossRef |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Mathematics |
| EISSN | 1436-4646 |
| ExternalDocumentID | 10_1007_s10107_025_02224_1 |
| GroupedDBID | --Z -~C -~X .4S .86 .DC .VR 06D 0R~ 0VY 199 1N0 203 29M 2J2 2JN 2JY 2KG 2KM 2LR 2~H 30V 4.4 406 408 409 40D 40E 5GY 5VS 67Z 6NX 6TJ 78A 7WY 8FL 8TC 8UJ 8VB 95- 95. 95~ 96X AAAVM AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAPKM AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYQN AAYXX ABAKF ABBBX ABBRH ABBXA ABDBE ABDBF ABDZT ABECU ABFSG ABFTV ABHLI ABHQN ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABRTQ ABSXP ABTEG ABTHY ABTKH ABTMW ABWNU ABXPI ACAOD ACDTI ACGFS ACGOD ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACNCT ACOKC ACOMO ACPIV ACSTC ACZOJ ADHHG ADHIR ADIMF ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEFQL AEGAL AEGNC AEJHL AEJRE AEMSY AENEX AEOHA AEPYU AETLH AEVLU AEXYK AEZWR AFBBN AFDZB AFHIU AFLOW AFOHR AFQWF AFWTZ AFZKB AGAYW AGDGC AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHPBZ AHSBF AHWEU AHYZX AIAKS AIGIU AIIXL AILAN AITGF AIXLP AJRNO AJZVZ AKVCP ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARMRJ ASPBG ATHPR AVWKF AXYYD AYFIA AYJHY AZFZN B-. BA0 BAPOH BGNMA BSONS CITATION CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 EAP EBLON EBR EBS EBU EIOEI EMI ESBYG EST ESX FEDTE FERAY FFXSO FIGPU FNLPD FRRFC FWDCC GGCAI GGRSB GJIRD GNWQR GQ7 GQ8 GXS HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I09 IAO IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ K60 K6~ KDC KOV LAS LLZTM M4Y MA- N9A NB0 NPVJJ NQJWS NU0 O93 O9G O9I O9J OAM P19 P2P P9R PF0 PT4 PT5 QOK QOS QWB R89 R9I RHV RNS ROL RPX RSV S16 S1Z S27 S3B SAP SDD SDH SDM SHX SISQX SJYHP SMT SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 TN5 TSG TSK TSV TUC U2A UG4 UOJIU UTJUX VC2 W23 W48 WH7 WK8 YLTOR Z45 ZL0 ZMTXR ~02 ~EX |
| ID | FETCH-LOGICAL-c244t-c2d339e8a85e9c1cefdaa18eab15e1c2ee5006b8575b610e5630b243521a8b9d3 |
| IEDL.DBID | RSV |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001471774400001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0025-5610 |
| IngestDate | Sat Nov 29 08:01:59 EST 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c244t-c2d339e8a85e9c1cefdaa18eab15e1c2ee5006b8575b610e5630b243521a8b9d3 |
| ORCID | 0000-0002-7764-3069 |
| OpenAccessLink | https://link.springer.com/content/pdf/10.1007/s10107-025-02224-1.pdf |
| ParticipantIDs | crossref_primary_10_1007_s10107_025_02224_1 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-04-21 |
| PublicationDateYYYYMMDD | 2025-04-21 |
| PublicationDate_xml | – month: 04 year: 2025 text: 2025-04-21 day: 21 |
| PublicationDecade | 2020 |
| PublicationTitle | Mathematical programming |
| PublicationYear | 2025 |
| References | T Guanchun (2224_CR15) 2023; 211 Y Nesterov (2224_CR54) 2004 DS Gonçalves (2224_CR36) 2017; 69 R Fletcher (2224_CR42) 1982; 17 2224_CR14 S Bubeck (2224_CR31) 2012; 5 A Beck (2224_CR30) 2003; 31 M Jaggi (2224_CR23) 2013; 28 RI Leine (2224_CR47) 2008 M Muehlebach (2224_CR10) 2021; 22 2224_CR16 D Garber (2224_CR27) 2015; 37 W Su (2224_CR2) 2016; 17 H Attouch (2224_CR32) 2021; 6 AS Nemirovski (2224_CR29) 1983 2224_CR7 2224_CR43 W Krichene (2224_CR5) 2015; 28 M Muehlebach (2224_CR8) 2019; 97 F Alvarez (2224_CR12) 2001; 9 KL Clarkson (2224_CR24) 2010; 6 AR Teel (2224_CR50) 1999; 44 M Muehlebach (2224_CR1) 2022; 23 C Studer (2224_CR18) 2009 J Diakonikolas (2224_CR4) 2021; 31 M-L Vladarean (2224_CR45) 2023; 195 H Attouch (2224_CR11) 2018; 168 G França (2224_CR6) 2020; 2020 AL Dontchev (2224_CR53) 2014 C Glocker (2224_CR17) 2001 C Wang (2224_CR19) 2006; 55 T Hastie (2224_CR55) 2009 N Doikov (2224_CR44) 2022; 32 CW Combettes (2224_CR28) 2020; 119 M Muehlebach (2224_CR9) 2020; 119 M Zhang (2224_CR26) 2020; 108 H Attouch (2224_CR13) 2011; 17 RI Leine (2224_CR49) 2008; 18 A Beck (2224_CR56) 2009; 2 A Wibisono (2224_CR3) 2016; 113 EG Birgin (2224_CR20) 2003; 23 2224_CR37 2224_CR38 J Bolte (2224_CR39) 2018; 28 P Kolev (2224_CR34) 2023; 36 BT Polyak (2224_CR52) 1987 V Bloom (2224_CR22) 2016; 17 DP Bertsekas (2224_CR40) 1977; 23 D Drusvyatskiy (2224_CR41) 2019; 178 2224_CR25 LD Piazza (2224_CR48) 2021; 29 S Schechtman (2224_CR35) 2023; 195 2224_CR21 H Attouch (2224_CR33) 2022; 193 RT Rockafellar (2224_CR51) 1970 BT Polyak (2224_CR46) 1964; 4 |
| References_xml | – volume-title: Implicit Functions and Solution Mappings year: 2014 ident: 2224_CR53 doi: 10.1007/978-1-4939-1037-3 – volume: 28 start-page: 427 issue: 1 year: 2013 ident: 2224_CR23 publication-title: Proc. Mach. Learn. Res. – volume: 23 start-page: 487 issue: 4 year: 1977 ident: 2224_CR40 publication-title: J. Optim. Theory Appl. doi: 10.1007/BF00933293 – volume: 17 start-page: 1 issue: 153 year: 2016 ident: 2224_CR2 publication-title: J. Mach. Learn. Res. – volume: 23 start-page: 1 issue: 256 year: 2022 ident: 2224_CR1 publication-title: J. Mach. Learn. Res. – ident: 2224_CR14 – volume: 2 start-page: 183 issue: 1 year: 2009 ident: 2224_CR56 publication-title: SIAM J. Imag. Sci. doi: 10.1137/080716542 – volume: 28 start-page: 2845 year: 2015 ident: 2224_CR5 publication-title: Adv. Neural Inf. Process. Syst. – volume-title: Stability and Convergence of Mechanical Systems with Unilateral Constraints year: 2008 ident: 2224_CR47 doi: 10.1007/978-3-540-76975-0 – volume: 17 start-page: 651 issue: 4 year: 2016 ident: 2224_CR22 publication-title: Optim. Eng. doi: 10.1007/s11081-016-9328-z – volume: 195 start-page: 3669 year: 2023 ident: 2224_CR45 publication-title: Proc. Mach. Learn. Res. – volume: 37 start-page: 541 year: 2015 ident: 2224_CR27 publication-title: Proc. Mach. Learn. Res. – volume-title: Problem Complexity and Method Efficiency in Optimization year: 1983 ident: 2224_CR29 – volume: 55 start-page: 301 issue: 3 year: 2006 ident: 2224_CR19 publication-title: Optimization doi: 10.1080/02331930600711448 – volume: 119 start-page: 2111 year: 2020 ident: 2224_CR28 publication-title: Proc. Mach. Learn. Res. – ident: 2224_CR16 doi: 10.1007/978-3-7091-2624-0_1 – ident: 2224_CR7 – volume-title: Numerics of Unilateral Contacts and Friction year: 2009 ident: 2224_CR18 doi: 10.1007/978-3-642-01100-9 – volume: 211 start-page: 1373 year: 2023 ident: 2224_CR15 publication-title: Proc. Mach. Learn. Res. – volume: 69 start-page: 525 issue: 3 year: 2017 ident: 2224_CR36 publication-title: J. Glob. Optim. doi: 10.1007/s10898-016-0493-6 – volume: 6 start-page: 1 issue: 1 year: 2021 ident: 2224_CR32 publication-title: Minimax Theory Its Appl. – volume: 32 start-page: 402 issue: 3 year: 2022 ident: 2224_CR44 publication-title: SIAM J. Optim. doi: 10.1137/21M1410063 – volume: 36 start-page: 1 year: 2023 ident: 2224_CR34 publication-title: Adv. Neural Inf. Process. Syst. – volume-title: Set-Valued Force Laws year: 2001 ident: 2224_CR17 doi: 10.1007/978-3-540-44479-4 – volume: 6 start-page: 1 issue: 4 year: 2010 ident: 2224_CR24 publication-title: ACM Trans. Algorithms doi: 10.1145/1824777.1824783 – ident: 2224_CR25 – volume: 17 start-page: 836 issue: 3 year: 2011 ident: 2224_CR13 publication-title: ESAIM Control Optim. Calc. Var. doi: 10.1051/cocv/2010024 – volume: 97 start-page: 4656 year: 2019 ident: 2224_CR8 publication-title: Proc. Mach. Learn. Res. – volume: 31 start-page: 167 issue: 3 year: 2003 ident: 2224_CR30 publication-title: Oper. Res. Lett. doi: 10.1016/S0167-6377(02)00231-6 – ident: 2224_CR43 doi: 10.1007/s10107-019-01382-3 – volume: 108 start-page: 4012 year: 2020 ident: 2224_CR26 publication-title: Proc. Mach. Learn. Res. – volume: 119 start-page: 7088 year: 2020 ident: 2224_CR9 publication-title: Proc. Mach. Learn. Res. – volume: 168 start-page: 123 issue: 1–2 year: 2018 ident: 2224_CR11 publication-title: Math. Program. Ser. B doi: 10.1007/s10107-016-0992-8 – volume: 18 start-page: 1435 issue: 5 year: 2008 ident: 2224_CR49 publication-title: Int. J. Bifurc. Chaos doi: 10.1142/S0218127408021099 – volume-title: Introductory Lectures on Convex Optimization—A Basic Course year: 2004 ident: 2224_CR54 doi: 10.1007/978-1-4419-8853-9 – volume: 44 start-page: 2169 issue: 11 year: 1999 ident: 2224_CR50 publication-title: IEEE Trans. Autom. Control doi: 10.1109/9.802938 – volume: 29 start-page: 361 issue: 2 year: 2021 ident: 2224_CR48 publication-title: Set-Valued Var. Anal. doi: 10.1007/s11228-020-00559-9 – volume: 4 start-page: 1 issue: 5 year: 1964 ident: 2224_CR46 publication-title: USSR Comput. Math. Math. Phys. doi: 10.1016/0041-5553(64)90137-5 – volume: 23 start-page: 539 year: 2003 ident: 2224_CR20 publication-title: IMA J. Numer. Anal. doi: 10.1093/imanum/23.4.539 – volume: 195 start-page: 1228 year: 2023 ident: 2224_CR35 publication-title: Proc. Mach. Learn. Res. – volume: 178 start-page: 503 year: 2019 ident: 2224_CR41 publication-title: Math. Program. doi: 10.1007/s10107-018-1311-3 – volume: 31 start-page: 915 issue: 1 year: 2021 ident: 2224_CR4 publication-title: SIAM J. Optim. doi: 10.1137/20M1322716 – volume: 2020 start-page: 1 issue: 12 year: 2020 ident: 2224_CR6 publication-title: J. Stat. Mech. Theory Exp. doi: 10.1088/1742-5468/abcaee – volume: 22 start-page: 1 issue: 73 year: 2021 ident: 2224_CR10 publication-title: J. Mach. Learn. Res. – volume: 9 start-page: 3 year: 2001 ident: 2224_CR12 publication-title: Set-Valued Var. Anal. doi: 10.1023/A:1011253113155 – ident: 2224_CR38 – volume: 28 start-page: 1867 issue: 2 year: 2018 ident: 2224_CR39 publication-title: SIAM J. Optim. doi: 10.1137/16M1133889 – volume: 17 start-page: 67 year: 1982 ident: 2224_CR42 publication-title: Math. Program. Stud. doi: 10.1007/BFb0120959 – volume-title: Convex Analysis year: 1970 ident: 2224_CR51 doi: 10.1515/9781400873173 – volume: 113 start-page: 7351 issue: 47 year: 2016 ident: 2224_CR3 publication-title: Proc. Natl. Acad. Sci. doi: 10.1073/pnas.1614734113 – volume: 193 start-page: 704 issue: 1–3 year: 2022 ident: 2224_CR33 publication-title: J. Optim. Theory Appl. doi: 10.1007/s10957-021-01859-2 – volume-title: Introduction to Optimization year: 1987 ident: 2224_CR52 – ident: 2224_CR21 doi: 10.1017/CBO9780511804458.003 – volume-title: The Elements of Statistical Learning year: 2009 ident: 2224_CR55 doi: 10.1007/978-0-387-84858-7 – volume: 5 start-page: 1 issue: 1 year: 2012 ident: 2224_CR31 publication-title: Found. Trends Mach. Learn. doi: 10.1561/2200000024 – ident: 2224_CR37 doi: 10.1103/PhysRevLett.133.057401 |
| SSID | ssj0001388 |
| Score | 2.446026 |
| Snippet | We exploit analogies between first-order algorithms for constrained optimization and non-smooth dynamical systems to design a new class of accelerated... |
| SourceID | crossref |
| SourceType | Index Database |
| Title | Accelerated first-order optimization under nonlinear constraints |
| WOSCitedRecordID | wos001471774400001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAVX databaseName: SpringerLINK Contemporary 1997-Present customDbUrl: eissn: 1436-4646 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0001388 issn: 0025-5610 databaseCode: RSV dateStart: 19970101 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV27TsMwFL2qKgYYeCPe8sAGlnAcJ_ZGhagYoEI8qm6R7VxLHSgoDXw_12laOsDQJZKlyI6OlXtOHvccgAtlnEMpUh5K43kaPNVBVBmXHlVqtcxUY6YzfMgHAz0amacOXP37BT82uYn4Oi2JncREODw-64gsiXEFzy_DRdkVUut5PmsUBW2HzN9TLLHQEp30t1a7kG3YbGUj6832eQc6ONmFjSUzQRo9LhxYp3tw0_OeGCUaQZQsjEnj8cZlk31QjXhvmy9Z7CCr2GRml2Er5qNajKER9XQf3vp3r7f3vE1L4J4ouqZjKaVBbbVC44XHUForNFonFAqfICq6w1xM5HQED0ZjMJeQWkqE1c6U8gC6tB4eArO5kxhCrlObp86F-KepsNfGBhoblx3B5Ry94nNmilH82h9HlApCqWhQKsTxSmefwHrSQJzyRJxCt66-8AzW_Hc9nlbnzZb_ABK4o2U |
| linkProvider | Springer Nature |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Accelerated+first-order+optimization+under+nonlinear+constraints&rft.jtitle=Mathematical+programming&rft.au=Muehlebach%2C+Michael&rft.au=Jordan%2C+Michael+I.&rft.date=2025-04-21&rft.issn=0025-5610&rft.eissn=1436-4646&rft_id=info:doi/10.1007%2Fs10107-025-02224-1&rft.externalDBID=n%2Fa&rft.externalDocID=10_1007_s10107_025_02224_1 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0025-5610&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0025-5610&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0025-5610&client=summon |