Convergence analysis of stochastic higher-order majorization-minimization algorithms
Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective valu...
Uloženo v:
| Vydáno v: | Optimization methods & software Ročník 39; číslo 2; s. 384 - 413 |
|---|---|
| Hlavní autoři: | , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
Abingdon
Taylor & Francis
03.03.2024
Taylor & Francis Ltd |
| Témata: | |
| ISSN: | 1055-6788, 1029-4937 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective value decreases. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higher-order upper bound approximations of the finite-sum objective function and minibatching. We derive convergence results for nonconvex and convex optimization problems when the higher-order approximation of the objective function yields an error that is p times differentiable and has Lipschitz continuous p derivative. More precisely, for general nonconvex problems we present asymptotic stationary point guarantees and under Kurdyka-Lojasiewicz property we derive local convergence rates ranging from sublinear to linear. For convex problems with uniformly convex objective function, we derive local (super)linear convergence results for our algorithm. Compared to existing stochastic (first-order) methods, our algorithm adapts to the problem's curvature and allows using any batch size. Preliminary numerical tests support the effectiveness of our algorithmic framework. |
|---|---|
| AbstractList | Majorization–minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective value decreases. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higher-order upper bound approximations of the finite-sum objective function and minibatching. We derive convergence results for nonconvex and convex optimization problems when the higher-order approximation of the objective function yields an error that is p times differentiable and has Lipschitz continuous p derivative. More precisely, for general nonconvex problems we present asymptotic stationary point guarantees and under Kurdyka–Lojasiewicz property we derive local convergence rates ranging from sublinear to linear. For convex problems with uniformly convex objective function, we derive local (super)linear convergence results for our algorithm. Compared to existing stochastic (first-order) methods, our algorithm adapts to the problem's curvature and allows using any batch size. Preliminary numerical tests support the effectiveness of our algorithmic framework. |
| Author | Necoara, Ion Lupu, Daniela |
| Author_xml | – sequence: 1 givenname: Daniela surname: Lupu fullname: Lupu, Daniela organization: University Politehnica Bucharest – sequence: 2 givenname: Ion surname: Necoara fullname: Necoara, Ion email: ion.necoara@upb.ro organization: Gheorghe Mihoc-Caius Iacob Institute of Mathematical Statistics and Applied Mathematics of the Romanian Academy |
| BookMark | eNqFkM1OwzAQhC1UJNrCIyBF4pyyjmMnFRdQxZ9UiUs5W45jN64Su9guqDw9CS0XDnDaXe3MavaboJF1ViF0iWGGoYRrDJSyoixnGWRklmWU5XlxgsYYsnmaz0kxGnpK00F0hiYhbAAgxzkbo9XC2Xfl18pKlQgr2n0wIXE6CdHJRoRoZNKYdaN86nytfNKJjfPmU0TjbNoZa7rjkIh23W9i04VzdKpFG9TFsU7R68P9avGULl8enxd3y1QSUsZUYVlLIpioCpIVWNEMcCmhlgKgpgoTzWrQumalLBhVpMo0JlU9V2yeMwoVmaKrw92td287FSLfuJ3vnwicQEF6Cjkpe9XNQSW9C8ErzaWJ35GjF6blGPiAkf9g5ANGfsTYu-kv99abTvj9v77bg89Y7XwnPpxvax7FvnVee2Gl6UP-feILl4SNTA |
| CitedBy_id | crossref_primary_10_1109_TCI_2022_3230584 crossref_primary_10_1109_TCI_2025_3555137 |
| Cites_doi | 10.1561/2200000016 10.1137/140957639 10.1137/070704277 10.1137/16M1080173 10.1007/s10107-012-0629-5 10.1137/17M1122943 10.1080/10556788.2019.1678033 10.1145/1961189.1961199 10.1137/050644641 10.1093/imanum/dry009 10.1007/s10107-006-0706-8 10.1007/s10107-016-1065-8 10.1007/s10107-019-01449-1 10.1007/s00245-019-09617-7 10.1007/s10957-021-01821-2 10.1007/s10957-020-01653-6 10.1007/s10107-020-01606-x 10.1016/j.jprocont.2010.12.010 10.1080/10556788.2020.1854252 10.1002/0471221317 10.1007/s10107-004-0559-y |
| ContentType | Journal Article |
| Copyright | 2023 Informa UK Limited, trading as Taylor & Francis Group 2023 2023 Informa UK Limited, trading as Taylor & Francis Group |
| Copyright_xml | – notice: 2023 Informa UK Limited, trading as Taylor & Francis Group 2023 – notice: 2023 Informa UK Limited, trading as Taylor & Francis Group |
| DBID | AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| DOI | 10.1080/10556788.2023.2256447 |
| DatabaseName | CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Computer and Information Systems Abstracts Technology Research Database Computer and Information Systems Abstracts – Academic Advanced Technologies Database with Aerospace ProQuest Computer Science Collection Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Computer and Information Systems Abstracts |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1029-4937 |
| EndPage | 413 |
| ExternalDocumentID | 10_1080_10556788_2023_2256447 2256447 |
| Genre | Research Article |
| GrantInformation_xml | – fundername: UEFISCDI grantid: NO Grants 2014-2021 RO-NO-2019-0184, under project ELO-Hyp, contract no. 24/2020 |
| GroupedDBID | .4S .7F .DC .QJ 0BK 0R~ 123 29N 30N 4.4 AAENE AAGDL AAHIA AAJMT AALDU AAMIU AAPUL AAQRR ABCCY ABDBF ABFIM ABHAV ABJNI ABLIJ ABPAQ ABPEM ABTAI ABXUL ABXYU ACGEJ ACGFS ACIWK ACTIO ACUHS ADCVX ADGTB ADXPE AEISY AENEX AEOZL AEPSL AEYOC AFKVX AFRVT AGDLA AGMYJ AHDZW AIJEM AIYEW AJWEG AKBVH AKOOK ALMA_UNASSIGNED_HOLDINGS ALQZU AMVHM AQRUH AQTUD ARCSS AVBZW AWYRJ BLEHA CCCUG CE4 CS3 DGEBU DKSSO DU5 EAP EBS EDO EMK EPL EST ESX E~A E~B F5P GTTXZ H13 HF~ HZ~ H~P I-F IPNFZ J.P KYCEM LJTGL M4Z NA5 NY~ O9- P2P PQQKQ RIG RNANH ROSJB RTWRZ S-T SNACF TASJS TBQAZ TDBHL TEJ TFL TFT TFW TTHFI TUROJ TUS TWF UT5 UU3 ZGOLN ~S~ AAYXX CITATION 7SC 8FD JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c338t-e1cdc3a6ab73271e52018c0dca00d5e13f6d0ffd68c765e3b2f13bd9e694650b3 |
| IEDL.DBID | TFW |
| ISICitedReferencesCount | 1 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001119296800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1055-6788 |
| IngestDate | Wed Aug 13 07:39:13 EDT 2025 Tue Nov 18 22:02:54 EST 2025 Sat Nov 29 02:36:07 EST 2025 Mon Oct 20 23:48:06 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 2 |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c338t-e1cdc3a6ab73271e52018c0dca00d5e13f6d0ffd68c765e3b2f13bd9e694650b3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| PQID | 3073447438 |
| PQPubID | 186278 |
| PageCount | 30 |
| ParticipantIDs | proquest_journals_3073447438 crossref_primary_10_1080_10556788_2023_2256447 crossref_citationtrail_10_1080_10556788_2023_2256447 informaworld_taylorfrancis_310_1080_10556788_2023_2256447 |
| PublicationCentury | 2000 |
| PublicationDate | 2024-03-03 |
| PublicationDateYYYYMMDD | 2024-03-03 |
| PublicationDate_xml | – month: 03 year: 2024 text: 2024-03-03 day: 03 |
| PublicationDecade | 2020 |
| PublicationPlace | Abingdon |
| PublicationPlace_xml | – name: Abingdon |
| PublicationTitle | Optimization methods & software |
| PublicationYear | 2024 |
| Publisher | Taylor & Francis Taylor & Francis Ltd |
| Publisher_xml | – name: Taylor & Francis – name: Taylor & Francis Ltd |
| References | e_1_3_3_30_1 e_1_3_3_18_1 e_1_3_3_17_1 e_1_3_3_19_1 e_1_3_3_13_1 e_1_3_3_16_1 e_1_3_3_35_1 e_1_3_3_15_1 e_1_3_3_36_1 e_1_3_3_34_1 e_1_3_3_12_1 e_1_3_3_31_1 e_1_3_3_11_1 e_1_3_3_32_1 Tripuraneni N. (e_1_3_3_33_1) 2018; 31 Defazio A. (e_1_3_3_10_1) 2014; 27 Moulines E. (e_1_3_3_20_1) 2011; 24 e_1_3_3_7_1 e_1_3_3_6_1 e_1_3_3_9_1 e_1_3_3_8_1 e_1_3_3_29_1 e_1_3_3_28_1 Goodfellow I. (e_1_3_3_14_1) 2016 e_1_3_3_25_1 e_1_3_3_24_1 e_1_3_3_27_1 e_1_3_3_26_1 e_1_3_3_3_1 e_1_3_3_21_1 e_1_3_3_2_1 e_1_3_3_5_1 e_1_3_3_23_1 e_1_3_3_4_1 e_1_3_3_22_1 |
| References_xml | – ident: e_1_3_3_2_1 – volume: 27 start-page: 1646 year: 2014 ident: e_1_3_3_10_1 article-title: SAGA: a fast incremental gradient method with support for non-strongly convex composite objectives publication-title: Adv. Neural. Inf. Process. Syst. – ident: e_1_3_3_30_1 – ident: e_1_3_3_7_1 doi: 10.1561/2200000016 – volume: 24 start-page: 451 year: 2011 ident: e_1_3_3_20_1 article-title: Non-asymptotic analysis of stochastic approximation algorithms for machine learning publication-title: Adv. Neural Inform. Process. Syst. – ident: e_1_3_3_21_1 doi: 10.1137/140957639 – ident: e_1_3_3_28_1 doi: 10.1137/070704277 – volume: 31 start-page: 2899 year: 2018 ident: e_1_3_3_33_1 article-title: Stochastic cubic regularization for fast nonconvex optimization publication-title: Adv. Neural. Inf. Process. Syst. – ident: e_1_3_3_3_1 doi: 10.1137/16M1080173 – ident: e_1_3_3_24_1 doi: 10.1007/s10107-012-0629-5 – ident: e_1_3_3_19_1 doi: 10.1137/17M1122943 – ident: e_1_3_3_8_1 doi: 10.1080/10556788.2019.1678033 – ident: e_1_3_3_13_1 – ident: e_1_3_3_9_1 doi: 10.1145/1961189.1961199 – ident: e_1_3_3_16_1 – ident: e_1_3_3_6_1 doi: 10.1137/050644641 – ident: e_1_3_3_5_1 doi: 10.1093/imanum/dry009 – ident: e_1_3_3_35_1 – ident: e_1_3_3_27_1 – ident: e_1_3_3_29_1 doi: 10.1007/s10107-006-0706-8 – ident: e_1_3_3_4_1 doi: 10.1007/s10107-016-1065-8 – ident: e_1_3_3_18_1 – ident: e_1_3_3_12_1 – ident: e_1_3_3_22_1 doi: 10.1007/s10107-019-01449-1 – ident: e_1_3_3_32_1 doi: 10.1007/s00245-019-09617-7 – ident: e_1_3_3_25_1 doi: 10.1007/s10957-021-01821-2 – ident: e_1_3_3_31_1 doi: 10.1007/s10957-020-01653-6 – ident: e_1_3_3_17_1 – ident: e_1_3_3_11_1 doi: 10.1007/s10107-020-01606-x – ident: e_1_3_3_34_1 – ident: e_1_3_3_26_1 doi: 10.1016/j.jprocont.2010.12.010 – ident: e_1_3_3_23_1 doi: 10.1080/10556788.2020.1854252 – volume-title: Deep Learning year: 2016 ident: e_1_3_3_14_1 – ident: e_1_3_3_15_1 doi: 10.1002/0471221317 – ident: e_1_3_3_36_1 doi: 10.1007/s10107-004-0559-y |
| SSID | ssj0004146 |
| Score | 2.3532667 |
| Snippet | Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and... Majorization–minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and... |
| SourceID | proquest crossref informaworld |
| SourceType | Aggregation Database Enrichment Source Index Database Publisher |
| StartPage | 384 |
| SubjectTerms | Algorithms Approximation Convergence convergence rates Convexity Finite-sum optimization Iterative methods majorization-minimization minibatch Optimization stochastic higher-order algorithms Upper bounds |
| Title | Convergence analysis of stochastic higher-order majorization-minimization algorithms |
| URI | https://www.tandfonline.com/doi/abs/10.1080/10556788.2023.2256447 https://www.proquest.com/docview/3073447438 |
| Volume | 39 |
| WOSCitedRecordID | wos001119296800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAWR databaseName: Taylor & Francis Online Journals customDbUrl: eissn: 1029-4937 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0004146 issn: 1055-6788 databaseCode: TFW dateStart: 19920101 isFulltext: true titleUrlDefault: https://www.tandfonline.com providerName: Taylor & Francis |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELZQxQADb0ShIA-sLkmdOPGIKiqmiqGIbpGfFNQHagK_n7PjQCuEOsAQRZZ1VnRn--6i775D6DrvcZtaxogSkSZJljPCKYsIRNqZlFwoeHyziWw4zMdj_hDQhGWAVboc2tZEEf6udodbyLJBxN34no6QunVd6-8ubEjw6a6eHCJ7B-obDZ6-KyNDfRFIECfS1PD8tsqad1rjLv1xV3sHNNj_h08_QHsh-sS39XY5RFtmfoR2VzgJj9Go73DoviTTYBEoS_DCYogS1UQ4Wmc88eAQ4mk78Uy8LpahmpM4ppJZGGAxfYaZajIrT9Dj4G7Uvyeh9QJRkLNWxMRKKyqYkBntZbFJIU7IVaTBopFOTUwt05G1muUqY6mhsmdjKjU3jCcQ80l6ilrzxdycIZwqCV4Q8iZpokTlisObCSM1U0zELGqjpFF5oQIvuWuPMS3iQF_aKK1wSiuC0tqo-yX2VhNzbBLgq_YsKv9HxNbtSwq6QbbTGL8IZxxE4HaEuYTm539Y-gLtwDDxsDbaQa1q-W4u0bb6qF7K5ZXfzZ-t5vEi |
| linkProvider | Taylor & Francis |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT8MwDI5gIAEH3ojBgB64ZrRLm7ZHhJiGGDsVsVuUpAkD7YG2wu_HSVPYhNAOcKiqynJUOQ_bkf19CF0mrVRHmlIsuZ_jME4oTgn1MUTasRApl_BYsom410v6_XS-F8aUVZocWpdAEfasNpvbXEZXJXFXltQRcrem4f5uwooEpx6vojXDTmcSsKz99N0b6TqMQAUbnaqL57dhFvzTAnrpj9PauqD2zn_8_C7adgGod12umD20osb7aGsOlvAAZTemFN12ZSqPO9QSb6I9CBTlgBtkZ29g60OwRe70Rvx1MnUNndiAlYzch8eHzyApBqPZIXps32Y3HezYF7CEtLXAKpC5JJxyEZNWHKgIQoVE-jlMqp9HKiCa5r7WOU1kTCNFREsHROSpomkIYZ8gR6g2nozVMfIiKcARQuoklB_KRKbwplyJnErKA-rXUVjZnEkHTW4YMoYscAimldGYMRpzRquj5pfaW4nNsUwhnZ9QVthLEV0ymDCyRLdRzT5z2xxU4IAEWUiSkz8MfYE2OtlDl3XvevenaBNEoa1yIw1UK6bv6gyty4_iZTY9t0v7E-XY9UU |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LT8MwDI5gIAQH3ojBgB64ZrRLm7ZHNJhAoGmHIXar8mSgvbQVfj9OmsImhHaAQ1VVlqPKdmI7sj8jdJk0Uh1pSrFgvsRhnFCcEupjiLRjzlMm4LHDJuJ2O-n10o6rJpy5skqTQ-sCKMKe1WZzT6QuK-Ku7ExHSN3qZvR3HQwSfHq8itYgdI6MYXdbz9-tka7BCFiw4SmbeH5bZsE9LYCX_jisrQdq7fzDv--ibRd-eteFveyhFTXaR1tzoIQHqNs0hei2J1N5zGGWeGPtQZgo-szgOnt9Wx2CLW6nN2Rv46lr58QGqmToPjw2eAFK3h_ODtFT67bbvMNu9gIWkLTmWAVCCsIo4zFpxIGKIFBIhC9Bpb6MVEA0lb7WkiYCpK4Ib-iAcJkqmoYQ9HFyhCqj8UgdIy8SHNwgJE5c-aFIRApvyhSXVFAWUL-KwlLkmXDA5GY-xiALHH5pKbTMCC1zQqui-hfbpEDmWMaQzuszy-2ViC7ml2RkCW-tVH7mNjmwwPEItJAkJ39Y-gJtdG5a2eN9--EUbQIltCVupIYq-fRdnaF18ZG_zqbn1rA_AQco8_c |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Convergence+analysis+of+stochastic+higher-order+majorization-minimization+algorithms&rft.jtitle=Optimization+methods+%26+software&rft.au=Lupu%2C+Daniela&rft.au=Necoara%2C+Ion&rft.date=2024-03-03&rft.pub=Taylor+%26+Francis&rft.issn=1055-6788&rft.eissn=1029-4937&rft.volume=39&rft.issue=2&rft.spage=384&rft.epage=413&rft_id=info:doi/10.1080%2F10556788.2023.2256447&rft.externalDocID=2256447 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1055-6788&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1055-6788&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1055-6788&client=summon |