Step Size Adaptation for Accelerated Stochastic Momentum Algorithm Using SDE Modeling and Lyapunov Drift Minimization
Training machine learning models often involves solving high-dimensional stochastic optimization problems, where stochastic gradient-based algorithms are hindered by slow convergence. Although momentum-based methods perform well in deterministic settings, their effectiveness diminishes under gradien...
Saved in:
| Published in: | IEEE transactions on signal processing Vol. 73; pp. 3124 - 3139 |
|---|---|
| Main Authors: | , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
New York
IEEE
2025
The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Subjects: | |
| ISSN: | 1053-587X, 1941-0476 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Training machine learning models often involves solving high-dimensional stochastic optimization problems, where stochastic gradient-based algorithms are hindered by slow convergence. Although momentum-based methods perform well in deterministic settings, their effectiveness diminishes under gradient noise. In this paper, we introduce a novel accelerated stochastic momentum algorithm. Specifically, we first model the trajectory of discrete-time momentum-based algorithms using continuous-time stochastic differential equations (SDEs). By leveraging a tailored Lyapunov function, we derive 2-D adaptive step sizes through Lyapunov drift minimization, which significantly enhance both convergence speed and noise stability. The proposed algorithm not only accelerates convergence but also eliminates the need for hyperparameter fine-tuning, consistently achieving robust accuracy in machine learning tasks. |
|---|---|
| AbstractList | Training machine learning models often involves solving high-dimensional stochastic optimization problems, where stochastic gradient-based algorithms are hindered by slow convergence. Although momentum-based methods perform well in deterministic settings, their effectiveness diminishes under gradient noise. In this paper, we introduce a novel accelerated stochastic momentum algorithm. Specifically, we first model the trajectory of discrete-time momentum-based algorithms using continuous-time stochastic differential equations (SDEs). By leveraging a tailored Lyapunov function, we derive 2-D adaptive step sizes through Lyapunov drift minimization, which significantly enhance both convergence speed and noise stability. The proposed algorithm not only accelerates convergence but also eliminates the need for hyperparameter fine-tuning, consistently achieving robust accuracy in machine learning tasks. |
| Author | Yuan, Yulan Tsang, Danny H. K. Lau, Vincent K. N. |
| Author_xml | – sequence: 1 givenname: Yulan orcidid: 0009-0004-6817-0310 surname: Yuan fullname: Yuan, Yulan email: yyuan202@connect.hkust-gz.edu.cn organization: Internet of Things Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China – sequence: 2 givenname: Danny H. K. orcidid: 0000-0003-0135-7098 surname: Tsang fullname: Tsang, Danny H. K. email: eetsang@ece.ust.hk organization: Internet of Things Thrust, The Hong Kong University of Science and Technology (Guangzhou), Guangzhou, Guangdong, China – sequence: 3 givenname: Vincent K. N. orcidid: 0000-0001-7769-6008 surname: Lau fullname: Lau, Vincent K. N. email: eeknlau@ece.ust.hk organization: Department of Electronic and Computer Engineering, The Hong Kong University of Science and Technology, Clear Water Bay, Hong Kong SAR, China |
| BookMark | eNpFkMtLAzEQxoNUsD7uHjwEPG_NazfJsbS-oKKwFbwtaTZrU3aTNZsV9K83tYKnmWF-33zDdwomzjsDwCVGM4yRvFmXLzOCSD6juSQFF0dgiiXDGWK8mKQe5TTLBX87AafDsEMIMyaLKRjLaHpY2m8D57Xqo4rWO9j4AOdam9YEFU0Ny-j1Vg3RavjkO-Pi2MF5--6DjdsOvg7WvcNyeZuWtWn3g3I1XH2pfnT-Ey6DbSJ8ss529vvX4BwcN6odzMVfPQPru9v14iFbPd8_LuarTBPGY6aZrBnapE8xo0VdI6wUU4IZKrVAXDC8aQhVikgicqJzLoTWFJmNEkLKmp6B68PZPviP0Qyx2vkxuORY0WSQ55IjnCh0oHTwwxBMU_XBdip8VRhV-2yrlG21z7b6yzZJrg4Sa4z5xxNbcMLoD-X9d94 |
| CODEN | ITPRED |
| Cites_doi | 10.1007/978-1-84628-696-4_13 10.1007/b98874 10.1007/978-1-4757-3863-6 10.18653/v1/2020.emnlp-main.634 10.4135/9781412983907.n1717 10.1109/JIOT.2022.3230452 10.1109/ICCA51439.2020.9264458 10.1109/JIOT.2024.3376821 10.1109/JIOT.2023.3340014 10.1007/s10107-021-01681-8 10.1016/0370-1573(76)90029-6 10.1109/TNNLS.2021.3056947 10.1016/0041-5553(64)90137-5 10.1109/TSP.2022.3186526 10.1109/TSP.2020.2981904 10.1109/ECC.2015.7330562 |
| ContentType | Journal Article |
| Copyright | Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| Copyright_xml | – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2025 |
| DBID | 97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D |
| DOI | 10.1109/TSP.2025.3592678 |
| DatabaseName | IEEE Xplore (IEEE) IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional |
| DatabaseTitle | CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional |
| DatabaseTitleList | Technology Research Database |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISSN | 1941-0476 |
| EndPage | 3139 |
| ExternalDocumentID | 10_1109_TSP_2025_3592678 11096724 |
| Genre | orig-research |
| GrantInformation_xml | – fundername: Research Grants Council (RGC), Hong Kong grantid: 16207221 – fundername: Guangzhou Municipal Science and Technology Project grantid: 2023A03J0011 funderid: 10.13039/501100010256 – fundername: Guangzhou Municipal Key Laboratory on Future Networked Systems grantid: 024A03J0623 – fundername: Guangdong Provincial Project grantid: 2021JC02X149 – fundername: Guangdong Provincial Key Laboratory of Integrated Communications, Sensing and Computation for Ubiquitous Internet of Things grantid: 2023B1212010007 – fundername: Research Grants Council (RGC), Hong Kong, under the Areas of Excellence Scheme grantid: AoE/E-601/22-R |
| GroupedDBID | -~X .DC 0R~ 29I 3EH 4.4 53G 5GY 5VS 6IK 85S 97E AAJGR AASAJ AAWTH ABFSI ABQJQ ABVLG ACGFO ACIWK ACKIV ACNCT AENEX AETIX AGQYO AGSQL AHBIQ AI. AIBXA AJQPL AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 E.L EBS EJD F5P HZ~ H~9 ICLAB IFIPE IFJZH IPLJI JAVBF LAI MS~ O9- OCL P2P RIA RIE RNS TAE TN5 VH1 AAYXX CITATION 7SC 7SP 8FD ABAZT JQ2 L7M L~C L~D |
| ID | FETCH-LOGICAL-c247t-c49d40b4491436dd01aa4a84e39c807841bf23aa292852c5788cc30eba8899d3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 0 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001566919900003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1053-587X |
| IngestDate | Sat Nov 01 15:11:50 EDT 2025 Sat Nov 29 07:34:48 EST 2025 Mon Dec 08 03:37:12 EST 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html https://doi.org/10.15223/policy-029 https://doi.org/10.15223/policy-037 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c247t-c49d40b4491436dd01aa4a84e39c807841bf23aa292852c5788cc30eba8899d3 |
| Notes | ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14 |
| ORCID | 0009-0004-6817-0310 0000-0003-0135-7098 0000-0001-7769-6008 |
| PQID | 3247559701 |
| PQPubID | 85478 |
| PageCount | 16 |
| ParticipantIDs | proquest_journals_3247559701 crossref_primary_10_1109_TSP_2025_3592678 ieee_primary_11096724 |
| PublicationCentury | 2000 |
| PublicationDate | 20250000 2025-00-00 20250101 |
| PublicationDateYYYYMMDD | 2025-01-01 |
| PublicationDate_xml | – year: 2025 text: 20250000 |
| PublicationDecade | 2020 |
| PublicationPlace | New York |
| PublicationPlace_xml | – name: New York |
| PublicationTitle | IEEE transactions on signal processing |
| PublicationTitleAbbrev | TSP |
| PublicationYear | 2025 |
| Publisher | IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| Publisher_xml | – name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE) |
| References | Orvieto (ref6) 2020 ref12 ref15 Li (ref18) 2019; 20 ref31 ref11 Malladi (ref20) 2022; 35 ref33 ref32 LeCun (ref35) ref1 Li (ref19) 2021 Guo (ref14) 2021 ref17 Itô (ref24) 1951 Su (ref16) 2016; 17 Kingma (ref7) 2014 Li (ref2) 2019 Sebbouh (ref3) 2021 Peng (ref10) 2020 Li (ref22) 2017 Keskar (ref8) 2016 ref23 ref26 Zhang (ref13) 2022 ref25 Smith (ref29) 2020 Reddi (ref34) 2019 ref21 ref28 Donsker (ref30) 1951 ref9 ref4 ref5 Ziyin (ref27) 2021 |
| References_xml | – start-page: 9058 volume-title: Proc. Int. Conf. Mach. Learn. year: 2020 ident: ref29 article-title: On the generalization benefit of noise in stochastic gradient descent – ident: ref23 doi: 10.1007/978-1-84628-696-4_13 – ident: ref12 doi: 10.1007/b98874 – year: 2022 ident: ref13 article-title: DRSOM: A dimension reduced second-order method – ident: ref28 doi: 10.1007/978-1-4757-3863-6 – ident: ref11 doi: 10.18653/v1/2020.emnlp-main.634 – ident: ref33 doi: 10.4135/9781412983907.n1717 – year: 1951 ident: ref30 article-title: An invariance principle for certain probability limit theorems – ident: ref9 doi: 10.1109/JIOT.2022.3230452 – volume: 35 start-page: 7697 volume-title: Proc. Adv. Neural Inf. Process. Syst. year: 2022 ident: ref20 article-title: On the SDEs and scaling rules for adaptive gradient algorithms – year: 2020 ident: ref10 article-title: How to fine-tune deep neural networks in few-shot learning? – ident: ref15 doi: 10.1109/ICCA51439.2020.9264458 – ident: ref25 doi: 10.1109/JIOT.2024.3376821 – year: 2021 ident: ref19 article-title: On the validity of modeling SGD with stochastic differential equations (SDEs) – ident: ref26 doi: 10.1109/JIOT.2023.3340014 – start-page: 983 volume-title: Proc. 22nd Int. Nat. Conf. Artif. Intell. Statist. year: 2019 ident: ref2 article-title: On the convergence of stochastic gradient descent with adaptive stepsizes – start-page: 3935 volume-title: Proc. Conf. Learn. Theory year: 2021 ident: ref3 article-title: Almost sure convergence rates for stochastic gradient descent and stochastic heavy ball – ident: ref17 doi: 10.1007/s10107-021-01681-8 – ident: ref21 doi: 10.1016/0370-1573(76)90029-6 – start-page: 2101 volume-title: Proc. Int. Conf. Mach. Learn. year: 2017 ident: ref22 article-title: Stochastic modified equations and adaptive stochastic gradient algorithms – ident: ref31 doi: 10.1109/TNNLS.2021.3056947 – year: 2019 ident: ref34 article-title: On the convergence of Adam and beyond – year: 2016 ident: ref8 article-title: On large-batch training for deep learning: Generalization gap and sharp minima – volume-title: On Stochastic Differential Equations year: 1951 ident: ref24 – start-page: 356 volume-title: Proc. Uncertainty Artif. Intell. year: 2020 ident: ref6 article-title: The role of memory in stochastic optimization – year: 2021 ident: ref14 article-title: A novel convergence analysis for algorithms of the Adam family and beyond – year: 2014 ident: ref7 article-title: Adam: A method for stochastic optimization – ident: ref4 doi: 10.1016/0041-5553(64)90137-5 – ident: ref32 doi: 10.1109/TSP.2022.3186526 – ident: ref1 doi: 10.1109/TSP.2020.2981904 – ident: ref5 doi: 10.1109/ECC.2015.7330562 – year: 2021 ident: ref27 article-title: Strength of minibatch noise in SGD – volume: 17 start-page: 1 issue: 153 year: 2016 ident: ref16 article-title: A differential equation for modeling Nesterov’s accelerated gradient method: Theory and insights publication-title: J. Mach. Learn. Res. – volume: 20 start-page: 1 issue: 40 year: 2019 ident: ref18 article-title: Stochastic modified equations and dynamics of stochastic gradient algorithms I: Mathematical foundations publication-title: J. Mach. Learn. Res. – ident: ref35 article-title: The MNIST database of handwritten digits |
| SSID | ssj0014496 |
| Score | 2.4671195 |
| Snippet | Training machine learning models often involves solving high-dimensional stochastic optimization problems, where stochastic gradient-based algorithms are... |
| SourceID | proquest crossref ieee |
| SourceType | Aggregation Database Index Database Publisher |
| StartPage | 3124 |
| SubjectTerms | Algorithms Approximation algorithms Cognitive tasks Convergence Differential equations Heuristic algorithms Liapunov functions Lyapunov drift Machine learning Mathematical models Momentum Noise Optimization Signal processing algorithms Stability analysis stochastic differential equation Stochastic momentum Stochastic processes Trajectory |
| Title | Step Size Adaptation for Accelerated Stochastic Momentum Algorithm Using SDE Modeling and Lyapunov Drift Minimization |
| URI | https://ieeexplore.ieee.org/document/11096724 https://www.proquest.com/docview/3247559701 |
| Volume | 73 |
| WOSCitedRecordID | wos001566919900003&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1941-0476 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014496 issn: 1053-587X databaseCode: RIE dateStart: 19910101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV09T8MwELUAMcDAN6J8yQMLQ9rUduJ4rKCIAapK7dAtcmwHItGkShMk-PX4nBQqIQa2DEkU3cW-d7679xC6gRyAcUm9lAnuMZlyT9gw5wUsTVWScGlM5MQm-GgUzWZi3A6ru1kYY4xrPjNduHS1fF2oGo7KesCOGXLCNtEm52EzrPVdMmDMiXFZvEC9IOKzVU3SF73pZGwzQRJ0aSBICIpqazHIiar82oldeHnY_-eHHaC9FkfiQeP4Q7Rh8iO0u8YueIxq6ODCk-zT4IGWi6bmji1IxQOlbLQBkgiNJ1WhXiWwNeNnYGOo6jkevL0UZVa9zrHrKMCT-yEG0TQYXccy1_jpQy7qvHjH92WWVvg5y7N5O9B5gqYPw-ndo9eqLHiKMF55ignN_MSaz0KnUGu_LyWTETNUKCCjZ_0kJVRKIkgUEGVXeKQU9U0iI5uraXqKtvIiN2cIE6m0TzS1WZKxW4HFojQEMRMFnDJBqDvodmX2eNFwacQuB_FFbF0Ug4vi1kUddAJm_rmvtXAHXa4cFberbRlbUMghM_L75388doF24O3N2ckl2qrK2lyhbfVeZcvy2v1IX1MhxiA |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LT-MwEB7tskgLB_YFojwWH_bCIZDaThwfKx5itW2F1B56ixzbgUg0qUqCBL8ej5OySIgDtxwSJZqJPd94Zr4P4A_mAFwoFuRcioCrXATShbkg4nmus0woaxMvNiHG42Q2k9fdsLqfhbHW-uYze4KXvpZvKt3gUdkpsmPGgvLP8CXinIbtuNZL0YBzL8flEAMLokTMVlXJUJ5OJ9cuF6TRCYskjVFT7VUU8rIqb_ZiH2Auv33w077DVockyaB1_Q_4ZMufsPmKX_AXNNjDRSbFkyUDoxZt1Z04mEoGWrt4gzQRhkzqSt8q5GsmI-RjqJs5GdzdVMuivp0T31NAJucXBGXTcHidqNKQ4aNaNGX1QM6XRV6TUVEW826kcxumlxfTs6ug01kINOWiDjSXhoeZM58DT7ExYV8prhJumdRIR8_7WU6ZUlTSJKLarfFEaxbaTCUuWzNsB9bKqrS7QKjSJqSGuTzJus3AoVEWo5yJRlaZKDY9OF6ZPV20bBqpz0JCmToXpeiitHNRD7bRzP_v6yzcg4OVo9Juvd2nDhYKzI3C_t47jx3B16vpaJgO_47_7cMGvqk9STmAtXrZ2ENY1w91cb_87X-qZ8yuyWc |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Step+Size+Adaptation+for+Accelerated+Stochastic+Momentum+Algorithm+Using+SDE+Modeling+and+Lyapunov+Drift+Minimization&rft.jtitle=IEEE+transactions+on+signal+processing&rft.au=Yuan%2C+Yulan&rft.au=Tsang%2C+Danny+H.+K.&rft.au=Lau%2C+Vincent+K.+N.&rft.date=2025&rft.pub=IEEE&rft.issn=1053-587X&rft.volume=73&rft.spage=3124&rft.epage=3139&rft_id=info:doi/10.1109%2FTSP.2025.3592678&rft.externalDocID=11096724 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1053-587X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1053-587X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1053-587X&client=summon |