RES: Regularized Stochastic BFGS Algorithm
RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate...
Uložené v:
| Vydané v: | arXiv.org |
|---|---|
| Hlavní autori: | , |
| Médium: | Paper |
| Jazyk: | English |
| Vydavateľské údaje: |
Ithaca
Cornell University Library, arXiv.org
29.01.2014
|
| Predmet: | |
| ISSN: | 2331-8422 |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments can be prohibitive in high dimensional problems. Application of second order methods, on the other hand, is impracticable because computation of objective function Hessian inverses incurs excessive computational cost. BFGS modifies gradient descent by introducing a Hessian approximation matrix computed from finite gradient differences. RES utilizes stochastic gradients in lieu of deterministic gradients for both, the determination of descent directions and the approximation of the objective function's curvature. Since stochastic gradients can be computed at manageable computational cost RES is realizable and retains the convergence rate advantages of its deterministic counterparts. Convergence results show that lower and upper bounds on the Hessian egeinvalues of the sample functions are sufficient to guarantee convergence to optimal arguments. Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS. An application of RES to the implementation of support vector machines is developed. |
|---|---|
| AbstractList | RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with stochastic objectives. The use of stochastic gradient descent algorithms is widespread, but the number of iterations required to approximate optimal arguments can be prohibitive in high dimensional problems. Application of second order methods, on the other hand, is impracticable because computation of objective function Hessian inverses incurs excessive computational cost. BFGS modifies gradient descent by introducing a Hessian approximation matrix computed from finite gradient differences. RES utilizes stochastic gradients in lieu of deterministic gradients for both, the determination of descent directions and the approximation of the objective function's curvature. Since stochastic gradients can be computed at manageable computational cost RES is realizable and retains the convergence rate advantages of its deterministic counterparts. Convergence results show that lower and upper bounds on the Hessian egeinvalues of the sample functions are sufficient to guarantee convergence to optimal arguments. Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS. An application of RES to the implementation of support vector machines is developed. |
| Author | Mokhtari, Aryan Ribeiro, Alejandro |
| Author_xml | – sequence: 1 givenname: Aryan surname: Mokhtari fullname: Mokhtari, Aryan – sequence: 2 givenname: Alejandro surname: Ribeiro fullname: Ribeiro, Alejandro |
| BookMark | eNotjkFLwzAYQIMoOOfuHgvehNbkS77ki7c5tikMhHX3kTbp1lFbbVoRf70DPb3be--GXbZdGxi7EzxThMgfXf9df2VCcZEZDXjBJiClSEkBXLNZjCfOOWgDiHLCHrbL_CnZhsPYuL7-CT7Jh648ujjUZfK8WufJvDl0fT0c32_ZVeWaGGb_nLLdarlbvKSbt_XrYr5JHQpIyZAiGwpbKJDGAzoLhfReWGMDVVqLQCWSA1MqZY0XqqhAlxWhx4Dn0ym7_9N-9N3nGOKwP3Vj356Le-AEhrThIH8B3nNCqA |
| ContentType | Paper |
| Copyright | 2014. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| Copyright_xml | – notice: 2014. This work is published under http://arxiv.org/licenses/nonexclusive-distrib/1.0/ (the “License”). Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License. |
| DBID | 8FE 8FG ABJCF ABUWG AFKRA AZQEC BENPR BGLVJ CCPQU DWQXO HCIFZ L6V M7S PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| DOI | 10.48550/arxiv.1401.7625 |
| DatabaseName | ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central (Alumni) ProQuest Central UK/Ireland ProQuest Central Essentials - QC ProQuest Central ProQuest Technology Collection ProQuest One ProQuest Central Korea SciTech Premium Collection ProQuest Engineering Collection Engineering Database Proquest Central Premium ProQuest One Academic (New) Publicly Available Content Database ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) ProQuest One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection |
| DatabaseTitle | Publicly Available Content Database Engineering Database Technology Collection ProQuest One Academic Middle East (New) ProQuest Central Essentials ProQuest One Academic Eastern Edition ProQuest Central (Alumni Edition) SciTech Premium Collection ProQuest One Community College ProQuest Technology Collection ProQuest SciTech Collection ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest One Academic UKI Edition ProQuest Central Korea Materials Science & Engineering Collection ProQuest Central (New) ProQuest One Academic ProQuest One Academic (New) Engineering Collection |
| DatabaseTitleList | Publicly Available Content Database |
| Database_xml | – sequence: 1 dbid: PIMPY name: ProQuest Publicly Available Content Database url: http://search.proquest.com/publiccontent sourceTypes: Aggregation Database |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Physics |
| EISSN | 2331-8422 |
| Genre | Working Paper/Pre-Print |
| GroupedDBID | 8FE 8FG ABJCF ABUWG AFKRA ALMA_UNASSIGNED_HOLDINGS AZQEC BENPR BGLVJ CCPQU DWQXO FRJ HCIFZ L6V M7S M~E PHGZM PHGZT PIMPY PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS |
| ID | FETCH-LOGICAL-a512-878489eb9b4237d25a92b3dd1979e8f661e8c58a27c4497d14bf26cf85d5e5233 |
| IEDL.DBID | BENPR |
| IngestDate | Mon Jun 30 09:36:01 EDT 2025 |
| IsDoiOpenAccess | true |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | false |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-a512-878489eb9b4237d25a92b3dd1979e8f661e8c58a27c4497d14bf26cf85d5e5233 |
| Notes | SourceType-Working Papers-1 ObjectType-Working Paper/Pre-Print-1 content type line 50 |
| OpenAccessLink | https://www.proquest.com/docview/2082786702?pq-origsite=%requestingapplication% |
| PQID | 2082786702 |
| PQPubID | 2050157 |
| ParticipantIDs | proquest_journals_2082786702 |
| PublicationCentury | 2000 |
| PublicationDate | 20140129 |
| PublicationDateYYYYMMDD | 2014-01-29 |
| PublicationDate_xml | – month: 01 year: 2014 text: 20140129 day: 29 |
| PublicationDecade | 2010 |
| PublicationPlace | Ithaca |
| PublicationPlace_xml | – name: Ithaca |
| PublicationTitle | arXiv.org |
| PublicationYear | 2014 |
| Publisher | Cornell University Library, arXiv.org |
| Publisher_xml | – name: Cornell University Library, arXiv.org |
| SSID | ssj0002672553 |
| Score | 1.5275116 |
| SecondaryResourceType | preprint |
| Snippet | RES, a regularized stochastic version of the Broyden-Fletcher-Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve convex optimization problems with... |
| SourceID | proquest |
| SourceType | Aggregation Database |
| SubjectTerms | Algorithms Approximation Computational efficiency Computational geometry Convergence Convexity Curvature Economic models Mathematical analysis Nonlinear programming Optimization Quasi Newton methods Support vector machines Upper bounds |
| Title | RES: Regularized Stochastic BFGS Algorithm |
| URI | https://www.proquest.com/docview/2082786702 |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV09T8MwELWgBYmJb_FRUAYmJJfEcWIfC6KoBQaqqO1QpsrxB60EbUlChfj12CGFAYmFMbIU2cnp3b3z0z2EzjRnxgdJsS1tOXazMDEHwbGAFIwfQsQVLc0mWLfLh0NIqoZbXskql5hYArWaSdcjtySdE8Zj5pOr-St2rlHudrWy0FhFdTepzMZ5vdXuJr3vLguJma2Zw6_7yXJ414XI3ieLpuMVTQsE0S8MLhNLZ_O_W9pC9UTMdbaNVvR0B62Xgk6Z76Jz-2UvvV7pNJ9NPrTy-sVMjoUby-y1Ord97_r5yb6sGL_soUGnPbi5w5UvAhY2PVv84pSDTiF1khZFIgEkDZUKgIHmxiZczWXEBWGSUmAqoKkhsTQ8UpG2vDPcR7XpbKoPkBcERqnYcUQDVOhQhEbEvhHgG8srA3GIGsuTj6rYzkc_xz76e_kYbdjywolcMIEGqhXZmz5Ba3JRTPLstPpVp05t2bdPyf1D8vgJX7eezw |
| linkProvider | ProQuest |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1JTwIxFH5B0OjJPS6oc9CLyeBMZ2tNjHFDCEuIcNDTpDNtgUQBAXH5T_5HXwfQg4k3Dp6bNO3r277263sAh5IGymKxa2JqS01dC9OkjFOTs4gpy2EeFW7SbCKoVun9Paul4HP6F0bTKqc-MXHUohvrO3IE6ZQE1A8sct57NnXXKP26Om2hMVaLknx_Rcg2OCte4_keEZK_aVwVzElXAZNjcEPrpy5lMmKRJoQI4nFGIkcImwVMUoXhStLYo5wEseuyQNhupIgfK-oJTyJqc3DaOci4qOtWGjK1YqX28H2pQ_wAU3Rn_Bya1Ao74f239iinYUwO_Y73y-UncSy__M8ksII75z3ZX4WU7KzBQkJXjQfrcIx6c2rcyaYm0bY_pDDqw27c4rrotHGZv60bF49NXPuw9bQBjVmsbhPSnW5HboFh20oIXyNgxVwuHe4o7luKM0sharb5NmSngg4nljsIf6S88_fwASwWGpVyWC5WS7uwhImUpvOYhGUhPey_yD2Yj0fD9qC_P9ESA8IZn8oXXUv2XA |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=RES%3A+Regularized+Stochastic+BFGS+Algorithm&rft.jtitle=arXiv.org&rft.au=Mokhtari%2C+Aryan&rft.au=Ribeiro%2C+Alejandro&rft.date=2014-01-29&rft.pub=Cornell+University+Library%2C+arXiv.org&rft.eissn=2331-8422&rft_id=info:doi/10.48550%2Farxiv.1401.7625 |