The Limitation and Practical Acceleration of Stochastic Gradient Algorithms in Inverse Problems
In this work we investigate the practicability of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems, such as space-varying image deblurring. Such algorithms have been shown in machine learning literature to have optimal comple...
Uložené v:
| Vydané v: | Proceedings of the ... IEEE International Conference on Acoustics, Speech and Signal Processing (1998) s. 7680 - 7684 |
|---|---|
| Hlavní autori: | , , |
| Médium: | Konferenčný príspevok.. |
| Jazyk: | English |
| Vydavateľské údaje: |
IEEE
01.05.2019
|
| Predmet: | |
| ISSN: | 2379-190X |
| On-line prístup: | Získať plný text |
| Tagy: |
Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
|
| Abstract | In this work we investigate the practicability of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems, such as space-varying image deblurring. Such algorithms have been shown in machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the full gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated full gradient method (FISTA), even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism to characterize whether a given inverse problem should be preferred to be solved by stochastic optimization technique with a known sampling pattern. Furthermore, to overcome another key bottleneck of stochastic optimization which is the heavy computation of proximal operators while maintaining fast convergence, we propose an accelerated primal-dual SGD algorithm and demonstrate the effectiveness of our approach in image deblurring experiments. |
|---|---|
| AbstractList | In this work we investigate the practicability of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging inverse problems, such as space-varying image deblurring. Such algorithms have been shown in machine learning literature to have optimal complexities in theory, and provide great improvement empirically over the full gradient methods. Surprisingly, in some tasks such as image deblurring, many of such methods fail to converge faster than the accelerated full gradient method (FISTA), even in terms of epoch counts. We investigate this phenomenon and propose a theory-inspired mechanism to characterize whether a given inverse problem should be preferred to be solved by stochastic optimization technique with a known sampling pattern. Furthermore, to overcome another key bottleneck of stochastic optimization which is the heavy computation of proximal operators while maintaining fast convergence, we propose an accelerated primal-dual SGD algorithm and demonstrate the effectiveness of our approach in image deblurring experiments. |
| Author | Egiazarian, Karen Tang, Junqi Davies, Mike |
| Author_xml | – sequence: 1 givenname: Junqi surname: Tang fullname: Tang, Junqi organization: School of Engineering, University of Edinburgh, UK – sequence: 2 givenname: Karen surname: Egiazarian fullname: Egiazarian, Karen organization: Noiseless Imaging Ltd, Finland – sequence: 3 givenname: Mike surname: Davies fullname: Davies, Mike organization: School of Engineering, University of Edinburgh, UK |
| BookMark | eNotkNFqwjAYRrOxwdT5BN7kBer-JK1JLkU2FQoTdLA7SdK_M6NNRxIGe_sV9OpcHDh8fFPyEIaAhCwYLBkD_bLfrI_Hw5ID00u1UkKs1B2Za6lYKbVWTDB2TyZcSF0wDZ9PZJrSNwAoWaoJOZ8uSGvf-2yyHwI1oaGHaFz2znR07Rx2GK9qaOkxD-5i0ijpNprGY8h03X0N0edLn6gPdB9-MSYcG4PtsE_P5LE1XcL5jTPy8fZ62uyK-n07Lq8LzznkAtWIqrLSgmg1WHCl0FqXlktsGa-kggZXaJvKCQnOgdXGGmQcoILSODEji2vXI-L5J_rexL_z7Q_xD-61V2U |
| ContentType | Conference Proceeding |
| DBID | 6IE 6IH CBEJK RIE RIO |
| DOI | 10.1109/ICASSP.2019.8683368 |
| DatabaseName | IEEE Electronic Library (IEL) Conference Proceedings IEEE Proceedings Order Plan (POP) 1998-present by volume IEEE Xplore All Conference Proceedings IEEE Electronic Library (IEL) IEEE Proceedings Order Plans (POP) 1998-present |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering |
| EISBN | 9781479981311 1479981311 |
| EISSN | 2379-190X |
| EndPage | 7684 |
| ExternalDocumentID | 8683368 |
| Genre | orig-research |
| GroupedDBID | 23M 29P 6IE 6IF 6IH 6IK 6IL 6IM 6IN AAJGR AAWTH ABLEC ACGFS ADZIZ ALMA_UNASSIGNED_HOLDINGS BEFXN BFFAM BGNUA BKEBE BPEOZ CBEJK CHZPO IEGSK IJVOP IPLJI M43 OCL RIE RIL RIO RNS |
| ID | FETCH-LOGICAL-i220t-e822055b7b03f90b0c439994b27ef125780de6ebd5c370cc0b9abae1200504ac3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 5 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000482554007182&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| IngestDate | Wed Aug 27 02:33:28 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | false |
| IsScholarly | true |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-i220t-e822055b7b03f90b0c439994b27ef125780de6ebd5c370cc0b9abae1200504ac3 |
| OpenAccessLink | https://www.pure.ed.ac.uk/ws/files/206650698/icassp2019.pdf |
| PageCount | 5 |
| ParticipantIDs | ieee_primary_8683368 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-May |
| PublicationDateYYYYMMDD | 2019-05-01 |
| PublicationDate_xml | – month: 05 year: 2019 text: 2019-May |
| PublicationDecade | 2010 |
| PublicationTitle | Proceedings of the ... IEEE International Conference on Acoustics, Speech and Signal Processing (1998) |
| PublicationTitleAbbrev | ICASSP |
| PublicationYear | 2019 |
| Publisher | IEEE |
| Publisher_xml | – name: IEEE |
| SSID | ssj0008748 |
| Score | 2.1535468 |
| Snippet | In this work we investigate the practicability of stochastic gradient descent and recently introduced variants with variance-reduction techniques in imaging... |
| SourceID | ieee |
| SourceType | Publisher |
| StartPage | 7680 |
| SubjectTerms | Acceleration Convergence Gradient methods Image Processing Inverse problems Machine learning Stochastic Optimization Task analysis |
| Title | The Limitation and Practical Acceleration of Stochastic Gradient Algorithms in Inverse Problems |
| URI | https://ieeexplore.ieee.org/document/8683368 |
| WOSCitedRecordID | wos000482554007182&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB7a4kEvPlrxTQ4eXZt9dJMcS_F1KYUq9FbymLULuiu7W3-_SbZUBS_eQiAMzCT5yGS-bwCuVZgJk6Q0cH9UQcKNsPegDAOmUY0kUpfp980m2HTKFwsx68DNlguDiL74DG_d0P_lm1KvXapsyFMexynvQpextOVqbW9dzhK-URUKqRg-Tcbz-cyVbtm90C771T_Fw8f9_v8MH8Dgm4dHZluEOYQOFkew90NCsA9LG2fiWUrexUQWhrQaRNb5ZKy1hZU2yKTMyLwp9Uo6aWbyUPlir4aM317LKm9W7zXJC-J0N6ram3WNZuoBvNzfPU8eg03ThCCPItoEyB11dqSYonEmqKLavThEoiKGWegOKDWYojIjHTOqNVVCKomhyy7RROr4GHpFWeAJEKlSo5EKpZgFcowFZlmWaEYxiaSFt1PoO1ctP1pdjOXGS2d_T5_DrotGWyx4Ab2mWuMl7OjPJq-rKx_ML5IGoiw |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1NS8NAEB1qFdSLH6347R48Grv5aDZ7LMXaYi2FVugt7G4mtqCJJKm_3-wmVAUv3kIgDMxk97Gz894DuJV2zCPPp5a-o7K8IOLlPihsiymUXYFUd_qN2QSbTILFgk8bcLfhwiCiGT7De_1o7vKjVK11q6wT-IHr-sEWbGvnrJqttdl3A-YFta6QTXln1O_NZlM9vFX-DdWHvxxUDIAMDv4X-hDa30w8Mt1gzBE0MDmG_R8igi0Iy0oTw1MySSYiiUilQlSmn_SUKoGlKjNJYzIrUrUUWpyZPGZm3KsgvbfXNFsVy_ecrBKilTey3ITVVjN5G14GD_P-0KptE6yV49DCwkCTZ7uSSerGnEqq9JmDe9JhGNt6idIIfZRRV7mMKkUlF1KgrftL1BPKPYFmkiZ4CkRIP1JIuZSshHJ0OcZx7ClG0XNECXBn0NKpCj8qZYywztL5369vYHc4fx6H49Hk6QL2dGWq0cFLaBbZGq9gR30Wqzy7NoX9AjEzpXU |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Abook&rft.genre=proceeding&rft.title=Proceedings+of+the+...+IEEE+International+Conference+on+Acoustics%2C+Speech+and+Signal+Processing+%281998%29&rft.atitle=The+Limitation+and+Practical+Acceleration+of+Stochastic+Gradient+Algorithms+in+Inverse+Problems&rft.au=Tang%2C+Junqi&rft.au=Egiazarian%2C+Karen&rft.au=Davies%2C+Mike&rft.date=2019-05-01&rft.pub=IEEE&rft.eissn=2379-190X&rft.spage=7680&rft.epage=7684&rft_id=info:doi/10.1109%2FICASSP.2019.8683368&rft.externalDocID=8683368 |