Convergence and Stability of Iteratively Re-weighted Least Squares Algorithms
In this paper, we study the theoretical properties of iteratively re-weighted least squares (IRLS) algorithms and their utility in sparse signal recovery in the presence of noise. We demonstrate a one-to-one correspondence between the IRLS algorithms and a class of Expectation-Maximization (EM) algo...
Uloženo v:
| Vydáno v: | IEEE transactions on signal processing Ročník 62; číslo 1; s. 183 - 195 |
|---|---|
| Hlavní autoři: | , , , |
| Médium: | Journal Article |
| Jazyk: | angličtina |
| Vydáno: |
New York, NY
IEEE
01.01.2014
Institute of Electrical and Electronics Engineers |
| Témata: | |
| ISSN: | 1053-587X, 1941-0476 |
| On-line přístup: | Získat plný text |
| Tagy: |
Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
|
| Abstract | In this paper, we study the theoretical properties of iteratively re-weighted least squares (IRLS) algorithms and their utility in sparse signal recovery in the presence of noise. We demonstrate a one-to-one correspondence between the IRLS algorithms and a class of Expectation-Maximization (EM) algorithms for constrained maximum likelihood estimation under a Gaussian scale mixture (GSM) distribution. The EM formalism, as well as the connection to GSMs, allow us to establish that the IRLS algorithms minimize smooth versions of the lν `norms', for . We leverage EM theory to show that the limit points of the sequence of IRLS iterates are stationary points of the smooth lν "norm" minimization problem on the constraint set. We employ techniques from Compressive Sampling (CS) theory to show that the IRLS algorithm is stable, if the limit point of the iterates coincides with the global minimizer. We further characterize the convergence rate of the IRLS algorithm, which implies global linear convergence for ν = 1 and local super-linear convergence for . We demonstrate our results via simulation experiments. The simplicity of IRLS, along with the theoretical guarantees provided in this contribution, make a compelling case for its adoption as a standard tool for sparse signal recovery. |
|---|---|
| AbstractList | In this paper, we study the theoretical properties of iteratively re-weighted least squares (IRLS) algorithms and their utility in sparse signal recovery in the presence of noise. We demonstrate a one-to-one correspondence between the IRLS algorithms and a class of Expectation-Maximization (EM) algorithms for constrained maximum likelihood estimation under a Gaussian scale mixture (GSM) distribution. The EM formalism, as well as the connection to GSMs, allow us to establish that the IRLS algorithms minimize smooth versions of the lν `norms', for . We leverage EM theory to show that the limit points of the sequence of IRLS iterates are stationary points of the smooth lν "norm" minimization problem on the constraint set. We employ techniques from Compressive Sampling (CS) theory to show that the IRLS algorithm is stable, if the limit point of the iterates coincides with the global minimizer. We further characterize the convergence rate of the IRLS algorithm, which implies global linear convergence for ν = 1 and local super-linear convergence for . We demonstrate our results via simulation experiments. The simplicity of IRLS, along with the theoretical guarantees provided in this contribution, make a compelling case for its adoption as a standard tool for sparse signal recovery. |
| Author | Purdon, Patrick L. Ba, Demba Brown, Emery N. Babadi, Behtash |
| Author_xml | – sequence: 1 givenname: Demba surname: Ba fullname: Ba, Demba email: demba@neurostat.mit.edu organization: Dept. of Brain & Cognitive Sci., Massachusetts Inst. of Technol., Cambridge, MA, USA – sequence: 2 givenname: Behtash surname: Babadi fullname: Babadi, Behtash email: behtash@nmr.mgh.harvard.edu organization: Dept. of Brain & Cognitive Sci., Massachusetts Inst. of Technol., Cambridge, MA, USA – sequence: 3 givenname: Patrick L. surname: Purdon fullname: Purdon, Patrick L. email: patrickp@nmr.mgh.harvard.edu organization: Dept. of Brain & Cognitive Sci., Massachusetts Inst. of Technol., Cambridge, MA, USA – sequence: 4 givenname: Emery N. surname: Brown fullname: Brown, Emery N. email: enb@neurostat.mit.edu organization: Dept. of Brain & Cognitive Sci., Massachusetts Inst. of Technol., Cambridge, MA, USA |
| BackLink | http://pascal-francis.inist.fr/vibad/index.php?action=getRecordDetail&idt=28203508$$DView record in Pascal Francis |
| BookMark | eNp9kM9P20AQhVdVkEpC75V68aVHh1l7fx5RVChSqlYEpN6s8Xo2LDI23V1S5b_HUaCHHjjNO7zvSfPN2WwYB2LsM4cl52DPbze_lhXwellVRisjP7BTbgUvQWg1mzLIupRG__7I5ik9AHAhrDplP1bjsKO4pcFRgUNXbDK2oQ95X4y-uM4UMYcd9fvihsq_FLb3mbpiTZhysfnzjJFScdFvxxjy_WM6Yyce-0SfXu-C3V1-u119L9c_r65XF-vS1bXMpVcolVWi5VqT916QMU4DgoWuFp1vhdUWrTYVou2c5KBAompdp7Q2rq0X7Otx9wmTw95HHFxIzVMMjxj3TWUqqCWYqaeOPRfHlCL5xoU8PTQOOWLoGw7NQV4zyWsO8ppXeRMI_4Fv2-8gX45IIKJ_daUkgID6BTPqfLk |
| CODEN | ITPRED |
| CitedBy_id | crossref_primary_10_1016_j_sigpro_2020_107853 crossref_primary_10_1016_j_neuroimage_2015_09_048 crossref_primary_10_1109_TAES_2024_3437348 crossref_primary_10_1109_ACCESS_2020_2989452 crossref_primary_10_1109_TVT_2022_3203487 crossref_primary_10_1007_s10107_021_01644_z crossref_primary_10_1109_TSP_2015_2428216 crossref_primary_10_1016_j_laa_2021_08_024 crossref_primary_10_1667_RADE_22_00012_1 crossref_primary_10_1109_TSIPN_2018_2869117 crossref_primary_10_1007_s10898_021_01093_0 crossref_primary_10_1016_j_measurement_2024_114895 crossref_primary_10_1109_LSP_2018_2875251 crossref_primary_10_1109_TSP_2014_2338839 crossref_primary_10_1109_TCSII_2023_3311814 crossref_primary_10_1109_TSP_2024_3421231 crossref_primary_10_1109_TCI_2016_2516909 crossref_primary_10_1109_TAES_2023_3241886 crossref_primary_10_1109_TSP_2022_3146790 crossref_primary_10_1007_s11336_023_09911_y crossref_primary_10_1109_TAES_2020_3021809 crossref_primary_10_1016_j_catena_2024_108576 crossref_primary_10_1109_ACCESS_2019_2922322 crossref_primary_10_1109_TBME_2017_2694339 crossref_primary_10_1214_20_STS816 crossref_primary_10_3934_ipi_2017030 crossref_primary_10_1109_LWC_2024_3518206 crossref_primary_10_1109_JSAC_2016_2544578 crossref_primary_10_1109_TKDE_2020_3037131 crossref_primary_10_1007_s10898_023_01361_1 crossref_primary_10_1016_j_jfranklin_2023_01_039 crossref_primary_10_1109_TWC_2022_3175740 crossref_primary_10_1016_j_jsv_2019_06_041 crossref_primary_10_1109_TMC_2021_3108960 crossref_primary_10_7554_eLife_68046 crossref_primary_10_1007_s11277_014_1820_1 crossref_primary_10_1109_ACCESS_2019_2948426 crossref_primary_10_1109_TAES_2025_3527939 crossref_primary_10_1016_j_neuroimage_2022_119496 crossref_primary_10_1109_TWC_2020_3006083 crossref_primary_10_1109_TSP_2016_2601299 crossref_primary_10_1109_TSP_2016_2572041 crossref_primary_10_1109_TII_2019_2899130 crossref_primary_10_1109_TAES_2024_3395233 crossref_primary_10_1038_s41598_025_17068_0 crossref_primary_10_1016_j_jfranklin_2023_11_031 crossref_primary_10_1109_JSTARS_2023_3270309 crossref_primary_10_1073_pnas_1320637111 crossref_primary_10_1109_TWC_2015_2485992 |
| Cites_doi | 10.1190/1.3509470 10.1109/TSP.2011.2161470 10.1007/BF01385796 10.1002/mrm.21391 10.1002/cpa.20303 10.1137/060657704 10.1109/TSP.2010.2048103 10.1109/JPROC.2010.2042415 10.1002/cpa.20124 10.2307/3316118 10.1155/2009/162824 10.1214/aos/1176346060 10.1109/78.558475 10.1002/0471725250 10.1109/ICASSP.2008.4518498 10.1111/j.2517-6161.1977.tb01600.x 10.1109/TIT.2011.2159959 10.1007/s00041-008-9045-x 10.1007/978-1-84800-155-8_7 10.2307/1390698 10.1007/978-1-4757-4182-7 10.1109/CISS.2009.5054762 10.1109/TIT.2005.860430 10.1109/TIT.2005.858979 10.1111/j.2517-6161.1974.tb00989.x 10.1109/TIT.2006.871582 10.1109/ICASSP.2008.4518502 |
| ContentType | Journal Article |
| Copyright | 2015 INIST-CNRS |
| Copyright_xml | – notice: 2015 INIST-CNRS |
| DBID | 97E RIA RIE AAYXX CITATION IQODW |
| DOI | 10.1109/TSP.2013.2287685 |
| DatabaseName | IEEE All-Society Periodicals Package (ASPP) 2005–Present IEEE All-Society Periodicals Package (ASPP) 1998–Present IEEE Electronic Library (IEL) CrossRef Pascal-Francis |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| Database_xml | – sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Engineering Applied Sciences |
| EISSN | 1941-0476 |
| EndPage | 195 |
| ExternalDocumentID | 28203508 10_1109_TSP_2013_2287685 6650040 |
| Genre | orig-research |
| GroupedDBID | -~X .DC 0R~ 29I 4.4 5GY 6IK 85S 97E AAJGR AARMG AASAJ AAWTH ABAZT ABQJQ ABVLG ACGFO ACIWK ACNCT AENEX AGQYO AGSQL AHBIQ AJQPL AKQYR ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 EBS EJD F5P HZ~ IFIPE IPLJI JAVBF LAI MS~ O9- OCL P2P RIA RIE RNS TAE TN5 3EH 53G 5VS AAYXX ABFSI ACKIV AETIX AI. AIBXA AKJIK ALLEH CITATION E.L H~9 ICLAB IFJZH VH1 AAYOK IQODW RIG |
| ID | FETCH-LOGICAL-c335t-f6a56964b177efff4e88c70a090d34dfb4979a9782aa9dc510605a6bcd6778cb3 |
| IEDL.DBID | RIE |
| ISICitedReferencesCount | 76 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000330291000015&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 1053-587X |
| IngestDate | Wed Apr 02 07:24:04 EDT 2025 Tue Nov 18 21:44:03 EST 2025 Sat Nov 29 04:10:30 EST 2025 Tue Aug 26 16:50:03 EDT 2025 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 1 |
| Keywords | Mobile radiocommunication GSM system expectation-maximization algorithms Wireless telecommunication Iterative method Compressive sampling Weighting Gaussian process Signal restoration Gaussian scale mixtures Simulation Least squares method constrained maximum likelihood estimation Convergence rate Signal processing Signal reconstruction Maximum likelihood EM algorithm Compressed sensing Mathematical programming |
| Language | English |
| License | https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html CC BY 4.0 |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c335t-f6a56964b177efff4e88c70a090d34dfb4979a9782aa9dc510605a6bcd6778cb3 |
| OpenAccessLink | http://hdl.handle.net/1721.1/86328 |
| PageCount | 13 |
| ParticipantIDs | crossref_citationtrail_10_1109_TSP_2013_2287685 pascalfrancis_primary_28203508 crossref_primary_10_1109_TSP_2013_2287685 ieee_primary_6650040 |
| PublicationCentury | 2000 |
| PublicationDate | 2014-Jan.1, 2014-1-00 2014 |
| PublicationDateYYYYMMDD | 2014-01-01 |
| PublicationDate_xml | – month: 01 year: 2014 text: 2014-Jan.1, day: 01 |
| PublicationDecade | 2010 |
| PublicationPlace | New York, NY |
| PublicationPlace_xml | – name: New York, NY |
| PublicationTitle | IEEE transactions on signal processing |
| PublicationTitleAbbrev | TSP |
| PublicationYear | 2014 |
| Publisher | IEEE Institute of Electrical and Electronics Engineers |
| Publisher_xml | – name: IEEE – name: Institute of Electrical and Electronics Engineers |
| References | ref13 ref12 neelamani (ref6) 2010; 75 ref14 dempster (ref18) 1980 ref33 ref11 ref32 ref10 dempster (ref21) 1977; 39 ref2 ref1 ref17 lange (ref22) 2004 bertsekas (ref27) 2003 ref24 ref23 ref26 bertsekas (ref16) 1999 huber (ref19) 1981; 1 zangwill (ref25) 1969 ref28 ref29 ref8 ref7 grant (ref30) 2011 grant (ref31) 2008 ref9 voronin (ref15) 2012 ref4 ref3 ref5 andrews (ref20) 1974 |
| References_xml | – volume: 75 start-page: wb15 year: 2010 ident: ref6 article-title: Efficient seismic forward modeling using simultaneous random sources and sparsity publication-title: Geophys doi: 10.1190/1.3509470 – ident: ref29 doi: 10.1109/TSP.2011.2161470 – ident: ref23 doi: 10.1007/BF01385796 – ident: ref5 doi: 10.1002/mrm.21391 – ident: ref14 doi: 10.1002/cpa.20303 – year: 2003 ident: ref27 publication-title: Convex Analysis and Optimization – ident: ref2 doi: 10.1137/060657704 – year: 1999 ident: ref16 publication-title: Nonlinear Programming – ident: ref4 doi: 10.1109/TSP.2010.2048103 – ident: ref3 doi: 10.1109/JPROC.2010.2042415 – ident: ref33 doi: 10.1002/cpa.20124 – ident: ref26 doi: 10.2307/3316118 – ident: ref7 doi: 10.1155/2009/162824 – ident: ref24 doi: 10.1214/aos/1176346060 – ident: ref8 doi: 10.1109/78.558475 – volume: 1 year: 1981 ident: ref19 publication-title: Robust Statistics doi: 10.1002/0471725250 – ident: ref11 doi: 10.1109/ICASSP.2008.4518498 – year: 2011 ident: ref30 publication-title: CVX MATLAB Software for Disciplined Convex Programming – start-page: 35 year: 1980 ident: ref18 publication-title: Multivariate Analysis V – volume: 39 start-page: 1 year: 1977 ident: ref21 article-title: Maximum likelihood from incomplete data via the EM algorithm publication-title: J Royal Statistical Society Series B (Methodological) doi: 10.1111/j.2517-6161.1977.tb01600.x – ident: ref13 doi: 10.1109/TIT.2011.2159959 – ident: ref32 doi: 10.1007/s00041-008-9045-x – start-page: 95 year: 2008 ident: ref31 publication-title: Recent Advances in Learning and Control doi: 10.1007/978-1-84800-155-8_7 – ident: ref17 doi: 10.2307/1390698 – year: 2004 ident: ref22 publication-title: Optimization doi: 10.1007/978-1-4757-4182-7 – ident: ref12 doi: 10.1109/CISS.2009.5054762 – ident: ref9 doi: 10.1109/TIT.2005.860430 – ident: ref28 doi: 10.1109/TIT.2005.858979 – start-page: 99 year: 1974 ident: ref20 article-title: Scale mixtures of normal distributions publication-title: J Royal Statistical Society Series B (Methodological) doi: 10.1111/j.2517-6161.1974.tb00989.x – ident: ref1 doi: 10.1109/TIT.2006.871582 – year: 1969 ident: ref25 publication-title: Nonlinear Programming A Unified Approach – year: 2012 ident: ref15 publication-title: Regularization of linear systems with sparsity constraints with applications to large scale inverse problems – ident: ref10 doi: 10.1109/ICASSP.2008.4518502 |
| SSID | ssj0014496 |
| Score | 2.417005 |
| Snippet | In this paper, we study the theoretical properties of iteratively re-weighted least squares (IRLS) algorithms and their utility in sparse signal recovery in... |
| SourceID | pascalfrancis crossref ieee |
| SourceType | Index Database Enrichment Source Publisher |
| StartPage | 183 |
| SubjectTerms | Applied sciences Compressive sampling constrained maximum likelihood estimation Convergence Convex functions Detection, estimation, filtering, equalization, prediction Exact sciences and technology expectation-maximization algorithms Gaussian scale mixtures Information, signal and communications theory mathematical programming Noise Random variables Sampling, quantization Signal and communications theory Signal processing algorithms Signal, noise Stability analysis Telecommunications and information theory |
| Title | Convergence and Stability of Iteratively Re-weighted Least Squares Algorithms |
| URI | https://ieeexplore.ieee.org/document/6650040 |
| Volume | 62 |
| WOSCitedRecordID | wos000330291000015&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1941-0476 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014496 issn: 1053-587X databaseCode: RIE dateStart: 19910101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LS8QwEB508aAH3-L6IgcvgnHrNm2a4yKKgor4gL2VNJmosO7qPhT_vZO0lhVE8FZoEspM0pkv8_gA9i359W3nJLeFQy7QINfk5XK0mIhjGZHJtIFsQl5fZ92uupmBw7oWBhFD8hke-ccQy7cDM_FXZa2U3AnadLMwK2Va1mrVEQMhAhcXuQsxTzLZ_Q5JRqp1f3fjc7jiozbBg9SzJk-ZoMCp4jMi9YiE4ko2iykTc7b0v49bhsXKlWSdUvcrMIP9VViYajC4BlcnPqk81Fci033LyLcM2bCfbODYReioTL-73ie7Rf4RbknRskvP58Pu3ia-Nol1eo-D4fP46WW0Dg9np_cn57wiUOAmjpMxd6lOUpUK32IKnXMCs8zISEcqsrGwrhBKKk04sq21soaOJ4EbnRbG-rZypog3oNEf9HETGGlRJLEpkizTQqaGgJqNtIpcQYsguia0vmWam6q7uCe56OUBZUQqJy3kXgt5pYUmHNQzXsvOGn-MXfMCr8dVsm7C3g-11e8JR_qIabb1-7xtmKfVRXmTsgON8XCCuzBn3sfPo-Fe2Fdfoh7MQg |
| linkProvider | IEEE |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1ZSwMxEB68QH3wFuuZB18EY9du9sijiKJYi2iFvi3ZZKKF2moPxX_vJF2XCiL4trDJ7jKTbOab6wM4NGTX16xNuMktcoEauSIrl6PBSJwmAR2ZxpNNJI1G2mrJuyk4LmthENEnn-GJu_SxfNPTI-cqq8ZkTtCim4ZZx5xVVGuVMQMhPBsXGQwhj9Kk9R2UDGS1-XDnsrjCkxoBhNjxJk8cQp5VxeVEqgGJxY75LCYOmcvl_33eCiwVxiQ7G2t_FaawuwaLEy0G1-H23KWV-wpLZKprGFmXPh_2k_Usu_Y9lemH1_lk98g_vJ8UDas7Rh_28DZy1UnsrPPU67eHzy-DDXi8vGieX_GCQoHrMIyG3MYqimUsXJMptNYKTFOdBCqQgQmFsbmQiVSEJGtKSaNpgxK8UXGujWssp_NwE2a6vS5uASM9iijUeZSmSiSxJqhmAiUDm9NDEG0Fqt8yzXTRX9zRXHQyjzMCmZEWMqeFrNBCBY7KGa_j3hp_jF13Ai_HFbKuwP4PtZX3CUm6mGm6_fu8A5i_at7Ws_p142YHFuhNYuxX2YWZYX-EezCn34ftQX_fr7EvcCrPiw |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Convergence+and+Stability+of+Iteratively+Re-weighted+Least+Squares+Algorithms&rft.jtitle=IEEE+transactions+on+signal+processing&rft.au=BA%2C+Demba&rft.au=BABADI%2C+Behtash&rft.au=PURDON%2C+Patrick+L&rft.au=BROWN%2C+Emery+N&rft.date=2014&rft.pub=Institute+of+Electrical+and+Electronics+Engineers&rft.issn=1053-587X&rft.volume=62&rft.issue=1-4&rft.spage=183&rft.epage=195&rft_id=info:doi/10.1109%2FTSP.2013.2287685&rft.externalDBID=n%2Fa&rft.externalDocID=28203508 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1053-587X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1053-587X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1053-587X&client=summon |