Finite Sample Analysis of Approximate Message Passing Algorithms

Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For...

Celý popis

Uložené v:

Podrobná bibliografia
Vydané v:	IEEE transactions on information theory Ročník 64; číslo 11; s. 7264 - 7286
Hlavní autori:	Rush, Cynthia, Venkataramanan, Ramji
Médium:	Journal Article
Jazyk:	English
Vydavateľské údaje:	New York IEEE 01.11.2018 The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Predmet:	Algorithms Approximate message passing Approximation algorithms Asymptotic properties compressed sensing concentration inequalities Empirical analysis Estimation Evolution Iterative methods large deviations Linear matrix inequalities Message passing Noise measurement non-asymptotic analysis Regression analysis Signal processing algorithms Sparse matrices state evolution Statistical analysis
ISSN:	0018-9448, 1557-9654
On-line prístup:	Získať plný text
Tagy:	Pridať tag Žiadne tagy, Buďte prvý, kto otaguje tento záznam!

Abstract	Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector <inline-formula> <tex-math notation="LaTeX">\beta _{0} </tex-math></inline-formula> from a noisy measurement <inline-formula> <tex-math notation="LaTeX">y=A \beta _{0} + w </tex-math></inline-formula>. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix <inline-formula> <tex-math notation="LaTeX">A </tex-math></inline-formula>, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension <inline-formula> <tex-math notation="LaTeX">n \times N </tex-math></inline-formula>. The result shows that the probability of deviation from the state evolution prediction falls exponentially in <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula> can grow no faster than order <inline-formula> <tex-math notation="LaTeX">({\log n}/{\log \log n}) </tex-math></inline-formula> for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
AbstractList	Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in highdimensional problems such as compressed sensing and lowrank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of highdimensional regression, where the goal is to estimate a highdimensional vector β0 from a noisy measurement y = Aβ0 + ω. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix A, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension n × N. The result shows that the probability of deviation from the state evolution prediction falls exponentially in n. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations t can grow no faster than order (log n/log log n) for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation. Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector <inline-formula> <tex-math notation="LaTeX">\beta _{0} </tex-math></inline-formula> from a noisy measurement <inline-formula> <tex-math notation="LaTeX">y=A \beta _{0} + w </tex-math></inline-formula>. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix <inline-formula> <tex-math notation="LaTeX">A </tex-math></inline-formula>, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension <inline-formula> <tex-math notation="LaTeX">n \times N </tex-math></inline-formula>. The result shows that the probability of deviation from the state evolution prediction falls exponentially in <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula> can grow no faster than order <inline-formula> <tex-math notation="LaTeX">({\log n}/{\log \log n}) </tex-math></inline-formula> for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
Author	Venkataramanan, Ramji Rush, Cynthia
Author_xml	– sequence: 1 givenname: Cynthia orcidid: 0000-0001-6857-2855 surname: Rush fullname: Rush, Cynthia email: cynthia.rush@columbia.edu organization: Department of Statistics, Columbia University, New York, NY, USA – sequence: 2 givenname: Ramji orcidid: 0000-0001-7915-5432 surname: Venkataramanan fullname: Venkataramanan, Ramji email: rv285@cam.ac.uk organization: Department of Engineering, University of Cambridge, Cambridge
BookMark	eNp9kEFLAzEQhYNUsK3eBS8LnrdmsptscnMpVgsVBes5pEm2pmx312QL7b83pcWDB0_D8OabefNGaNC0jUXoFvAEAIuH5Xw5IRj4hHBgjMMFGgKlRSoYzQdoiKOUijznV2gUwia2OQUyRI8z17jeJh9q29U2KRtVH4ILSVslZdf5du-2KsqvNgS1tsm7CsE166Ss1613_dc2XKPLStXB3pzrGH3OnpbTl3Tx9jyflotUZ7ToU00KIJaxqtIEWzBCG0K0tqYynBpOVhnnbBV1ZcAYkoMyuRIrTYURFOc0G6P7095o6ntnQy837c5Hu0ESIJkoCDAcp_BpSvs2BG8r2fn4gT9IwPKYk4w5yWNO8pxTRNgfRLte9a5teq9c_R94dwKdtfb3Ds-AM0GzH5uGdzM
CODEN	IETTAW
CitedBy_id	crossref_primary_10_1073_pnas_2322232121 crossref_primary_10_1214_25_AAP2186 crossref_primary_10_1109_TIT_2024_3486685 crossref_primary_10_1109_TIT_2025_3560070 crossref_primary_10_1109_TWC_2022_3206608 crossref_primary_10_1109_TSP_2019_2899286 crossref_primary_10_1109_TSP_2019_2910451 crossref_primary_10_1109_TSP_2020_3005545 crossref_primary_10_1007_s42081_019_00057_9 crossref_primary_10_1109_TIT_2023_3307553 crossref_primary_10_1007_s10208_021_09531_x crossref_primary_10_1109_TWC_2025_3529935 crossref_primary_10_1073_pnas_2302930120 crossref_primary_10_1109_TIT_2018_2882177 crossref_primary_10_1109_TIT_2021_3083733 crossref_primary_10_1214_25_AOS2544 crossref_primary_10_1214_23_AOS2257 crossref_primary_10_1109_TIT_2022_3220046 crossref_primary_10_1109_LWC_2023_3256300 crossref_primary_10_1214_23_AAP1953 crossref_primary_10_1214_24_AAP2056 crossref_primary_10_1109_TIT_2019_2934152 crossref_primary_10_1109_TIT_2024_3396472 crossref_primary_10_1109_TIT_2020_3025272
Cites_doi	10.1007/s00220-013-1862-3 10.1088/1742-5468/2012/08/P08009 10.1214/14-AAP1010 10.1073/pnas.0909892106 10.1109/TIT.2013.2274513 10.1109/ISIT.2017.8006797 10.1109/ISIT.2014.6875082 10.1109/TSP.2014.2357776 10.1109/ISIT.2017.8006975 10.1109/ACCESS.2017.2653119 10.1017/CBO9780511794308.010 10.1093/imaiai/iaw017 10.1109/ISIT.2017.8006524 10.1109/ISIT.2011.6033942 10.1093/imaiai/iat004 10.1109/TIT.2010.2094817 10.1109/ISIT.2014.6875223 10.1109/TIT.2016.2556702 10.1109/TIT.2015.2457942 10.1109/TIT.2017.2713833 10.1007/s00440-015-0675-z 10.1109/TIT.2017.2649460 10.1109/TSP.2014.2357773 10.1109/TIT.2011.2174612 10.1109/ALLERTON.2015.7447070 10.1109/ISIT.2012.6283056 10.1109/ISIT.2017.8006578 10.1093/acprof:oso/9780199535255.001.0001 10.1109/TIT.2014.2309005
ContentType	Journal Article
Copyright	Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
Copyright_xml	– notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
DBID	97E RIA RIE AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
DOI	10.1109/TIT.2018.2816681
DatabaseName	IEEE All-Society Periodicals Package (ASPP) 2005-present IEEE All-Society Periodicals Package (ASPP) 1998-Present IEEE Electronic Library (IEL) CrossRef Computer and Information Systems Abstracts Electronics & Communications Abstracts Technology Research Database ProQuest Computer Science Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional
DatabaseTitle	CrossRef Technology Research Database Computer and Information Systems Abstracts – Academic Electronics & Communications Abstracts ProQuest Computer Science Collection Computer and Information Systems Abstracts Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Professional
DatabaseTitleList	Technology Research Database
Database_xml	– sequence: 1 dbid: RIE name: IEEE Electronic Library (IEL) url: https://ieeexplore.ieee.org/ sourceTypes: Publisher
DeliveryMethod	fulltext_linktorsrc
Discipline	Engineering Computer Science
EISSN	1557-9654
EndPage	7286
ExternalDocumentID	10_1109_TIT_2018_2816681 8318695
Genre	orig-research
GrantInformation_xml	– fundername: FP7 People: Marie-Curie Actions grantid: 631489 funderid: 10.13039/100011264
GroupedDBID	-~X .DC 0R~ 29I 3EH 4.4 5GY 5VS 6IK 97E AAJGR AARMG AASAJ AAWTH ABAZT ABFSI ABQJQ ABVLG ACGFO ACGFS ACGOD ACIWK AENEX AETEA AETIX AGQYO AGSQL AHBIQ AI. AIBXA AKJIK AKQYR ALLEH ALMA_UNASSIGNED_HOLDINGS ASUFR ATWAV BEFXN BFFAM BGNUA BKEBE BPEOZ CS3 DU5 E.L EBS EJD F5P HZ~ H~9 IAAWW IBMZZ ICLAB IDIHD IFIPE IFJZH IPLJI JAVBF LAI M43 MS~ O9- OCL P2P PQQKQ RIA RIE RNS RXW TAE TN5 VH1 VJK AAYXX CITATION 7SC 7SP 8FD JQ2 L7M L~C L~D
ID	FETCH-LOGICAL-c357t-c2712e66ffc20e1d9cd22ccedfd85d82b3886b66fad1dd241ad4a9bc59d950453
IEDL.DBID	RIE
ISICitedReferencesCount	44
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000448029300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	0018-9448
IngestDate	Sun Nov 30 04:28:55 EST 2025 Tue Nov 18 22:32:50 EST 2025 Sat Nov 29 03:31:40 EST 2025 Wed Aug 27 02:52:49 EDT 2025
IsPeerReviewed	true
IsScholarly	true
Issue	11
Language	English
License	https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c357t-c2712e66ffc20e1d9cd22ccedfd85d82b3886b66fad1dd241ad4a9bc59d950453
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0001-6857-2855 0000-0001-7915-5432
PQID	2123972160
PQPubID	36024
PageCount	23
ParticipantIDs	proquest_journals_2123972160 ieee_primary_8318695 crossref_primary_10_1109_TIT_2018_2816681 crossref_citationtrail_10_1109_TIT_2018_2816681
PublicationCentury	2000
PublicationDate	2018-11-01
PublicationDateYYYYMMDD	2018-11-01
PublicationDate_xml	– month: 11 year: 2018 text: 2018-11-01 day: 01
PublicationDecade	2010
PublicationPlace	New York
PublicationPlace_xml	– name: New York
PublicationTitle	IEEE transactions on information theory
PublicationTitleAbbrev	TIT
PublicationYear	2018
Publisher	IEEE The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml	– name: IEEE – name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References	ref13 ref12 ref15 ref30 ref11 ref10 ref1 ref17 ref16 ref19 ref18 barbier (ref14) 2017 donoho (ref2) 2009; 106 ref24 ref23 donoho (ref26) 2015; 166 ref25 ref20 ref22 ref21 ref28 ref27 ref29 ref8 ref7 ref9 ref4 ref3 ref6 ref5
References_xml	– ident: ref21 doi: 10.1007/s00220-013-1862-3 – ident: ref3 doi: 10.1088/1742-5468/2012/08/P08009 – ident: ref8 doi: 10.1214/14-AAP1010 – volume: 106 start-page: 18914 year: 2009 ident: ref2 article-title: Message-passing algorithms for compressed sensing publication-title: Proc Nat Acad Sci USA doi: 10.1073/pnas.0909892106 – ident: ref22 doi: 10.1109/TIT.2013.2274513 – ident: ref30 doi: 10.1109/ISIT.2017.8006797 – ident: ref15 doi: 10.1109/ISIT.2014.6875082 – ident: ref19 doi: 10.1109/TSP.2014.2357776 – ident: ref23 doi: 10.1109/ISIT.2017.8006975 – ident: ref28 doi: 10.1109/ACCESS.2017.2653119 – ident: ref4 doi: 10.1017/CBO9780511794308.010 – ident: ref12 doi: 10.1093/imaiai/iaw017 – start-page: 424 year: 2017 ident: ref14 article-title: Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula publication-title: Proc Adv Neural Inf Process Syst – ident: ref24 doi: 10.1109/ISIT.2017.8006524 – ident: ref5 doi: 10.1109/ISIT.2011.6033942 – ident: ref7 doi: 10.1093/imaiai/iat004 – ident: ref1 doi: 10.1109/TIT.2010.2094817 – ident: ref10 doi: 10.1109/ISIT.2014.6875223 – ident: ref18 doi: 10.1109/TIT.2016.2556702 – ident: ref11 doi: 10.1109/TIT.2015.2457942 – ident: ref17 doi: 10.1109/TIT.2017.2713833 – volume: 166 start-page: 935 year: 2015 ident: ref26 article-title: High dimensional robust M-estimation: Asymptotic variance via approximate message passing publication-title: Probab Theory Rel Fields doi: 10.1007/s00440-015-0675-z – ident: ref16 doi: 10.1109/TIT.2017.2649460 – ident: ref20 doi: 10.1109/TSP.2014.2357773 – ident: ref6 doi: 10.1109/TIT.2011.2174612 – ident: ref13 doi: 10.1109/ALLERTON.2015.7447070 – ident: ref9 doi: 10.1109/ISIT.2012.6283056 – ident: ref29 doi: 10.1109/ISIT.2017.8006578 – ident: ref25 doi: 10.1093/acprof:oso/9780199535255.001.0001 – ident: ref27 doi: 10.1109/TIT.2014.2309005
SSID	ssj0014512
Score	2.5670106
Snippet	Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing... Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in highdimensional problems such as compressed sensing...
SourceID	proquest crossref ieee
SourceType	Aggregation Database Enrichment Source Index Database Publisher
StartPage	7264
SubjectTerms	Algorithms Approximate message passing Approximation algorithms Asymptotic properties compressed sensing concentration inequalities Empirical analysis Estimation Evolution Iterative methods large deviations Linear matrix inequalities Message passing Noise measurement non-asymptotic analysis Regression analysis Signal processing algorithms Sparse matrices state evolution Statistical analysis
Title	Finite Sample Analysis of Approximate Message Passing Algorithms
URI	https://ieeexplore.ieee.org/document/8318695 https://www.proquest.com/docview/2123972160
Volume	64
WOSCitedRecordID	wos000448029300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVIEE databaseName: IEEE Electronic Library (IEL) customDbUrl: eissn: 1557-9654 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0014512 issn: 0018-9448 databaseCode: RIE dateStart: 19630101 isFulltext: true titleUrlDefault: https://ieeexplore.ieee.org/ providerName: IEEE
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3fS8MwED624YM-ON0Up1Py4ItgtzZt2uTNIQ59cAycsLeSJukszFW2TvzzTfoLRRF8KzQXyl2vd9fLfR_AJWGx4k4sLZtLbHnM5xb3vdjylNARVjDBPZ6TTQSTCZ3P2bQB1_UsjFIqP3ymBuYy7-XLVGzNr7IhdQ2BEmlCMwj8Ylar7hh4xCmQwR3twLrmqFqSNhvOHmbmDBcdYNMko863EJRzqvz4EOfRZdz-33MdwH6ZRaJRYfZDaKhVB9oVQwMqHbYDe1_gBrtwM05MhomeuIEERhUeCUpjNDLQ4h-JTl8VejSsKAuFpjqv1nJotFyk6yR7ed0cwfP4bnZ7b5UMCpZwSZBZAgcOVr4fxwLbypFMSIyFUDKWlEiKI5dSP9L3uXSk1MGcS4-zSBAmGdHJnnsMrVW6UieAXBoTHfGwa9ZxlzAc6VpQCh44TOhNejCslBqKEl7csFwsw7zMsFmozRAaM4SlGXpwVUu8FdAaf6ztGrXX60qN96Bf2S0sfW8TmmCcYxLZp79LncGu2buYKOxDK1tv1TnsiPcs2awv8tfqE3SzyrQ
linkProvider	IEEE
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV3dS8MwED_mFNQHp5vidGoefBHs1qZNl7w5xLHhNgZW2FvJknQO5ib7EP98k36hKIJvhebactfr3fVyvx_ANWGR4k4kLZtLbHnM5xb3vcjylNARVjDBPR6TTTQHAzoasWEBbvNZGKVUvPlM1c1h3MuXC7Exv8oa1DUESmQLtonnYTuZ1sp7Bh5xEmxwR7uwrjqypqTNGkE3MLu4aB2bNhl1vgWhmFXlx6c4ji_t0v-e7BAO0jwStRLDH0FBzctQyjgaUOqyZdj_AjhYgbv21OSY6IkbUGCUIZKgRYRaBlz8Y6oTWIX6hhdlotBQZ9ZaDrVmk8Vyun55XR3Dc_shuO9YKYeCJVzSXFsCNx2sfD-KBLaVI5mQGAuhZCQpkRSPXUr9sT7PpSOlDudcepyNBWGSEZ3uuSdQnC_m6hSQSyOiYx52zTruEobHuhqUgjcdJvRFqtDIlBqKFGDc8FzMwrjQsFmozRAaM4SpGapwk0u8JeAaf6ytGLXn61KNV6GW2S1MvW8VmnAcoxLZZ79LXcFuJ-j3wl538HgOe-Y-yXxhDYrr5UZdwI54X09Xy8v4FfsEKxfN-w
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Finite+Sample+Analysis+of+Approximate+Message+Passing+Algorithms&rft.jtitle=IEEE+transactions+on+information+theory&rft.au=Rush%2C+Cynthia&rft.au=Venkataramanan%2C+Ramji&rft.date=2018-11-01&rft.pub=IEEE&rft.issn=0018-9448&rft.volume=64&rft.issue=11&rft.spage=7264&rft.epage=7286&rft_id=info:doi/10.1109%2FTIT.2018.2816681&rft.externalDocID=8318695
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9448&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9448&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9448&client=summon