Finite Sample Analysis of Approximate Message Passing Algorithms

Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on information theory Vol. 64; no. 11; pp. 7264 - 7286
Main Authors: Rush, Cynthia, Venkataramanan, Ramji
Format: Journal Article
Language:English
Published: New York IEEE 01.11.2018
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Approximate message passing
Approximation algorithms
Asymptotic properties
compressed sensing
concentration inequalities
Empirical analysis
Estimation
Evolution
Iterative methods
large deviations
Linear matrix inequalities
Message passing
Noise measurement
non-asymptotic analysis
Regression analysis
Signal processing algorithms
Sparse matrices
state evolution
Statistical analysis
ISSN:0018-9448, 1557-9654
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Abstract Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector <inline-formula> <tex-math notation="LaTeX">\beta _{0} </tex-math></inline-formula> from a noisy measurement <inline-formula> <tex-math notation="LaTeX">y=A \beta _{0} + w </tex-math></inline-formula>. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix <inline-formula> <tex-math notation="LaTeX">A </tex-math></inline-formula>, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension <inline-formula> <tex-math notation="LaTeX">n \times N </tex-math></inline-formula>. The result shows that the probability of deviation from the state evolution prediction falls exponentially in <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula> can grow no faster than order <inline-formula> <tex-math notation="LaTeX">({\log n}/{\log \log n}) </tex-math></inline-formula> for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
AbstractList Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in highdimensional problems such as compressed sensing and lowrank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of highdimensional regression, where the goal is to estimate a highdimensional vector β0 from a noisy measurement y = Aβ0 + ω. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix A, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension n × N. The result shows that the probability of deviation from the state evolution prediction falls exponentially in n. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations t can grow no faster than order (log n/log log n) for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector <inline-formula> <tex-math notation="LaTeX">\beta _{0} </tex-math></inline-formula> from a noisy measurement <inline-formula> <tex-math notation="LaTeX">y=A \beta _{0} + w </tex-math></inline-formula>. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix <inline-formula> <tex-math notation="LaTeX">A </tex-math></inline-formula>, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension <inline-formula> <tex-math notation="LaTeX">n \times N </tex-math></inline-formula>. The result shows that the probability of deviation from the state evolution prediction falls exponentially in <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula> can grow no faster than order <inline-formula> <tex-math notation="LaTeX">({\log n}/{\log \log n}) </tex-math></inline-formula> for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
Author Venkataramanan, Ramji
Rush, Cynthia
Author_xml – sequence: 1
  givenname: Cynthia
  orcidid: 0000-0001-6857-2855
  surname: Rush
  fullname: Rush, Cynthia
  email: cynthia.rush@columbia.edu
  organization: Department of Statistics, Columbia University, New York, NY, USA
– sequence: 2
  givenname: Ramji
  orcidid: 0000-0001-7915-5432
  surname: Venkataramanan
  fullname: Venkataramanan, Ramji
  email: rv285@cam.ac.uk
  organization: Department of Engineering, University of Cambridge, Cambridge
BookMark eNp9kEFLAzEQhYNUsK3eBS8LnrdmsptscnMpVgsVBes5pEm2pmx312QL7b83pcWDB0_D8OabefNGaNC0jUXoFvAEAIuH5Xw5IRj4hHBgjMMFGgKlRSoYzQdoiKOUijznV2gUwia2OQUyRI8z17jeJh9q29U2KRtVH4ILSVslZdf5du-2KsqvNgS1tsm7CsE166Ss1613_dc2XKPLStXB3pzrGH3OnpbTl3Tx9jyflotUZ7ToU00KIJaxqtIEWzBCG0K0tqYynBpOVhnnbBV1ZcAYkoMyuRIrTYURFOc0G6P7095o6ntnQy837c5Hu0ESIJkoCDAcp_BpSvs2BG8r2fn4gT9IwPKYk4w5yWNO8pxTRNgfRLte9a5teq9c_R94dwKdtfb3Ds-AM0GzH5uGdzM
CODEN IETTAW
CitedBy_id crossref_primary_10_1073_pnas_2322232121
crossref_primary_10_1214_25_AAP2186
crossref_primary_10_1109_TIT_2024_3486685
crossref_primary_10_1109_TIT_2025_3560070
crossref_primary_10_1109_TWC_2022_3206608
crossref_primary_10_1109_TSP_2019_2899286
crossref_primary_10_1109_TSP_2019_2910451
crossref_primary_10_1109_TSP_2020_3005545
crossref_primary_10_1007_s42081_019_00057_9
crossref_primary_10_1109_TIT_2023_3307553
crossref_primary_10_1007_s10208_021_09531_x
crossref_primary_10_1109_TWC_2025_3529935
crossref_primary_10_1073_pnas_2302930120
crossref_primary_10_1109_TIT_2018_2882177
crossref_primary_10_1109_TIT_2021_3083733
crossref_primary_10_1214_25_AOS2544
crossref_primary_10_1214_23_AOS2257
crossref_primary_10_1109_TIT_2022_3220046
crossref_primary_10_1109_LWC_2023_3256300
crossref_primary_10_1214_23_AAP1953
crossref_primary_10_1214_24_AAP2056
crossref_primary_10_1109_TIT_2019_2934152
crossref_primary_10_1109_TIT_2024_3396472
crossref_primary_10_1109_TIT_2020_3025272
Cites_doi 10.1007/s00220-013-1862-3
10.1088/1742-5468/2012/08/P08009
10.1214/14-AAP1010
10.1073/pnas.0909892106
10.1109/TIT.2013.2274513
10.1109/ISIT.2017.8006797
10.1109/ISIT.2014.6875082
10.1109/TSP.2014.2357776
10.1109/ISIT.2017.8006975
10.1109/ACCESS.2017.2653119
10.1017/CBO9780511794308.010
10.1093/imaiai/iaw017
10.1109/ISIT.2017.8006524
10.1109/ISIT.2011.6033942
10.1093/imaiai/iat004
10.1109/TIT.2010.2094817
10.1109/ISIT.2014.6875223
10.1109/TIT.2016.2556702
10.1109/TIT.2015.2457942
10.1109/TIT.2017.2713833
10.1007/s00440-015-0675-z
10.1109/TIT.2017.2649460
10.1109/TSP.2014.2357773
10.1109/TIT.2011.2174612
10.1109/ALLERTON.2015.7447070
10.1109/ISIT.2012.6283056
10.1109/ISIT.2017.8006578
10.1093/acprof:oso/9780199535255.001.0001
10.1109/TIT.2014.2309005
ContentType Journal Article
Copyright Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
Copyright_xml – notice: Copyright The Institute of Electrical and Electronics Engineers, Inc. (IEEE) 2018
DBID 97E
RIA
RIE
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
DOI 10.1109/TIT.2018.2816681
DatabaseName IEEE All-Society Periodicals Package (ASPP) 2005–Present
IEEE All-Society Periodicals Package (ASPP) 1998–Present
IEEE/IET Electronic Library (IEL)
CrossRef
Computer and Information Systems Abstracts
Electronics & Communications Abstracts
Technology Research Database
ProQuest Computer Science Collection
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts – Academic
Computer and Information Systems Abstracts Professional
DatabaseTitle CrossRef
Technology Research Database
Computer and Information Systems Abstracts – Academic
Electronics & Communications Abstracts
ProQuest Computer Science Collection
Computer and Information Systems Abstracts
Advanced Technologies Database with Aerospace
Computer and Information Systems Abstracts Professional
DatabaseTitleList Technology Research Database
Database_xml – sequence: 1
  dbid: RIE
  name: IEEE/IET Electronic Library (IEL)
  url: https://ieeexplore.ieee.org/
  sourceTypes: Publisher
DeliveryMethod fulltext_linktorsrc
Discipline Engineering
Computer Science
EISSN 1557-9654
EndPage 7286
ExternalDocumentID 10_1109_TIT_2018_2816681
8318695
Genre orig-research
GrantInformation_xml – fundername: FP7 People: Marie-Curie Actions
  grantid: 631489
  funderid: 10.13039/100011264
GroupedDBID -~X
.DC
0R~
29I
3EH
4.4
5GY
5VS
6IK
97E
AAJGR
AARMG
AASAJ
AAWTH
ABAZT
ABFSI
ABQJQ
ABVLG
ACGFO
ACGFS
ACGOD
ACIWK
AENEX
AETEA
AETIX
AGQYO
AGSQL
AHBIQ
AI.
AIBXA
AKJIK
AKQYR
ALLEH
ALMA_UNASSIGNED_HOLDINGS
ASUFR
ATWAV
BEFXN
BFFAM
BGNUA
BKEBE
BPEOZ
CS3
DU5
E.L
EBS
EJD
F5P
HZ~
H~9
IAAWW
IBMZZ
ICLAB
IDIHD
IFIPE
IFJZH
IPLJI
JAVBF
LAI
M43
MS~
O9-
OCL
P2P
PQQKQ
RIA
RIE
RNS
RXW
TAE
TN5
VH1
VJK
AAYXX
CITATION
7SC
7SP
8FD
JQ2
L7M
L~C
L~D
ID FETCH-LOGICAL-c357t-c2712e66ffc20e1d9cd22ccedfd85d82b3886b66fad1dd241ad4a9bc59d950453
IEDL.DBID RIE
ISICitedReferencesCount 44
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000448029300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 0018-9448
IngestDate Sun Nov 30 04:28:55 EST 2025
Tue Nov 18 22:32:50 EST 2025
Sat Nov 29 03:31:40 EST 2025
Wed Aug 27 02:52:49 EDT 2025
IsPeerReviewed true
IsScholarly true
Issue 11
Language English
License https://ieeexplore.ieee.org/Xplorehelp/downloads/license-information/IEEE.html
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-c357t-c2712e66ffc20e1d9cd22ccedfd85d82b3886b66fad1dd241ad4a9bc59d950453
Notes ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ORCID 0000-0001-6857-2855
0000-0001-7915-5432
PQID 2123972160
PQPubID 36024
PageCount 23
ParticipantIDs proquest_journals_2123972160
ieee_primary_8318695
crossref_primary_10_1109_TIT_2018_2816681
crossref_citationtrail_10_1109_TIT_2018_2816681
PublicationCentury 2000
PublicationDate 2018-11-01
PublicationDateYYYYMMDD 2018-11-01
PublicationDate_xml – month: 11
  year: 2018
  text: 2018-11-01
  day: 01
PublicationDecade 2010
PublicationPlace New York
PublicationPlace_xml – name: New York
PublicationTitle IEEE transactions on information theory
PublicationTitleAbbrev TIT
PublicationYear 2018
Publisher IEEE
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Publisher_xml – name: IEEE
– name: The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
References ref13
ref12
ref15
ref30
ref11
ref10
ref1
ref17
ref16
ref19
ref18
barbier (ref14) 2017
donoho (ref2) 2009; 106
ref24
ref23
donoho (ref26) 2015; 166
ref25
ref20
ref22
ref21
ref28
ref27
ref29
ref8
ref7
ref9
ref4
ref3
ref6
ref5
References_xml – ident: ref21
  doi: 10.1007/s00220-013-1862-3
– ident: ref3
  doi: 10.1088/1742-5468/2012/08/P08009
– ident: ref8
  doi: 10.1214/14-AAP1010
– volume: 106
  start-page: 18914
  year: 2009
  ident: ref2
  article-title: Message-passing algorithms for compressed sensing
  publication-title: Proc Nat Acad Sci USA
  doi: 10.1073/pnas.0909892106
– ident: ref22
  doi: 10.1109/TIT.2013.2274513
– ident: ref30
  doi: 10.1109/ISIT.2017.8006797
– ident: ref15
  doi: 10.1109/ISIT.2014.6875082
– ident: ref19
  doi: 10.1109/TSP.2014.2357776
– ident: ref23
  doi: 10.1109/ISIT.2017.8006975
– ident: ref28
  doi: 10.1109/ACCESS.2017.2653119
– ident: ref4
  doi: 10.1017/CBO9780511794308.010
– ident: ref12
  doi: 10.1093/imaiai/iaw017
– start-page: 424
  year: 2017
  ident: ref14
  article-title: Mutual information for symmetric rank-one matrix estimation: A proof of the replica formula
  publication-title: Proc Adv Neural Inf Process Syst
– ident: ref24
  doi: 10.1109/ISIT.2017.8006524
– ident: ref5
  doi: 10.1109/ISIT.2011.6033942
– ident: ref7
  doi: 10.1093/imaiai/iat004
– ident: ref1
  doi: 10.1109/TIT.2010.2094817
– ident: ref10
  doi: 10.1109/ISIT.2014.6875223
– ident: ref18
  doi: 10.1109/TIT.2016.2556702
– ident: ref11
  doi: 10.1109/TIT.2015.2457942
– ident: ref17
  doi: 10.1109/TIT.2017.2713833
– volume: 166
  start-page: 935
  year: 2015
  ident: ref26
  article-title: High dimensional robust M-estimation: Asymptotic variance via approximate message passing
  publication-title: Probab Theory Rel Fields
  doi: 10.1007/s00440-015-0675-z
– ident: ref16
  doi: 10.1109/TIT.2017.2649460
– ident: ref20
  doi: 10.1109/TSP.2014.2357773
– ident: ref6
  doi: 10.1109/TIT.2011.2174612
– ident: ref13
  doi: 10.1109/ALLERTON.2015.7447070
– ident: ref9
  doi: 10.1109/ISIT.2012.6283056
– ident: ref29
  doi: 10.1109/ISIT.2017.8006578
– ident: ref25
  doi: 10.1093/acprof:oso/9780199535255.001.0001
– ident: ref27
  doi: 10.1109/TIT.2014.2309005
SSID ssj0014512
Score 2.5669482
Snippet Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing...
Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in highdimensional problems such as compressed sensing...
SourceID proquest
crossref
ieee
SourceType Aggregation Database
Enrichment Source
Index Database
Publisher
StartPage 7264
SubjectTerms Algorithms
Approximate message passing
Approximation algorithms
Asymptotic properties
compressed sensing
concentration inequalities
Empirical analysis
Estimation
Evolution
Iterative methods
large deviations
Linear matrix inequalities
Message passing
Noise measurement
non-asymptotic analysis
Regression analysis
Signal processing algorithms
Sparse matrices
state evolution
Statistical analysis
Title Finite Sample Analysis of Approximate Message Passing Algorithms
URI https://ieeexplore.ieee.org/document/8318695
https://www.proquest.com/docview/2123972160
Volume 64
WOSCitedRecordID wos000448029300024&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
journalDatabaseRights – providerCode: PRVIEE
  databaseName: IEEE/IET Electronic Library (IEL)
  customDbUrl:
  eissn: 1557-9654
  dateEnd: 99991231
  omitProxy: false
  ssIdentifier: ssj0014512
  issn: 0018-9448
  databaseCode: RIE
  dateStart: 19630101
  isFulltext: true
  titleUrlDefault: https://ieeexplore.ieee.org/
  providerName: IEEE
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1LSwMxEB6seNCDj1axvsjBi-C2m30luVnEoqClYJXelk0mWxdqK32IP99kH0VRBG8Lm1mWTGa_yU7m-wDOMdERMh06yJE7gUJ0JOOpY6BaJ9IXUuRyb8_3rNfjw6Hor8HlqhdGa50fPtMte5nX8nGqlvZXWZv7VkAprEGNsajo1VpVDIKQFszg1ASw2XNUJUlXtAd3A3uGi7c8WyTj9BsE5ZoqPz7EObp0d_73XruwXWaRpFO4fQ_W9KQOO5VCAykDtg5bX-gGG3DVzWyGSR4TSwlMKj4SMk1Jx1KLf2QmfdXkwaqijDTpm7za2JHOeDSdZYuX1_k-PHVvBte3Tqmg4Cg_ZAtHeYx6OorSVHmupigUep5SGlPkIXJP-pxH0txPkCIaME8wSIRUoUARmmTPP4D1yXSiD4EIn6IXhQbNAhFQqqRizNUJiyRKxdOoCe1qUmNV0otblYtxnG8zXBEbN8TWDXHphiZcrCzeCmqNP8Y27LSvxpUz3oSTym9xGXvz2IJxzknkHv1udQyb9tlFR-EJrC9mS30KG-p9kc1nZ_my-gTt58sH
linkProvider IEEE
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwlV1bS8MwFD54A_XBecV5zYMvgnVN2rTJm0McE-cQnOJbaXJSHegm2xR_vkkvQ1EE3wrNoSUnp99JT873ARxhaiKMDfdQoPBCjeipWGSehWqTqkAqmcu93Xfiblc8PMibGTiZ9sIYY_LDZ-bUXea1fBzqN_errCECJ6DEZ2GehyHzi26tac0g5LTgBqc2hO2uoypK-rLRu-y5U1zilLkymaDfQChXVfnxKc7xpVX735utwkqZR5Jm4fg1mDGDdahVGg2kDNl1WP5COLgBZ62-yzHJbepIgUnFSEKGGWk6cvGPvk1gDbl2uiiPhtzYzNrakebz43DUnzy9jDfhrnXRO297pYaCpwMeTzzNYspMFGWZZr6hKDUyprXBDAVHwVQgRKTs_RQpooXzFMNUKs0lSm7TvWAL5gbDgdkGIgOKLOIWz0IZUqqVjmPfpHGkUGmRRXVoVJOa6JJg3OlcPCf5RsOXiXVD4tyQlG6ow_HU4rUg1_hj7Iab9um4csbrsFf5LSmjb5w4OM5Zifyd360OYbHdu-4kncvu1S4suecU_YV7MDcZvZl9WNDvk_54dJAvsU-oY85O
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Finite+Sample+Analysis+of+Approximate+Message+Passing+Algorithms&rft.jtitle=IEEE+transactions+on+information+theory&rft.au=Rush%2C+Cynthia&rft.au=Venkataramanan%2C+Ramji&rft.date=2018-11-01&rft.pub=The+Institute+of+Electrical+and+Electronics+Engineers%2C+Inc.+%28IEEE%29&rft.issn=0018-9448&rft.eissn=1557-9654&rft.volume=64&rft.issue=11&rft.spage=7264&rft_id=info:doi/10.1109%2FTIT.2018.2816681&rft.externalDBID=NO_FULL_TEXT
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0018-9448&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0018-9448&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0018-9448&client=summon