Finite Sample Analysis of Approximate Message Passing Algorithms

Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on information theory Vol. 64; no. 11; pp. 7264 - 7286
Main Authors: Rush, Cynthia, Venkataramanan, Ramji
Format: Journal Article
Language:English
Published: New York IEEE 01.11.2018
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Approximate message passing
Approximation algorithms
Asymptotic properties
compressed sensing
concentration inequalities
Empirical analysis
Estimation
Evolution
Iterative methods
large deviations
Linear matrix inequalities
Message passing
Noise measurement
non-asymptotic analysis
Regression analysis
Signal processing algorithms
Sparse matrices
state evolution
Statistical analysis
ISSN:0018-9448, 1557-9654
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Approximate message passing (AMP) refers to a class of efficient algorithms for statistical estimation in high-dimensional problems such as compressed sensing and low-rank matrix estimation. This paper analyzes the performance of AMP in the regime where the problem dimension is large but finite. For concreteness, we consider the setting of high-dimensional regression, where the goal is to estimate a high-dimensional vector <inline-formula> <tex-math notation="LaTeX">\beta _{0} </tex-math></inline-formula> from a noisy measurement <inline-formula> <tex-math notation="LaTeX">y=A \beta _{0} + w </tex-math></inline-formula>. AMP is a low-complexity, scalable algorithm for this problem. Under suitable assumptions on the measurement matrix <inline-formula> <tex-math notation="LaTeX">A </tex-math></inline-formula>, AMP has the attractive feature that its performance can be accurately characterized in the large system limit by a simple scalar iteration called state evolution. Previous proofs of the validity of state evolution have all been asymptotic convergence results. In this paper, we derive a concentration inequality for AMP with Gaussian matrices with independent and identically distributed (i.i.d.) entries and finite dimension <inline-formula> <tex-math notation="LaTeX">n \times N </tex-math></inline-formula>. The result shows that the probability of deviation from the state evolution prediction falls exponentially in <inline-formula> <tex-math notation="LaTeX">n </tex-math></inline-formula>. This provides theoretical support for empirical findings that have demonstrated excellent agreement of AMP performance with state evolution predictions for moderately large dimensions. The concentration inequality also indicates that the number of AMP iterations <inline-formula> <tex-math notation="LaTeX">t </tex-math></inline-formula> can grow no faster than order <inline-formula> <tex-math notation="LaTeX">({\log n}/{\log \log n}) </tex-math></inline-formula> for the performance to be close to the state evolution predictions with high probability. The analysis can be extended to obtain similar non-asymptotic results for AMP in other settings such as low-rank matrix estimation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2018.2816681