The LASSO Risk for Gaussian Matrices

We consider the problem of learning a coefficient vector x ο ∈ R N from noisy linear observation y = Ax o + ∈ R n . In many contexts (ranging from model selection to image processing), it is desirable to construct a sparse estimator x̂. In this case, a popular approach consists in solving an ℓ 1 -pe...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on information theory Vol. 58; no. 4; pp. 1997 - 2017
Main Authors: Bayati, M., Montanari, A.
Format: Journal Article
Language:English
Published: New York, NY IEEE 01.04.2012
Institute of Electrical and Electronics Engineers
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithm design and analysis
Applied sciences
Calibration
Compressed sensing
Detection, estimation, filtering, equalization, prediction
Equations
Exact sciences and technology
graphical models
Image processing
Image processing systems
Information theory
Information, signal and communications theory
Mean square error methods
message passing algorithms
Noise
Noise measurement
Normal distribution
random matrix theory
Services and terminals of telecommunications
Signal and communications theory
Signal processing
Signal, noise
Sparsity
state evolution
statistical learning
Systems, networks and services of telecommunications
Telecommunications
Telecommunications and information theory
Telemetry. Remote supervision. Telewarning. Remote control
Vectors
state evolution
Range finding
statistical learning
Random matrix
message passing algorithms
Image processing
Noise reduction
Model selection
Least squares problem
Algorithm
Mean square error
Learning
Message passing
Algorithm performance
Simulation
graphical models
Signal processing
random matrix theory
Compressed sensing
ISSN:0018-9448, 1557-9654
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We consider the problem of learning a coefficient vector x ο ∈ R N from noisy linear observation y = Ax o + ∈ R n . In many contexts (ranging from model selection to image processing), it is desirable to construct a sparse estimator x̂. In this case, a popular approach consists in solving an ℓ 1 -penalized least-squares problem known as the LASSO or basis pursuit denoising. For sequences of matrices A of increasing dimensions, with independent Gaussian entries, we prove that the normalized risk of the LASSO converges to a limit, and we obtain an explicit expression for this limit. Our result is the first rigorous derivation of an explicit formula for the asymptotic mean square error of the LASSO for random instances. The proof technique is based on the analysis of AMP, a recently developed efficient algorithm, that is inspired from graphical model ideas. Simulations on real data matrices suggest that our results can be relevant in a broad array of practical applications.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2011.2174612