Phase Transitions and Sample Complexity in Bayes-Optimal Matrix Factorization

We analyze the matrix factorization problem. Given a noisy measurement of a product of two matrices, the problem is to estimate back the original matrices. It arises in many applications, such as dictionary learning, blind matrix calibration, sparse principal component analysis, blind source separat...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on information theory Vol. 62; no. 7; pp. 4228 - 4265
Main Authors: Kabashima, Yoshiyuki, Krzakala, Florent, Mezard, Marc, Sakata, Ayaka, Zdeborova, Lenka
Format: Journal Article
Language:English
Published: New York IEEE 01.07.2016
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithm design and analysis
Algorithms
Bayesian analysis
Complexity theory
Computation
Factorization
Inference
Matrix
Noise measurement
Optimization
Phase transitions
Prediction algorithms
Principal component analysis
Probability distribution
Samples
Sparse matrices
Statistical analysis
Statistical mechanics
Statistical methods
statistical and computational tradeoff
statistical physics
message passing algorithms
phase transitions
computational barriers
Statistical inference
probabilistic matrix factorization
dictionary learning
sparse coding
ISSN:0018-9448, 1557-9654
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We analyze the matrix factorization problem. Given a noisy measurement of a product of two matrices, the problem is to estimate back the original matrices. It arises in many applications, such as dictionary learning, blind matrix calibration, sparse principal component analysis, blind source separation, low rank matrix completion, robust principal component analysis, or factor analysis. It is also important in machine learning: unsupervised representation learning can often be studied through matrix factorization. We use the tools of statistical mechanics-the cavity and replica methods-to analyze the achievability and computational tractability of the inference problems in the setting of Bayes-optimal inference, which amounts to assuming that the two matrices have random-independent elements generated from some known distribution, and this information is available to the inference algorithm. In this setting, we compute the minimal mean-squared-error achievable, in principle, in any computational time, and the error that can be achieved by an efficient approximate message passing algorithm. The computation is based on the asymptotic state-evolution analysis of the algorithm. The performance that our analysis predicts, both in terms of the achieved mean-squared-error and in terms of sample complexity, is extremely promising and motivating for a further development of the algorithm.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0018-9448
1557-9654
DOI:10.1109/TIT.2016.2556702