Variational Inference: A Review for Statisticians

One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review va...

Full description

Saved in:
Bibliographic Details
Published in:Journal of the American Statistical Association Vol. 112; no. 518; pp. 859 - 877
Main Authors: Blei, David M., Kucukelbir, Alp, McAuliffe, Jon D.
Format: Journal Article
Language:English
Published: Alexandria Taylor & Francis 03.04.2017
Taylor & Francis Group,LLC
Taylor & Francis Ltd
Subjects:
Algorithms
Americans
artificial intelligence
Bayesian analysis
Bayesian theory
catalytic activity
Closeness
Computationally intensive methods
Density
Families & family life
Inference
Machine learning
Markov analysis
Markov chain
Markov chains
mixtures
Optimization
Probability
probability distribution
Review
Sampling
Scholarship
Statistical analysis
Statistical computing
Statistical inference
Statistics
ISSN:0162-1459, 1537-274X, 1537-274X
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:One of the core problems of modern statistics is to approximate difficult-to-compute probability densities. This problem is especially important in Bayesian statistics, which frames all inference about unknown quantities as a calculation involving the posterior density. In this article, we review variational inference (VI), a method from machine learning that approximates probability densities through optimization. VI has been used in many applications and tends to be faster than classical methods, such as Markov chain Monte Carlo sampling. The idea behind VI is to first posit a family of densities and then to find a member of that family which is close to the target density. Closeness is measured by Kullback-Leibler divergence. We review the ideas behind mean-field variational inference, discuss the special case of VI applied to exponential family models, present a full example with a Bayesian mixture of Gaussians, and derive a variant that uses stochastic optimization to scale up to massive data. We discuss modern research in VI and highlight important open problems. VI is powerful, but it is not yet well understood. Our hope in writing this article is to catalyze statistical research on this class of algorithms. Supplementary materials for this article are available online.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:0162-1459
1537-274X
1537-274X
DOI:10.1080/01621459.2017.1285773