Minimizing finite sums with the stochastic average gradient

We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method’s iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradie...

Full description

Saved in:
Bibliographic Details
Published in:Mathematical programming Vol. 162; no. 1-2; pp. 83 - 112
Main Authors: Schmidt, Mark, Le Roux, Nicolas, Bach, Francis
Format: Journal Article
Language:English
Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.03.2017
Springer Nature B.V
Springer Verlag
Subjects:
Algorithms
Calculus of Variations and Optimal Control; Optimization
Combinatorics
Computation
Computer Science
Convergence
Convex analysis
Full Length Paper
Machine Learning
Mathematical analysis
Mathematical and Computational Physics
Mathematical Methods in Physics
Mathematical models
Mathematics
Mathematics and Statistics
Mathematics of Computing
Methods
Numerical Analysis
Optimization
Optimization and Control
Regression analysis
Sag
Sampling
Statistics
Stochasticity
Strategy
Studies
Texts
Theoretical
68Q25
90C30
65K05
Convex optimization
90C25
90C06
Stochastic gradient methods
90C15
Convergence Rates
First-order methods
62L20
Stochastic Gradient
Convex Optimization
ISSN:0025-5610, 1436-4646
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We analyze the stochastic average gradient (SAG) method for optimizing the sum of a finite number of smooth convex functions. Like stochastic gradient (SG) methods, the SAG method’s iteration cost is independent of the number of terms in the sum. However, by incorporating a memory of previous gradient values the SAG method achieves a faster convergence rate than black-box SG methods. The convergence rate is improved from O ( 1 / k ) to O (1 /  k ) in general, and when the sum is strongly-convex the convergence rate is improved from the sub-linear O (1 /  k ) to a linear convergence rate of the form O ( ρ k ) for ρ < 1 . Further, in many cases the convergence rate of the new method is also faster than black-box deterministic gradient methods, in terms of the number of gradient evaluations. This extends our earlier work Le Roux et al. (Adv Neural Inf Process Syst, 2012 ), which only lead to a faster rate for well-conditioned strongly-convex problems. Numerical experiments indicate that the new algorithm often dramatically outperforms existing SG and deterministic gradient methods, and that the performance may be further improved through the use of non-uniform sampling strategies.
Bibliography:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0025-5610
1436-4646
DOI:10.1007/s10107-016-1030-6