Regularized stochastic BFGS algorithm

A regularized stochastic version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve optimization problems with stochastic objectives that arise in large scale machine learning. Stochastic gradient descent is the currently preferred solution methodology but the n...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:2013 IEEE Global Conference on Signal and Information Processing (GlobalSIP) s. 1109 - 1112
Hlavní autori: Mokhtari, Aryan, Ribeiro, Alejandro
Médium: Konferenčný príspevok..
Jazyk:English
Vydavateľské údaje: IEEE 01.12.2013
Predmet:
Approximation algorithms
Approximation methods
Convergence
Eigenvalues and eigenfunctions
Linear programming
Machine learning algorithms
Vectors
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:A regularized stochastic version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve optimization problems with stochastic objectives that arise in large scale machine learning. Stochastic gradient descent is the currently preferred solution methodology but the number of iterations required to approximate optimal arguments can be prohibitive in high dimensional problems. BFGS modifies gradient descent by introducing a Hessian approximation matrix computed from finite gradient differences. This paper utilizes stochastic gradient differences and introduces a regularization to ensure that the Hessian approximation matrix remains well conditioned. The resulting regularized stochastic BFGS method is shown to converge to optimal arguments almost surely over realizations of the stochastic gradient sequence. Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS.
DOI:10.1109/GlobalSIP.2013.6737088