Regularized stochastic BFGS algorithm

A regularized stochastic version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve optimization problems with stochastic objectives that arise in large scale machine learning. Stochastic gradient descent is the currently preferred solution methodology but the n...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:2013 IEEE Global Conference on Signal and Information Processing (GlobalSIP) s. 1109 - 1112
Hlavní autoři: Mokhtari, Aryan, Ribeiro, Alejandro
Médium: Konferenční příspěvek
Jazyk:angličtina
Vydáno: IEEE 01.12.2013
Témata:
Approximation algorithms
Approximation methods
Convergence
Eigenvalues and eigenfunctions
Linear programming
Machine learning algorithms
Vectors
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:A regularized stochastic version of the Broyden-Fletcher- Goldfarb-Shanno (BFGS) quasi-Newton method is proposed to solve optimization problems with stochastic objectives that arise in large scale machine learning. Stochastic gradient descent is the currently preferred solution methodology but the number of iterations required to approximate optimal arguments can be prohibitive in high dimensional problems. BFGS modifies gradient descent by introducing a Hessian approximation matrix computed from finite gradient differences. This paper utilizes stochastic gradient differences and introduces a regularization to ensure that the Hessian approximation matrix remains well conditioned. The resulting regularized stochastic BFGS method is shown to converge to optimal arguments almost surely over realizations of the stochastic gradient sequence. Numerical experiments showcase reductions in convergence time relative to stochastic gradient descent algorithms and non-regularized stochastic versions of BFGS.
DOI:10.1109/GlobalSIP.2013.6737088