On the Use of Stochastic Hessian Information in Optimization Methods for Machine Learning

This paper describes how to incorporate sampled curvature information in a Newton-CG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We foll...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:SIAM journal on optimization Ročník 21; číslo 3; s. 977 - 995
Hlavní autori: Byrd, Richard H., Chin, Gillian M., Neveitt, Will, Nocedal, Jorge
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Philadelphia Society for Industrial and Applied Mathematics 01.07.2011
Predmet:
Accuracy
Algorithms
Approximation
Computer engineering
Computer science
Machine learning
Methods
Motivation
Optimization techniques
Voice recognition
ISSN:1052-6234, 1095-7189
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:This paper describes how to incorporate sampled curvature information in a Newton-CG method and in a limited memory quasi-Newton method for statistical learning. The motivation for this work stems from supervised machine learning applications involving a very large number of training points. We follow a batch approach, also known in the stochastic optimization literature as a sample average approximation approach. Curvature information is incorporated in two subsampled Hessian algorithms, one based on a matrix-free inexact Newton iteration and one on a preconditioned limited memory BFGS iteration. A crucial feature of our technique is that Hessian-vector multiplications are carried out with a significantly smaller sample size than is used for the function and gradient. The efficiency of the proposed methods is illustrated using a machine learning application involving speech recognition. [PUBLICATION ABSTRACT]
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1052-6234
1095-7189
DOI:10.1137/10079923X