Fast curvature matrix-vector products for second-order gradient descent

We propose a generic method for iteratively approximating various second-order gradient steps - Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient - in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniq...

Full description

Saved in:

Bibliographic Details
Published in:	Neural computation Vol. 14; no. 7; p. 1723
Main Author:	Schraudolph, Nicol N
Format:	Journal Article
Language:	English
Published:	United States 01.07.2002
Subjects:	Acceleration Algorithms Neural Networks (Computer) Stochastic Processes
ISSN:	0899-7667
Online Access:	Get more information
Tags:	Add Tag No Tags, Be the first to tag this record!

Description
Summary:	We propose a generic method for iteratively approximating various second-order gradient steps - Newton, Gauss-Newton, Levenberg-Marquardt, and natural gradient - in linear time per iteration, using special curvature matrix-vector products that can be computed in O(n). Two recent acceleration techniques for on-line learning, matrix momentum and stochastic meta-descent (SMD), implement this approach. Since both were originally derived by very different routes, this offers fresh insight into their operation, resulting in further improvements to SMD.
Bibliography:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 23
ISSN:	0899-7667
DOI:	10.1162/08997660260028683