Riemannian Stochastic Variance-Reduced Cubic Regularized Newton Method for Submanifold Optimization

We propose a stochastic variance-reduced cubic regularized Newton algorithm to optimize the finite-sum problem over a Riemannian submanifold of the Euclidean space. The proposed algorithm requires a full gradient and Hessian update at the beginning of each epoch while it performs stochastic variance...

Full description

Saved in:
Bibliographic Details
Published in:Journal of optimization theory and applications Vol. 196; no. 1; pp. 324 - 361
Main Authors: Zhang, Dewei, Davanloo Tajbakhsh, Sam
Format: Journal Article
Language:English
Published: New York Springer US 01.01.2023
Springer Nature B.V
Subjects:
Algorithms
Applications of Mathematics
Calculus of Variations and Optimal Control; Optimization
Complexity
Eigenvalues
Engineering
Euclidean geometry
Euclidean space
Iterative methods
Manifolds (mathematics)
Mathematics
Mathematics and Statistics
Matrices (mathematics)
Newton methods
Numerical methods
Operations Research/Decision Theory
Optimization
Theory of Computation
Stochastic optimization
Cubic regularization
Manifold optimization
Variance reduction
Riemannian optimization
ISSN:0022-3239, 1573-2878
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We propose a stochastic variance-reduced cubic regularized Newton algorithm to optimize the finite-sum problem over a Riemannian submanifold of the Euclidean space. The proposed algorithm requires a full gradient and Hessian update at the beginning of each epoch while it performs stochastic variance-reduced updates in the iterations within each epoch. The iteration complexity of O ( ϵ - 3 / 2 ) to obtain an ( ϵ , ϵ ) -second-order stationary point, i.e., a point with the Riemannian gradient norm upper bounded by ϵ and minimum eigenvalue of Riemannian Hessian lower bounded by - ϵ , is established when the manifold is embedded in the Euclidean space. Furthermore, the paper proposes a computationally more appealing modification of the algorithm which only requires an inexact solution of the cubic regularized Newton subproblem with the same iteration complexity. The proposed algorithm is evaluated and compared with three other Riemannian second-order methods over two numerical studies on estimating the inverse scale matrix of the multivariate t-distribution on the manifold of symmetric positive definite matrices and estimating the parameter of a linear classifier on the sphere manifold.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0022-3239
1573-2878
DOI:10.1007/s10957-022-02137-5