Stochastic Bigger Subspace Algorithms for Nonconvex Stochastic Optimization

It is well known that the stochastic optimization problem can be regarded as one of the most hard problems since, in most of the cases, the values of f and its gradient are often not easily to be solved, or the F(∙, ξ) is normally not given clearly and (or) the distribution function P is equivocal....

Full description

Saved in:
Bibliographic Details
Published in:IEEE access Vol. 9; p. 1
Main Authors: Yuan, Gonglin, Zhou, Yingjie, Wang, Liping, Yang, Qingyuan
Format: Journal Article
Language:English
Published: Piscataway IEEE 01.01.2021
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Approximation algorithms
Complexity
Complexity analysis
Complexity theory
Convergence
Convergence property
Distribution functions
Machine learning
Machine learning algorithms
Nonconvex function
Optimization
Quasi Newton methods
Random variables
Stochastic processes
Stochastic subspace algorithm
ISSN:2169-3536, 2169-3536
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:It is well known that the stochastic optimization problem can be regarded as one of the most hard problems since, in most of the cases, the values of f and its gradient are often not easily to be solved, or the F(∙, ξ) is normally not given clearly and (or) the distribution function P is equivocal. Then an effective optimization algorithm is successfully designed and used to solve this problem that is an interesting work. This paper designs stochastic bigger subspace algorithms for solving nonconvex stochastic optimization problems. A general framework for such algorithm is presented for convergence analysis, where the so-called the sufficient descent property, the trust region feature, and the global convergence of the stationary points are proved under the suitable conditions. In the worst-case, we will turn out that the complexity is competitive under a given accuracy parameter. We will proved that the SFO-calls complexity of the presented algorithm with diminishing steplength is O(ϵ-1/1-β) and the SFO-calls complexity of the given algorithm with random constant steplength is O(ϵ-2) respectively, where β ∈ (0.5,1) and ϵ is accuracy and the needed conditions are weaker than the quasi-Newton methods and the normal conjugate gradient algorithms. The detail algorithm framework with variance reduction is also proposed for experiments and the nonconvex binary classification problem is done to demonstrate the performance of the given algorithm.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2021.3108418