Convergence analysis of stochastic higher-order majorization-minimization algorithms

Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective valu...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Optimization methods & software Ročník 39; číslo 2; s. 384 - 413
Hlavní autoři: Lupu, Daniela, Necoara, Ion
Médium: Journal Article
Jazyk:angličtina
Vydáno: Abingdon Taylor & Francis 03.03.2024
Taylor & Francis Ltd
Témata:
Algorithms
Approximation
Convergence
convergence rates
Convexity
Finite-sum optimization
Iterative methods
majorization-minimization
minibatch
Optimization
stochastic higher-order algorithms
Upper bounds
ISSN:1055-6788, 1029-4937
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Majorization-minimization schemes are a broad class of iterative methods targeting general optimization problems, including nonconvex, nonsmooth and stochastic. These algorithms minimize successively a sequence of upper bounds of the objective function so that along the iterations the objective value decreases. We present a stochastic higher-order algorithmic framework for minimizing the average of a very large number of sufficiently smooth functions. Our stochastic framework is based on the notion of stochastic higher-order upper bound approximations of the finite-sum objective function and minibatching. We derive convergence results for nonconvex and convex optimization problems when the higher-order approximation of the objective function yields an error that is p times differentiable and has Lipschitz continuous p derivative. More precisely, for general nonconvex problems we present asymptotic stationary point guarantees and under Kurdyka-Lojasiewicz property we derive local convergence rates ranging from sublinear to linear. For convex problems with uniformly convex objective function, we derive local (super)linear convergence results for our algorithm. Compared to existing stochastic (first-order) methods, our algorithm adapts to the problem's curvature and allows using any batch size. Preliminary numerical tests support the effectiveness of our algorithmic framework.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1055-6788
1029-4937
DOI:10.1080/10556788.2023.2256447