An elementary approach to tight worst case complexity analysis of gradient based methods

This work presents a novel analysis that allows to achieve tight complexity bounds of gradient-based methods for convex optimization. We start by identifying some of the pitfalls rooted in the classical complexity analysis of the gradient descent method, and show how they can be remedied. Our method...

Full description

Saved in:
Bibliographic Details
Published in:Mathematical programming Vol. 201; no. 1-2; pp. 63 - 96
Main Authors: Teboulle, Marc, Vaisbourd, Yakov
Format: Journal Article
Language:English
Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.09.2023
Springer
Subjects:
Calculus of Variations and Optimal Control; Optimization
Combinatorics
Full Length Paper
Mathematical and Computational Physics
Mathematical Methods in Physics
Mathematics
Mathematics and Statistics
Mathematics of Computing
Numerical Analysis
Theoretical
Convex minimization
Gradient descent
68Q25
Performance estimation problem
Proximal schemes
90C30
Composite minimization
Global rate of convergence
90C25
90C60
Worst-case complexity analysis
ISSN:0025-5610, 1436-4646
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:This work presents a novel analysis that allows to achieve tight complexity bounds of gradient-based methods for convex optimization. We start by identifying some of the pitfalls rooted in the classical complexity analysis of the gradient descent method, and show how they can be remedied. Our methodology hinges on elementary and direct arguments in the spirit of the classical analysis. It allows us to establish some new (and reproduce known) tight complexity results for several fundamental algorithms including, gradient descent, proximal point and proximal gradient methods which previously could be proven only through computer-assisted convergence proof arguments.
ISSN:0025-5610
1436-4646
DOI:10.1007/s10107-022-01899-0