Parallel and distributed asynchronous adaptive stochastic gradient methods

Stochastic gradient methods (SGMs) are the predominant approaches to train deep learning models. The adaptive versions (e.g., Adam and AMSGrad) have been extensively used in practice, partly because they achieve faster convergence than the non-adaptive versions while incurring little overhead. On th...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Mathematical programming computation Ročník 15; číslo 3; s. 471 - 508
Hlavní autoři:	Xu, Yangyang, Xu, Yibo, Yan, Yonggui, Sutcher-Shepard, Colin, Grinberg, Leopold, Chen, Jie
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	Berlin/Heidelberg Springer Berlin Heidelberg 01.09.2023 Springer Nature B.V
Témata:	Convergence Deep learning Full Length Paper Machine learning Mathematics Mathematics and Statistics Mathematics of Computing Operations Research/Decision Theory Optimization Staling Theory of Computation Deep learning 65K05 Stochastic gradient method 90C15 68W15 Adaptive learning rate 65Y05
ISSN:	1867-2949, 1867-2957
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Popis
Shrnutí:	Stochastic gradient methods (SGMs) are the predominant approaches to train deep learning models. The adaptive versions (e.g., Adam and AMSGrad) have been extensively used in practice, partly because they achieve faster convergence than the non-adaptive versions while incurring little overhead. On the other hand, asynchronous (async) parallel computing has exhibited significantly higher speed-up over its synchronous (sync) counterpart. Async-parallel non-adaptive SGMs have been well studied in the literature from the perspectives of both theory and practical performance. Adaptive SGMs can also be implemented without much difficulty in an async-parallel way. However, to the best of our knowledge, no theoretical result of async-parallel adaptive SGMs has been established. The difficulty for analyzing adaptive SGMs with async updates originates from the second moment term. In this paper, we propose an async-parallel adaptive SGM based on AMSGrad. We show that the proposed method inherits the convergence guarantee of AMSGrad for both convex and non-convex problems, if the staleness (also called delay) caused by asynchrony is bounded. Our convergence rate results indicate a nearly linear parallelization speed-up if τ = o ( K 1 4 ) , where τ is the staleness and K is the number of iterations. The proposed method is tested on both convex and non-convex machine learning problems, and the numerical results demonstrate its clear advantages over the sync counterpart and the async-parallel nonadaptive SGM. Our code has been released at https://github.com/RPI-OPT/APAM .
Bibliografie:	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ISSN:	1867-2949 1867-2957
DOI:	10.1007/s12532-023-00237-5