Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentia...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Mathematical programming Ročník 188; číslo 1; s. 19 - 51
Hlavní autori: Bolte, Jérôme, Pauwels, Edouard
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Berlin/Heidelberg Springer Berlin Heidelberg 01.07.2021
Springer Nature B.V
Predmet:
Algorithms
Back propagation
Calculus
Calculus of Variations and Optimal Control; Optimization
Combinatorics
Continuity (mathematics)
Deep learning
Differentiation
Full Length Paper
Machine learning
Mathematical analysis
Mathematical and Computational Physics
Mathematical Methods in Physics
Mathematics
Mathematics and Statistics
Mathematics of Computing
Numerical Analysis
Theoretical
o-Minimal structures
62M45 Neural nets and related approaches to inference from stochastic processes
90C06 Large-scale problems
68T05 Learning and adaptive systems in artificial intelligence
Stochastic gradient
Automatic differentiation
Deep learning
Backpropagation algorithm
Clarke subdifferential
49M27 Decomposition methods
First order methods
Nonsmooth stochastic optimization
Definable sets
65K10 Numerical optimization and variational techniques
49J53 Set-valued and variational analysis
ISSN:0025-5610, 1436-4646
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0025-5610
1436-4646
DOI:10.1007/s10107-020-01501-5