Conservative set valued fields, automatic differentiation, stochastic gradient methods and deep learning

Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentia...

Full description

Saved in:
Bibliographic Details
Published in:Mathematical programming Vol. 188; no. 1; pp. 19 - 51
Main Authors: Bolte, Jérôme, Pauwels, Edouard
Format: Journal Article
Language:English
Published: Berlin/Heidelberg Springer Berlin Heidelberg 01.07.2021
Springer Nature B.V
Subjects:
Algorithms
Back propagation
Calculus
Calculus of Variations and Optimal Control; Optimization
Combinatorics
Continuity (mathematics)
Deep learning
Differentiation
Full Length Paper
Machine learning
Mathematical analysis
Mathematical and Computational Physics
Mathematical Methods in Physics
Mathematics
Mathematics and Statistics
Mathematics of Computing
Numerical Analysis
Theoretical
o-Minimal structures
62M45 Neural nets and related approaches to inference from stochastic processes
90C06 Large-scale problems
68T05 Learning and adaptive systems in artificial intelligence
Stochastic gradient
Automatic differentiation
Deep learning
Backpropagation algorithm
Clarke subdifferential
49M27 Decomposition methods
First order methods
Nonsmooth stochastic optimization
Definable sets
65K10 Numerical optimization and variational techniques
49J53 Set-valued and variational analysis
ISSN:0025-5610, 1436-4646
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Modern problems in AI or in numerical analysis require nonsmooth approaches with a flexible calculus. We introduce generalized derivatives called conservative fields for which we develop a calculus and provide representation formulas. Functions having a conservative field are called path differentiable: convex, concave, Clarke regular and any semialgebraic Lipschitz continuous functions are path differentiable. Using Whitney stratification techniques for semialgebraic and definable sets, our model provides variational formulas for nonsmooth automatic differentiation oracles, as for instance the famous backpropagation algorithm in deep learning. Our differential model is applied to establish the convergence in values of nonsmooth stochastic gradient methods as they are implemented in practice.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0025-5610
1436-4646
DOI:10.1007/s10107-020-01501-5