A multiobjective continuation method to compute the regularization path of deep neural networks

Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparse...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Machine learning with applications Ročník 19; s. 100625
Hlavní autori: Amakor, Augustina Chidinma, Sonntag, Konstantin, Peitz, Sebastian
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Elsevier Ltd 01.03.2025
Elsevier
Predmet:
Continuation method
Corrector
Deep neural network
High-dimensional
Multiobjective optimization
Non-smooth
Predictor
Proximal gradient
Regularization
Deep neural network
Predictor
Corrector
Multiobjective optimization
Proximal gradient
Non-smooth
Continuation method
Regularization
High-dimensional
ISSN:2666-8270, 2666-8270
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Sparsity is a highly desired feature in deep neural networks (DNNs) since it ensures numerical efficiency, improves the interpretability (due to the smaller number of relevant features), and robustness. For linear models, it is well known that there exists a regularization path connecting the sparsest solution in terms of the ℓ1 norm, i.e., zero weights and the non-regularized solution. Recently, there was a first attempt to extend the concept of regularization paths to DNNs by means of treating the empirical loss and sparsity (ℓ1 norm) as two conflicting criteria and solving the resulting multiobjective optimization problem. However, due to the non-smoothness of the ℓ1 norm and the large number of parameters, this approach is not very efficient from a computational perspective. To overcome this limitation, we present an algorithm that allows for the approximation of the entire Pareto front for the above-mentioned objectives in a very efficient manner for high-dimensional DNNs with millions of parameters. We present numerical examples using both deterministic and stochastic gradients. We furthermore demonstrate that knowledge of the regularization path allows for a well-generalizing network parametrization. •Extension of regularization paths to high-dimensional deep learning problems.•Demonstration of advantages of multiobjective continuation for generalization.•Increased resource efficiency by beginning with extremely sparse networks.
ISSN:2666-8270
2666-8270
DOI:10.1016/j.mlwa.2025.100625