Itao: a new iterative thresholding algorithm based optimizer for deep neural networks

In this paper, we propose a new iterative thresholding algorithm based optimizer (Itao) for deep neural networks. It is a first-order gradient-based algorithm with Tikhonov regularization for stochastic objective functions. It is fast and straightforward to implement. It acts on the parameters and t...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:IEEE access Ročník 10; s. 1
Hlavní autoři: Merrouchi, M., Atifi, K., Skittou, M., T.Gadi
Médium: Journal Article
Jazyk:angličtina
Vydáno: Piscataway IEEE 2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Témata:
Algorithms
Artificial neural networks
Back propagation
Back propagation networks
Deep learning
Inverse problems
Iterative algorithms
Iterative methods
Iterative thresholding algorithm based optimizer (Itao)
Linear programming
Machine learning
Mathematical models
Neural networks
Optimization
optimizer algorithm
Parameters
Regularization
Robustness (mathematics)
Training
ISSN:2169-3536, 2169-3536
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In this paper, we propose a new iterative thresholding algorithm based optimizer (Itao) for deep neural networks. It is a first-order gradient-based algorithm with Tikhonov regularization for stochastic objective functions. It is fast and straightforward to implement. It acts on the parameters and their gradients, with respect to the objective function, in only one step in the backpropagation system when training a neural network. This reduces the learning time and makes it well suited for neural networks with large parameters and/or large datasets. We have experimented this algorithm on several types of loss functions such as mean squared error, mean absolute error and categorical crossentropy. Different types of models such as regression and classification are studied. The robustness of this optimizer against the noisy labels is also verified. Many of the empirical results of conducted experiments in this study show that our optimizer works well in practice. It can outperform other state-of-the-art optimizers in terms of accuracy or at least give the same results in addition to the reduction of learning time.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:2169-3536
2169-3536
DOI:10.1109/ACCESS.2022.3222311