Research on a learning rate with energy index in deep learning

The stochastic gradient descent algorithm (SGD) is the main optimization solution in deep learning. The performance of SGD depends critically on how learning rates are tuned over time. In this paper, we propose a novel energy index based optimization method (EIOM) to automatically adjust the learnin...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Neural networks Ročník 110; s. 225 - 231
Hlavní autoři: Zhao, Huizhen, Liu, Fuxian, Zhang, Han, Liang, Zhibing
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States Elsevier Ltd 01.02.2019
Témata:
Convolutional neural network
Deep learning
Energy index
Learning rate
Stochastic gradient algorithm
Deep learning
Energy index
Stochastic gradient algorithm
Convolutional neural network
Learning rate
ISSN:0893-6080, 1879-2782, 1879-2782
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:The stochastic gradient descent algorithm (SGD) is the main optimization solution in deep learning. The performance of SGD depends critically on how learning rates are tuned over time. In this paper, we propose a novel energy index based optimization method (EIOM) to automatically adjust the learning rate in the backpropagation. Since a frequently occurring feature is more important than a rarely occurring feature, we update the features to different extents according to their frequencies. We first define an energy neuron model and then design an energy index to describe the frequency of a feature. The learning rate is taken as a hyperparameter function according to the energy index. To empirically evaluate the EIOM, we investigate different optimizers with three popular machine learning models: logistic regression, multilayer perceptron, and convolutional neural network. The experiments demonstrate the promising performance of the proposed EIOM compared with that of other optimization algorithms.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0893-6080
1879-2782
1879-2782
DOI:10.1016/j.neunet.2018.12.009