Research on a learning rate with energy index in deep learning

The stochastic gradient descent algorithm (SGD) is the main optimization solution in deep learning. The performance of SGD depends critically on how learning rates are tuned over time. In this paper, we propose a novel energy index based optimization method (EIOM) to automatically adjust the learnin...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Neural networks Ročník 110; s. 225 - 231
Hlavní autori: Zhao, Huizhen, Liu, Fuxian, Zhang, Han, Liang, Zhibing
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: United States Elsevier Ltd 01.02.2019
Predmet:
Convolutional neural network
Deep learning
Energy index
Learning rate
Stochastic gradient algorithm
Deep learning
Energy index
Stochastic gradient algorithm
Convolutional neural network
Learning rate
ISSN:0893-6080, 1879-2782, 1879-2782
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:The stochastic gradient descent algorithm (SGD) is the main optimization solution in deep learning. The performance of SGD depends critically on how learning rates are tuned over time. In this paper, we propose a novel energy index based optimization method (EIOM) to automatically adjust the learning rate in the backpropagation. Since a frequently occurring feature is more important than a rarely occurring feature, we update the features to different extents according to their frequencies. We first define an energy neuron model and then design an energy index to describe the frequency of a feature. The learning rate is taken as a hyperparameter function according to the energy index. To empirically evaluate the EIOM, we investigate different optimizers with three popular machine learning models: logistic regression, multilayer perceptron, and convolutional neural network. The experiments demonstrate the promising performance of the proposed EIOM compared with that of other optimization algorithms.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0893-6080
1879-2782
1879-2782
DOI:10.1016/j.neunet.2018.12.009