A Method to Solve overflow or underflow errors Resulting from Activation functions in Convolutional Neural Networks

In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponenti...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:International Information Institute (Tokyo). Information Ročník 20; číslo 11; s. 8151 - 8158
Hlavní autori: Jeong, JunMo, Choi, SeJin, Kim, ChiYong
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Koganei International Information Institute 01.11.2017
Predmet:
Activation
Algorithms
Artificial neural networks
Classification
Divergence
Entropy
Error correction
Methods
Neural networks
Neurons
ISSN:1343-4500, 1344-8994
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponential and log computations, they have a problem of overflow or underflow errors, which produce a range of numbers such as convergence to zero or infinite divergence when a value of the final output layer becomes too large. In the present paper, the above problem is solved by transforming an equation without affecting the functions of the conventional softmax and cross-entropy functions. Thus, training can be achieved to correcting the overflow or underflow error problem regardless of a size of values in the final output value.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1343-4500
1344-8994