A Method to Solve overflow or underflow errors Resulting from Activation functions in Convolutional Neural Networks

In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponenti...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:International Information Institute (Tokyo). Information Ročník 20; číslo 11; s. 8151 - 8158
Hlavní autoři: Jeong, JunMo, Choi, SeJin, Kim, ChiYong
Médium: Journal Article
Jazyk:angličtina
Vydáno: Koganei International Information Institute 01.11.2017
Témata:
Activation
Algorithms
Artificial neural networks
Classification
Divergence
Entropy
Error correction
Methods
Neural networks
Neurons
ISSN:1343-4500, 1344-8994
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponential and log computations, they have a problem of overflow or underflow errors, which produce a range of numbers such as convergence to zero or infinite divergence when a value of the final output layer becomes too large. In the present paper, the above problem is solved by transforming an equation without affecting the functions of the conventional softmax and cross-entropy functions. Thus, training can be achieved to correcting the overflow or underflow error problem regardless of a size of values in the final output value.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1343-4500
1344-8994