A Method to Solve overflow or underflow errors Resulting from Activation functions in Convolutional Neural Networks

In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponenti...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:International Information Institute (Tokyo). Information Jg. 20; H. 11; S. 8151 - 8158
Hauptverfasser: Jeong, JunMo, Choi, SeJin, Kim, ChiYong
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Koganei International Information Institute 01.11.2017
Schlagworte:
Activation
Algorithms
Artificial neural networks
Classification
Divergence
Entropy
Error correction
Methods
Neural networks
Neurons
ISSN:1343-4500, 1344-8994
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponential and log computations, they have a problem of overflow or underflow errors, which produce a range of numbers such as convergence to zero or infinite divergence when a value of the final output layer becomes too large. In the present paper, the above problem is solved by transforming an equation without affecting the functions of the conventional softmax and cross-entropy functions. Thus, training can be achieved to correcting the overflow or underflow error problem regardless of a size of values in the final output value.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1343-4500
1344-8994