A Method to Solve overflow or underflow errors Resulting from Activation functions in Convolutional Neural Networks

In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponenti...

Full description

Saved in:
Bibliographic Details
Published in:International Information Institute (Tokyo). Information Vol. 20; no. 11; pp. 8151 - 8158
Main Authors: Jeong, JunMo, Choi, SeJin, Kim, ChiYong
Format: Journal Article
Language:English
Published: Koganei International Information Institute 01.11.2017
Subjects:
Activation
Algorithms
Artificial neural networks
Classification
Divergence
Entropy
Error correction
Methods
Neural networks
Neurons
ISSN:1343-4500, 1344-8994
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponential and log computations, they have a problem of overflow or underflow errors, which produce a range of numbers such as convergence to zero or infinite divergence when a value of the final output layer becomes too large. In the present paper, the above problem is solved by transforming an equation without affecting the functions of the conventional softmax and cross-entropy functions. Thus, training can be achieved to correcting the overflow or underflow error problem regardless of a size of values in the final output value.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1343-4500
1344-8994