A Method to Solve overflow or underflow errors Resulting from Activation functions in Convolutional Neural Networks - Jeong, JunMo; Choi, SeJin; Kim, ChiYong (International Information Institute (Tokyo). Information, 2017)
In this paper, a method is proposed to correct overflow errors of resulting values that can occur when the cross-entropy loss function is used as a softmax and ReLU as an activation function of the convolutional neural network. Since conventional softmax and cross-entropy functions include exponential and log computations, they have a problem of overflow or underflow errors, which produce a range of numbers such as convergence to zero or infinite divergence when a value of the final output layer becomes too large. In the present paper, the above problem is solved by transforming an equation without affecting the functions of the conventional softmax and cross-entropy functions. Thus, training can be achieved to correcting the overflow or underflow error problem regardless of a size of values in the final output value.