Noise-enhanced convolutional neural networks

Injecting carefully chosen noise can speed convergence in the backpropagation training of a convolutional neural network (CNN). The Noisy CNN algorithm speeds training on average because the backpropagation algorithm is a special case of the generalized expectation–maximization (EM) algorithm and be...

Full description

Saved in:
Bibliographic Details
Published in:Neural networks Vol. 78; pp. 15 - 23
Main Authors: Audhkhasi, Kartik, Osoba, Osonde, Kosko, Bart
Format: Journal Article
Language:English
Published: United States Elsevier Ltd 01.06.2016
Subjects:
Algorithms
Back propagation
Backpropagation
Convolutional neural network
Expectation–maximization algorithm
Hyperplanes
Learning - physiology
Likelihood Functions
Neural networks
Neural Networks (Computer)
Noise
Noise injection
Normal Distribution
Random Allocation
Sampling from big data sets
Stochastic resonance
Training
Backpropagation
Noise injection
Sampling from big data sets
Stochastic resonance
Expectation–maximization algorithm
Convolutional neural network
ISSN:0893-6080, 1879-2782, 1879-2782
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Injecting carefully chosen noise can speed convergence in the backpropagation training of a convolutional neural network (CNN). The Noisy CNN algorithm speeds training on average because the backpropagation algorithm is a special case of the generalized expectation–maximization (EM) algorithm and because such carefully chosen noise always speeds up the EM algorithm on average. The CNN framework gives a practical way to learn and recognize images because backpropagation scales with training data. It has only linear time complexity in the number of training samples. The Noisy CNN algorithm finds a special separating hyperplane in the network’s noise space. The hyperplane arises from the likelihood-based positivity condition that noise-boosts the EM algorithm. The hyperplane cuts through a uniform-noise hypercube or Gaussian ball in the noise space depending on the type of noise used. Noise chosen from above the hyperplane speeds training on average. Noise chosen from below slows it on average. The algorithm can inject noise anywhere in the multilayered network. Adding noise to the output neurons reduced the average per-iteration training-set cross entropy by 39% on a standard MNIST image test set of handwritten digits. It also reduced the average per-iteration training-set classification error by 47%. Adding noise to the hidden layers can also reduce these performance measures. The noise benefit is most pronounced for smaller data sets because the largest EM hill-climbing gains tend to occur in the first few iterations. This noise effect can assist random sampling from large data sets because it allows a smaller random sample to give the same or better performance than a noiseless sample gives.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:0893-6080
1879-2782
1879-2782
DOI:10.1016/j.neunet.2015.09.014