Neural network with unbounded activation functions is universal approximator

This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorki...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Applied and computational harmonic analysis Ročník 43; číslo 2; s. 233 - 268
Hlavní autoři: Sonoda, Sho, Murata, Noboru
Médium: Journal Article
Jazyk:angličtina
Vydáno: Elsevier Inc 01.09.2017
Témata:
Admissibility condition
Backprojection filter
Bounded extension to [formula omitted]
Integral representation
Lizorkin distribution
Neural network
Radon transform
Rectified linear unit (ReLU)
Ridgelet transform
Universal approximation
Admissibility condition
Universal approximation
Radon transform
Lizorkin distribution
Neural network
Bounded extension to L2
Integral representation
Backprojection filter
Rectified linear unit (ReLU)
Ridgelet transform
ISSN:1063-5203, 1096-603X
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:This paper presents an investigation of the approximation property of neural networks with unbounded activation functions, such as the rectified linear unit (ReLU), which is the new de-facto standard of deep learning. The ReLU network can be analyzed by the ridgelet transform with respect to Lizorkin distributions. By showing three reconstruction formulas by using the Fourier slice theorem, the Radon transform, and Parseval's relation, it is shown that a neural network with unbounded activation functions still satisfies the universal approximation property. As an additional consequence, the ridgelet transform, or the backprojection filter in the Radon domain, is what the network learns after backpropagation. Subject to a constructive admissibility condition, the trained network can be obtained by simply discretizing the ridgelet transform, without backpropagation. Numerical examples not only support the consistency of the admissibility condition but also imply that some non-admissible cases result in low-pass filtering.
ISSN:1063-5203
1096-603X
DOI:10.1016/j.acha.2015.12.005