KAF + RSigELU: a nonlinear and kernel-based activation function for deep neural networks

Activation functions (AFs) are the basis for neural network architectures used in real-world problems to accurately model and learn complex relationships between variables. They are preferred to process the input information coming to the network and to produce the corresponding output. The kernel-b...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Neural computing & applications Jg. 34; H. 16; S. 13909 - 13923
Hauptverfasser: Kiliçarslan, Serhat, Celik, Mete
Format: Journal Article
Sprache:Englisch
Veröffentlicht: London Springer London 01.08.2022
Springer Nature B.V
Schlagworte:
Artificial Intelligence
Artificial neural networks
Complex variables
Computational Biology/Bioinformatics
Computational Science and Engineering
Computer architecture
Computer Science
Data Mining and Knowledge Discovery
Image Processing and Computer Vision
Kernel functions
Machine learning
Neural networks
Original Article
Performance evaluation
Probability and Statistics in Computer Science
Process parameters
Kernel-based activation function (KAF)
CNN
Deep neural network
KAF + RSigELUS
KAF + RSigELUD
ISSN:0941-0643, 1433-3058
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Activation functions (AFs) are the basis for neural network architectures used in real-world problems to accurately model and learn complex relationships between variables. They are preferred to process the input information coming to the network and to produce the corresponding output. The kernel-based activation function (KAF) offers an extended version of ReLU and sigmoid AFs. Therefore, KAF faced with the problems of bias shift originating from the negative region, vanishing gradient, adaptability, flexibility, and neuron death in parameters during the learning process. In this study, hybrid KAF + RSigELUS and KAF + RSigELUD AFs, which are extended versions of KAF, are proposed. In the proposed AFs, the gauss kernel function is used. The proposed KAF + RSigELUS and KAF + RSigELUD AFs are effective in the positive, negative, and linear activation regions. Performance evaluations of them were conducted on the MNIST, Fashion MNIST, CIFAR-10, and SVHN benchmark datasets. The experimental evaluations show that the proposed AFs overcome existing problems and outperformed ReLU, LReLU, ELU, PReLU, and KAF AFs.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0941-0643
1433-3058
DOI:10.1007/s00521-022-07211-7