Exploration and generalization in deep learning with SwitchPath activations

This work provides a comprehensive theoretical and empirical analysis of SwitchPath, a stochastic activation function that improves learning dynamics by probabilistically toggling between a neuron standard activation and its negation. We develop theoretical foundations and demonstrate its impact in...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Machine learning Ročník 114; číslo 9; s. 200
Hlavní autoři: Di Cecco, Antonio, Papini, Andrea, Metta, Carlo, Fantozzi, Marco, Galfrè, Silvia Giulia, Morandin, Francesco, Parton, Maurizio
Médium: Journal Article
Jazyk:angličtina
Vydáno: Dordrecht Springer Nature B.V 01.09.2025
Témata:
Adaptability
Algorithms
Back propagation
Decision making
Decision trees
Deep learning
Empirical analysis
Generative networks
Gradient flow
Machine learning
Neural network algorithms
Neural networks
Neurons
Regularization
ISSN:0885-6125, 1573-0565, 1573-0565
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:This work provides a comprehensive theoretical and empirical analysis of SwitchPath, a stochastic activation function that improves learning dynamics by probabilistically toggling between a neuron standard activation and its negation. We develop theoretical foundations and demonstrate its impact in multiple scenarios. By maintaining gradient flow and injecting controlled stochasticity, the method improves generalization, uncertainty estimation, and training efficiency. Experiments in classification show consistent gains over ReLU and Leaky ReLU across CNNs and Vision Transformers, with reduced overfitting and better test accuracy. In generative modeling, a novel two-phase training scheme significantly mitigates mode collapse and accelerates convergence. Our theoretical analysis reveals that SwitchPath introduces a form of multiplicative noise that acts as a structural regularizer. Additional empirical investigations show improved information propagation and reduced model complexity. These results establish this activation mechanism as a simple yet effective way to enhance exploration, regularization, and reliability in modern neural networks.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-6125
1573-0565
1573-0565
DOI:10.1007/s10994-025-06840-y