A Universal Approximation Result for Difference of Log-Sum-Exp Neural Networks

We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node, referred to as log-sum-exp (LSE) network, is a smooth universal approxi...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transaction on neural networks and learning systems Vol. 31; no. 12; pp. 5603 - 5612
Main Authors: Calafiore, Giuseppe C., Gaubert, Stephane, Possieri, Corrado
Format: Journal Article
Language:English
Published: United States IEEE 01.12.2020
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Algorithms
Artificial neural networks
Computer Science
Continuity (mathematics)
Convex functions
Data-driven optimization
Design
Design optimization
Diabetes mellitus
difference of convex (DC) programming
Feedforward neural networks
feedforward neural networks (FFNs)
log-sum-exp (LSE) networks
Machine Learning
Mathematical analysis
Mathematical models
Mathematics
Neural networks
Numerical methods
Optimization
Optimization and Control
Subtraction
subtraction-free expressions
surrogate models
Transforms
universal approximation
LSE networks
DC programming
Subtraction-freeexpressions
Surrogate models
Feedforward neural networks
Data-driven optimization
Universal ap-proximation
ISSN:2162-237X, 2162-2388, 2162-2388
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:We show that a neural network whose output is obtained as the difference of the outputs of two feedforward networks with exponential activation function in the hidden layer and logarithmic activation function in the output node, referred to as log-sum-exp (LSE) network, is a smooth universal approximator of continuous functions over convex, compact sets. By using a logarithmic transform, this class of network maps to a family of subtraction-free ratios of generalized posynomials (GPOS), which we also show to be universal approximators of positive functions over log-convex, compact subsets of the positive orthant. The main advantage of difference-LSE networks with respect to classical feedforward neural networks is that, after a standard training phase, they provide surrogate models for a design that possesses a specific difference-of-convex-functions form, which makes them optimizable via relatively efficient numerical methods. In particular, by adapting an existing difference-of-convex algorithm to these models, we obtain an algorithm for performing an effective optimization-based design. We illustrate the proposed approach by applying it to the data-driven design of a diet for a patient with type-2 diabetes and to a nonconvex optimization problem.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:2162-237X
2162-2388
2162-2388
DOI:10.1109/TNNLS.2020.2975051