Optimization of non-smooth functions via differentiable surrogates

Mathematical optimization is fundamental across many scientific and engineering applications. While data-driven models like gradient boosting and random forests excel at prediction tasks, they often lack mathematical regularity, being non-differentiable or even discontinuous. These models are common...

Full description

Saved in:
Bibliographic Details
Published in:PloS one Vol. 20; no. 5; p. e0321862
Main Authors: Chen, Shikun, Huang, Zebin, Zheng, Wenlong
Format: Journal Article
Language:English
Published: United States Public Library of Science 30.05.2025
Public Library of Science (PLoS)
Subjects:
Accuracy
Algorithms
Analysis
Approximation
Benchmarks
Constraints
Cost control
Efficiency
Energy consumption
Genetic algorithms
Heuristic
Machine learning
Mathematical analysis
Mathematical functions
Mathematical models
Mathematical optimization
Medical informatics
Methods
Models, Theoretical
Neural networks
Neural Networks, Computer
Optimization
Parameters
Predictions
Simulation
Variables
China
ISSN:1932-6203, 1932-6203
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Mathematical optimization is fundamental across many scientific and engineering applications. While data-driven models like gradient boosting and random forests excel at prediction tasks, they often lack mathematical regularity, being non-differentiable or even discontinuous. These models are commonly used to predict outputs based on a combination of fixed parameters and adjustable variables. A key transition in optimization involves moving beyond simple prediction to determine optimal variable values. Specifically, the challenge lies in identifying values of adjustable variables that maximize the output quality according to the model’s predictions, given a set of fixed parameters. To address this challenge, we propose a method that combines XGBoost’s superior prediction accuracy with neural networks’ differentiability as optimization surrogates. The approach leverages gradient information from neural networks to guide SLSQP optimization while maintaining XGBoost’s prediction precision. Through extensive testing on classical optimization benchmarks including Rosenbrock, Levy, and Rastrigin functions with varying dimensions and constraint conditions, we demonstrate that our method achieves solutions up to 40% better than traditional methods while reducing computation time by orders of magnitude. The framework consistently maintains near-zero constraint violations across all test cases, even as problem complexity increases. This approach bridges the gap between model accuracy and optimization efficiency, offering a practical solution for optimizing non-differentiable machine learning models that can be extended to other tree-based ensemble algorithms. The method has been successfully applied to real-world steel alloy optimization, where it achieved superior performance while maintaining all metallurgical composition constraints.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:1932-6203
1932-6203
DOI:10.1371/journal.pone.0321862