Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference....

Full description

Saved in:
Bibliographic Details
Published in:Mathematical biosciences and engineering : MBE Vol. 21; no. 6; pp. 6289 - 6335
Main Authors: Tan, Jia Mian, Liao, Haoran, Liu, Wei, Fan, Changjun, Huang, Jincai, Liu, Zhong, Yan, Junchi
Format: Journal Article
Language:English
Published: United States AIMS Press 14.06.2024
Subjects:
bayesian optimization
deep neural networks
hyperparameter optimization
machine learning
survey
survey
hyperparameter optimization
machine learning
bayesian optimization
deep neural networks
ISSN:1551-0018, 1551-0018
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference. The primary objective is to mitigate the challenges associated with manual hyperparameter tuning, which can be ad-hoc, reliant on human expertise, and consequently hinders reproducibility while inflating deployment costs. Recognizing the growing significance of HPO, this paper surveyed classical HPO methods, approaches for accelerating the optimization process, HPO in an online setting (dynamic algorithm configuration, DAC), and when there is more than one objective to optimize (multi-objective HPO). Acceleration strategies were categorized into multi-fidelity, bandit-based, and early stopping; DAC algorithms encompassed gradient-based, population-based, and reinforcement learning-based methods; multi-objective HPO can be approached via scalarization, metaheuristics, and model-based algorithms tailored for multi-objective situation. A tabulated overview of popular frameworks and tools for HPO was provided, catering to the interests of practitioners.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1551-0018
1551-0018
DOI:10.3934/mbe.2024275