Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference....

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Mathematical biosciences and engineering : MBE Jg. 21; H. 6; S. 6289 - 6335
Hauptverfasser: Tan, Jia Mian, Liao, Haoran, Liu, Wei, Fan, Changjun, Huang, Jincai, Liu, Zhong, Yan, Junchi
Format: Journal Article
Sprache:Englisch
Veröffentlicht: United States AIMS Press 14.06.2024
Schlagworte:
bayesian optimization
deep neural networks
hyperparameter optimization
machine learning
survey
survey
hyperparameter optimization
machine learning
bayesian optimization
deep neural networks
ISSN:1551-0018, 1551-0018
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference. The primary objective is to mitigate the challenges associated with manual hyperparameter tuning, which can be ad-hoc, reliant on human expertise, and consequently hinders reproducibility while inflating deployment costs. Recognizing the growing significance of HPO, this paper surveyed classical HPO methods, approaches for accelerating the optimization process, HPO in an online setting (dynamic algorithm configuration, DAC), and when there is more than one objective to optimize (multi-objective HPO). Acceleration strategies were categorized into multi-fidelity, bandit-based, and early stopping; DAC algorithms encompassed gradient-based, population-based, and reinforcement learning-based methods; multi-objective HPO can be approached via scalarization, metaheuristics, and model-based algorithms tailored for multi-objective situation. A tabulated overview of popular frameworks and tools for HPO was provided, catering to the interests of practitioners.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1551-0018
1551-0018
DOI:10.3934/mbe.2024275