Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference....

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Mathematical biosciences and engineering : MBE Ročník 21; číslo 6; s. 6289 - 6335
Hlavní autori: Tan, Jia Mian, Liao, Haoran, Liu, Wei, Fan, Changjun, Huang, Jincai, Liu, Zhong, Yan, Junchi
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: United States AIMS Press 14.06.2024
Predmet:
bayesian optimization
deep neural networks
hyperparameter optimization
machine learning
survey
survey
hyperparameter optimization
machine learning
bayesian optimization
deep neural networks
ISSN:1551-0018, 1551-0018
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference. The primary objective is to mitigate the challenges associated with manual hyperparameter tuning, which can be ad-hoc, reliant on human expertise, and consequently hinders reproducibility while inflating deployment costs. Recognizing the growing significance of HPO, this paper surveyed classical HPO methods, approaches for accelerating the optimization process, HPO in an online setting (dynamic algorithm configuration, DAC), and when there is more than one objective to optimize (multi-objective HPO). Acceleration strategies were categorized into multi-fidelity, bandit-based, and early stopping; DAC algorithms encompassed gradient-based, population-based, and reinforcement learning-based methods; multi-objective HPO can be approached via scalarization, metaheuristics, and model-based algorithms tailored for multi-objective situation. A tabulated overview of popular frameworks and tools for HPO was provided, catering to the interests of practitioners.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1551-0018
1551-0018
DOI:10.3934/mbe.2024275