Hyperparameter optimization: Classics, acceleration, online, multi-objective, and tools

Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference....

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Mathematical biosciences and engineering : MBE Ročník 21; číslo 6; s. 6289 - 6335
Hlavní autoři: Tan, Jia Mian, Liao, Haoran, Liu, Wei, Fan, Changjun, Huang, Jincai, Liu, Zhong, Yan, Junchi
Médium: Journal Article
Jazyk:angličtina
Vydáno: United States AIMS Press 14.06.2024
Témata:
bayesian optimization
deep neural networks
hyperparameter optimization
machine learning
survey
survey
hyperparameter optimization
machine learning
bayesian optimization
deep neural networks
ISSN:1551-0018, 1551-0018
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:Hyperparameter optimization (HPO) has been well-developed and evolved into a well-established research topic over the decades. With the success and wide application of deep learning, HPO has garnered increased attention, particularly within the realm of machine learning model training and inference. The primary objective is to mitigate the challenges associated with manual hyperparameter tuning, which can be ad-hoc, reliant on human expertise, and consequently hinders reproducibility while inflating deployment costs. Recognizing the growing significance of HPO, this paper surveyed classical HPO methods, approaches for accelerating the optimization process, HPO in an online setting (dynamic algorithm configuration, DAC), and when there is more than one objective to optimize (multi-objective HPO). Acceleration strategies were categorized into multi-fidelity, bandit-based, and early stopping; DAC algorithms encompassed gradient-based, population-based, and reinforcement learning-based methods; multi-objective HPO can be approached via scalarization, metaheuristics, and model-based algorithms tailored for multi-objective situation. A tabulated overview of popular frameworks and tools for HPO was provided, catering to the interests of practitioners.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1551-0018
1551-0018
DOI:10.3934/mbe.2024275