Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers

Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process of designing HPO algorithms is still an unsystematic and manual process: New algorithms are often built on top of prior work, wher...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on evolutionary computation Jg. 26; H. 6; S. 1
Hauptverfasser: Moosbauer, Julia, Binder, Martin, Schneider, Lennart, Pfisterer, Florian, Becker, Marc, Lang, Michel, Kotthoff, Lars, Bischl, Bernd
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York IEEE 01.12.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Schlagworte:
Ablation
algorithm analysis
Algorithm design
Algorithms
Approximation algorithms
automated machine learning
Automation
Bayes methods
Benchmarks
Configurations
Design optimization
hyperparameter optimization
Machine learning
Machine learning algorithms
Mathematical models
multifidelity
Optimization
Prediction algorithms
Software algorithms
ISSN:1089-778X, 1941-0026
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process of designing HPO algorithms is still an unsystematic and manual process: New algorithms are often built on top of prior work, where limitations are identified and improvements are proposed. Even though this approach is guided by expert knowledge, it is still somewhat arbitrary. The process rarely allows for gaining a holistic understanding of which algorithmic components drive performance and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multi-fidelity HPO (MF-HPO). First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to, common existing HPO algorithms and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration (in some ways, simpler than established methods) performs very well as long as some critical configuration parameters are set to the right value.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1089-778X
1941-0026
DOI:10.1109/TEVC.2022.3211336