Automated Benchmark-Driven Design and Explanation of Hyperparameter Optimizers

Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process of designing HPO algorithms is still an unsystematic and manual process: New algorithms are often built on top of prior work, wher...

Full description

Saved in:
Bibliographic Details
Published in:IEEE transactions on evolutionary computation Vol. 26; no. 6; p. 1
Main Authors: Moosbauer, Julia, Binder, Martin, Schneider, Lennart, Pfisterer, Florian, Becker, Marc, Lang, Michel, Kotthoff, Lars, Bischl, Bernd
Format: Journal Article
Language:English
Published: New York IEEE 01.12.2022
The Institute of Electrical and Electronics Engineers, Inc. (IEEE)
Subjects:
Ablation
algorithm analysis
Algorithm design
Algorithms
Approximation algorithms
automated machine learning
Automation
Bayes methods
Benchmarks
Configurations
Design optimization
hyperparameter optimization
Machine learning
Machine learning algorithms
Mathematical models
multifidelity
Optimization
Prediction algorithms
Software algorithms
ISSN:1089-778X, 1941-0026
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Automated hyperparameter optimization (HPO) has gained great popularity and is an important component of most automated machine learning frameworks. However, the process of designing HPO algorithms is still an unsystematic and manual process: New algorithms are often built on top of prior work, where limitations are identified and improvements are proposed. Even though this approach is guided by expert knowledge, it is still somewhat arbitrary. The process rarely allows for gaining a holistic understanding of which algorithmic components drive performance and carries the risk of overlooking good algorithmic design choices. We present a principled approach to automated benchmark-driven algorithm design applied to multi-fidelity HPO (MF-HPO). First, we formalize a rich space of MF-HPO candidates that includes, but is not limited to, common existing HPO algorithms and then present a configurable framework covering this space. To find the best candidate automatically and systematically, we follow a programming-by-optimization approach and search over the space of algorithm candidates via Bayesian optimization. We challenge whether the found design choices are necessary or could be replaced by more naive and simpler ones by performing an ablation analysis. We observe that using a relatively simple configuration (in some ways, simpler than established methods) performs very well as long as some critical configuration parameters are set to the right value.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1089-778X
1941-0026
DOI:10.1109/TEVC.2022.3211336