Pruning SMAC search space based on key hyperparameters

Summary Machine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a cru...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Concurrency and computation Jg. 34; H. 9
Hauptverfasser: Li, Hui, Liang, Qingqing, Chen, Mei, Dai, Zhenyu, Li, Huanjun, Zhu, Ming
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Hoboken, USA John Wiley & Sons, Inc 25.04.2022
Wiley Subscription Services, Inc
Schlagworte:
Algorithms
AutoML
Configurations
hyperparameter optimization
key hyperparameters
Machine learning
Optimization
pruning
Regression models
Searching
SMAC
ISSN:1532-0626, 1532-0634
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Summary Machine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a crucial role to optimize hyperparameters of algorithms. Generally, the procedure tuning the exposed hyperparameters of ML algorithm to achieve better performance is called Hyperparameters Optimization. Traditional hyperparameters optimization methods are manually exhaustive search, which is unavailable for high dimensional search spaces and large datasets. Recent automated sequential model‐based optimization led to substantial improvements for this problem, whose core idea is fitting a regression model to describe the importance and dependence of algorithm's performance on certain given hyperparameter setting. Sequential model‐based algorithm configuration (SMAC) is a the‐state‐of‐art approach, which is specified by four components, Initialize, FitModel, SelectConfigurations, and Intensify. In this article, we propose to add a pruning procedure into SMAC approach, it quantifies the importance of hyperparameters by analyzing the performance of a list of promising configurations and reduces search space by discarding noncritical and bad key hyperparameters. To investigate the impact of pruning for model's performance, we conducted experiments on the configuration space constructed by Auto‐Sklearn and compared the effect of run time and pruning ratio with our algorithm. The experiments results verified that, our method made the configuration selected by SMAC more stable and achieved better performance.
Bibliographie:Funding information
Program for Innovative Talent of Guizhou Province (2017), Innovation Team of the Data Analysis Cloud Service of Guizhou Province, [2015]53; National Natural Science Foundation of China, 71964009; U1531246; 61562010; 61462012
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1532-0626
1532-0634
DOI:10.1002/cpe.5805