Pruning SMAC search space based on key hyperparameters

Summary Machine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a cru...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Concurrency and computation Ročník 34; číslo 9
Hlavní autori: Li, Hui, Liang, Qingqing, Chen, Mei, Dai, Zhenyu, Li, Huanjun, Zhu, Ming
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: Hoboken, USA John Wiley & Sons, Inc 25.04.2022
Wiley Subscription Services, Inc
Predmet:
Algorithms
AutoML
Configurations
hyperparameter optimization
key hyperparameters
Machine learning
Optimization
pruning
Regression models
Searching
SMAC
ISSN:1532-0626, 1532-0634
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Summary Machine learning (ML) has been widely applied in many areas in recent decades. However, because of its inherent complexity and characteristics, the efficiency and effectiveness of ML algorithm often to be heavily relies on the technical experts' experience and expertise which play a crucial role to optimize hyperparameters of algorithms. Generally, the procedure tuning the exposed hyperparameters of ML algorithm to achieve better performance is called Hyperparameters Optimization. Traditional hyperparameters optimization methods are manually exhaustive search, which is unavailable for high dimensional search spaces and large datasets. Recent automated sequential model‐based optimization led to substantial improvements for this problem, whose core idea is fitting a regression model to describe the importance and dependence of algorithm's performance on certain given hyperparameter setting. Sequential model‐based algorithm configuration (SMAC) is a the‐state‐of‐art approach, which is specified by four components, Initialize, FitModel, SelectConfigurations, and Intensify. In this article, we propose to add a pruning procedure into SMAC approach, it quantifies the importance of hyperparameters by analyzing the performance of a list of promising configurations and reduces search space by discarding noncritical and bad key hyperparameters. To investigate the impact of pruning for model's performance, we conducted experiments on the configuration space constructed by Auto‐Sklearn and compared the effect of run time and pruning ratio with our algorithm. The experiments results verified that, our method made the configuration selected by SMAC more stable and achieved better performance.
Bibliografia:Funding information
Program for Innovative Talent of Guizhou Province (2017), Innovation Team of the Data Analysis Cloud Service of Guizhou Province, [2015]53; National Natural Science Foundation of China, 71964009; U1531246; 61562010; 61462012
ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1532-0626
1532-0634
DOI:10.1002/cpe.5805