Speeding up algorithm selection using average ranking and active testing by introducing runtime

Algorithm selection methods can be speeded-up substantially by incorporating multi-objective measures that give preference to algorithms that are both promising and fast to evaluate. In this paper, we introduce such a measure, A3R, and incorporate it into two algorithm selection techniques: average...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Machine learning Jg. 107; H. 1; S. 79 - 108
Hauptverfasser: Abdulrahman, Salisu Mamman, Brazdil, Pavel, van Rijn, Jan N., Vanschoren, Joaquin
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York Springer US 01.01.2018
Springer Nature B.V
Schlagworte:
Accuracy
Algorithms
Artificial Intelligence
Computer Science
Control
Machine learning
Mechatronics
Natural Language Processing (NLP)
Ranking
Robotics
Run time (computers)
Simulation and Modeling
Special Issue on Metalearning and Algorithm Selection
Meta-learning
Average ranking
Algorithm selection
Active testing
Mean interval loss
Ranking of algorithms
Loss curves
ISSN:0885-6125, 1573-0565
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Algorithm selection methods can be speeded-up substantially by incorporating multi-objective measures that give preference to algorithms that are both promising and fast to evaluate. In this paper, we introduce such a measure, A3R, and incorporate it into two algorithm selection techniques: average ranking and active testing . Average ranking combines algorithm rankings observed on prior datasets to identify the best algorithms for a new dataset. The aim of the second method is to iteratively select algorithms to be tested on the new dataset, learning from each new evaluation to intelligently select the next best candidate. We show how both methods can be upgraded to incorporate a multi-objective measure A3R that combines accuracy and runtime. It is necessary to establish the correct balance between accuracy and runtime, as otherwise time will be wasted by conducting less informative tests. The correct balance can be set by an appropriate parameter setting within function A3R that trades off accuracy and runtime. Our results demonstrate that the upgraded versions of Average Ranking and Active Testing lead to much better mean interval loss values than their accuracy-based counterparts.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:0885-6125
1573-0565
DOI:10.1007/s10994-017-5687-8