Analysing differences between algorithm configurations through ablation

Developers of high-performance algorithms for hard computational problems increasingly take advantage of automated parameter tuning and algorithm configuration tools, and consequently often create solvers with many parameters and vast configuration spaces. However, there has been very little work to...

Full description

Saved in:
Bibliographic Details
Published in:Journal of heuristics Vol. 22; no. 4; pp. 431 - 458
Main Authors: Fawcett, Chris, Hoos, Holger H.
Format: Journal Article
Language:English
Published: New York Springer US 01.08.2016
Springer Nature B.V
Subjects:
Ablation
Algorithms
Artificial Intelligence
Automation
Calculus of Variations and Optimal Control; Optimization
Design of experiments
Factorial experiments
Genetic algorithms
Heuristic
Integer programming
Management Science
Mathematical analysis
Mathematics
Mathematics and Statistics
Operations Research
Operations Research/Decision Theory
Optimization algorithms
Production planning
Studies
Variance analysis
Parameter importance
Empirical analysis
Ablation analysis
Automated algorithm configuration
ISSN:1381-1231, 1572-9397
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:Developers of high-performance algorithms for hard computational problems increasingly take advantage of automated parameter tuning and algorithm configuration tools, and consequently often create solvers with many parameters and vast configuration spaces. However, there has been very little work to help these algorithm developers answer questions about the high-quality configurations produced by these tools, specifically about which parameter changes contribute most to improved performance. In this work, we present an automated technique for answering such questions by performing ablation analysis between two algorithm configurations. We perform an extensive empirical analysis of our technique on five scenarios from propositional satisfiability, mixed-integer programming and AI planning, and show that in all of these scenarios more than 95 % of the performance gains between default configurations and optimised configurations obtained from automated configuration tools can be explained by modifying the values of a small number of parameters (1–4 in the scenarios we studied). We also investigate the use of our ablation analysis procedure for producing configurations that generalise well to previously-unseen problem domains, as well as for analysing the structure of the algorithm parameter response surface near and between high-performance configurations.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1381-1231
1572-9397
DOI:10.1007/s10732-014-9275-9