Sample size calculations for the experimental comparison of multiple algorithms on multiple problem instances

This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on the desire...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Journal of heuristics Ročník 26; číslo 6; s. 851 - 883
Hlavní autoři: Campelo, Felipe, Wanner, Elizabeth F.
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.12.2020
Springer Nature B.V
Témata:
Algorithms
Artificial Intelligence
Calculus of Variations and Optimal Control; Optimization
Management Science
Mathematical analysis
Mathematics
Mathematics and Statistics
Operations Research
Operations Research/Decision Theory
Simulated annealing
Iterative sampling
Statistical methods
Multiple hypotheses testing
Sample size estimation
Experimental comparison of algorithms
ISSN:1381-1231, 1572-9397
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:This work presents a statistically principled method for estimating the required number of instances in the experimental comparison of multiple algorithms on a given problem class of interest. This approach generalises earlier results by allowing researchers to design experiments based on the desired best, worst, mean or median-case statistical power to detect differences between algorithms larger than a certain threshold. Holm’s step-down procedure is used to maintain the overall significance level controlled at desired levels, without resulting in overly conservative experiments. This paper also presents an approach for sampling each algorithm on each instance, based on optimal sample size ratios that minimise the total required number of runs subject to a desired accuracy in the estimation of paired differences. A case study investigating the effect of 21 variants of a custom-tailored Simulated Annealing for a class of scheduling problems is used to illustrate the application of the proposed methods for sample size calculations in the experimental comparison of algorithms.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1381-1231
1572-9397
DOI:10.1007/s10732-020-09454-w