A systematic literature review of techniques and metrics to reduce the cost of mutation testing

•Comprehensive classifications for cost reduction goals and cost reduction techniques.•Overview of all selected studies grouped by cost reduction goal.•Characterization of metrics used to measure cost reduction.•Summary, analysis and discussion about cost savings and test effectiveness.•Recommendati...

Full description

Saved in:
Bibliographic Details
Published in:The Journal of systems and software Vol. 157; p. 110388
Main Authors: Pizzoleto, Alessandro Viola, Ferrari, Fabiano Cutigi, Offutt, Jeff, Fernandes, Leo, Ribeiro, Márcio
Format: Journal Article
Language:English
Published: Elsevier Inc 01.11.2019
Subjects:
Cost reduction
Mutation analysis
Mutation testing
Systematic review
Systematic review
Mutation analysis
Mutation testing
Cost reduction
ISSN:0164-1212, 1873-1228
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:•Comprehensive classifications for cost reduction goals and cost reduction techniques.•Overview of all selected studies grouped by cost reduction goal.•Characterization of metrics used to measure cost reduction.•Summary, analysis and discussion about cost savings and test effectiveness.•Recommendations for future research. Historically, researchers have proposed and applied many techniques to reduce the cost of mutation testing. It has become difficult to find all techniques and to understand the cost-benefit tradeoffs among them, which is critical to transitioning this technology to practice. This paper extends a prior workshop paper to summarize and analyze the current knowledge about reducing the cost of mutation testing through a systematic literature review. We selected 175 peer-reviewed studies, from which 153 present either original or updated contributions. Our analysis resulted in six main goals for cost reduction and 21 techniques. In the last decade, a growing number of studies explored techniques such as selective mutation, evolutionary algorithms, control-flow analysis, and higher-order mutation. Furthermore, we characterized 18 metrics, with particular interest in the number of mutants to be executed, test cases required, equivalent mutants generated and detected, and mutant execution speedup. We found that cost reduction for mutation is increasingly becoming interdisciplinary, often combining multiple techniques. Additionally, measurements vary even for studies that use the same techniques. Researchers can use our results to find more detailed information about particular techniques, and to design comparable and reproducible experiments.
ISSN:0164-1212
1873-1228
DOI:10.1016/j.jss.2019.07.100