Crossover Designs in Software Engineering Experiments: Benefits and Perils

In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticism...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:IEEE transactions on software engineering Jg. 42; H. 2; S. 120 - 135
Hauptverfasser: Vegas, Sira, Apa, Cecilia, Juristo, Natalia
Format: Journal Article
Sprache:Englisch
Veröffentlicht: New York IEEE 01.02.2016
IEEE Computer Society
Schlagworte:
Animals
Atmospheric measurements
Avoidance
carryover
Constraining
crossover designs
Crossovers
Data analysis
Design
Design analysis
Design engineering
Experiments
Information processing
Literature reviews
Particle measurements
Psychology
Risk assessment
Software engineering
Studies
Threat evaluation
US Government agencies
controlled experiment
carryover
data analysis
Experimental software engineering
crossover design
ISSN:0098-5589, 1939-3520
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:In experiments with crossover design subjects apply more than one treatment. Crossover designs are widespread in software engineering experimentation: they require fewer subjects and control the variability among subjects. However, some researchers disapprove of crossover designs. The main criticisms are: the carryover threat and its troublesome analysis. Carryover is the persistence of the effect of one treatment when another treatment is applied later. It may invalidate the results of an experiment. Additionally, crossover designs are often not properly designed and/or analysed, limiting the validity of the results. In this paper, we aim to make SE researchers aware of the perils of crossover experiments and provide risk avoidance good practices. We study how another discipline (medicine) runs crossover experiments. We review the SE literature and discuss which good practices tend not to be adhered to, giving advice on how they should be applied in SE experiments. We illustrate the concepts discussed analysing a crossover experiment that we have run. We conclude that crossover experiments can yield valid results, provided they are properly designed and analysed, and that, if correctly addressed, carryover is no worse than other validity threats.
Bibliographie:SourceType-Scholarly Journals-1
ObjectType-Feature-1
content type line 14
ObjectType-Article-1
ObjectType-Feature-2
content type line 23
ISSN:0098-5589
1939-3520
DOI:10.1109/TSE.2015.2467378