Empirical software engineering experts on the use of students and professionals in experiments

[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:Empirical software engineering : an international journal Ročník 23; číslo 1; s. 452 - 489
Hlavní autoři: Falessi, Davide, Juristo, Natalia, Wohlin, Claes, Turhan, Burak, Münch, Jürgen, Jedlitschka, Andreas, Oivo, Markku
Médium: Journal Article
Jazyk:angličtina
Vydáno: New York Springer US 01.02.2018
Springer Nature B.V
Témata:
Compilers
Computer Science
Controlled experiment
Education
Empirical analysis
Empirical Software Engineering
Engineering education
Experimentation
Experiments
External validities
Generalization
Interpreters
Participants in experiments
Programming Languages
Researchers
Software engineering
Software engineering experiments
Software engineering theories
Software Engineering/Programming and Operating Systems
Students
Subjects of experiments
Surveys
Threats to validity
Threats to validity
Participants in experiments
Experimentation
Generalization
Subjects of experiments
ISSN:1382-3256, 1573-7616, 1573-7616
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Popis
Shrnutí:[Context] Controlled experiments are an important empirical method to generate and validate theories. Many software engineering experiments are conducted with students. It is often claimed that the use of students as participants in experiments comes at the cost of low external validity while using professionals does not. [Objective] We believe a deeper understanding is needed on the external validity of software engineering experiments conducted with students or with professionals. We aim to gain insight about the pros and cons of using students and professionals in experiments. [Method] We performed an unconventional, focus group approach and a follow-up survey. First, during a session at ISERN 2014, 65 empirical researchers, including the seven authors, argued and discussed the use of students in experiments with an open mind. Afterwards, we revisited the topic and elicited experts’ opinions to foster discussions. Then we derived 14 statements and asked the ISERN attendees excluding the authors, to provide their level of agreement with the statements. Finally, we analyzed the researchers’ opinions and used the findings to further discuss the statements. [Results] Our survey results showed that, in general, the respondents disagreed with us about the drawbacks of professionals. We, on the contrary, strongly believe that no population (students, professionals, or others) can be deemed better than another in absolute terms. [Conclusion] Using students as participants remains a valid simplification of reality needed in laboratory contexts. It is an effective way to advance software engineering theories and technologies but, like any other aspect of study settings, should be carefully considered during the design, execution, interpretation, and reporting of an experiment. The key is to understand which developer population portion is being represented by the participants in an experiment. Thus, a proposal for describing experimental participants is put forward.
Bibliografie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ISSN:1382-3256
1573-7616
1573-7616
DOI:10.1007/s10664-017-9523-3