Sampling in software engineering research: a critical review and guidelines

Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality...

Celý popis

Uloženo v:

Podrobná bibliografie
Vydáno v:	Empirical software engineering : an international journal Ročník 27; číslo 4
Hlavní autoři:	Baltes, Sebastian, Ralph, Paul
Médium:	Journal Article
Jazyk:	angličtina
Vydáno:	New York Springer US 01.07.2022 Springer Nature B.V
Témata:	Compilers Computer Science Engineering research Guidelines Interpreters Programming Languages Qualitative research Random sampling Software engineering Software Engineering/Programming and Operating Systems Representative sampling Random sampling Purposive sampling Sampling Probability sampling Research methodology
ISSN:	1382-3256, 1573-7616
On-line přístup:	Získat plný text
Tagy:	Přidat tag Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!

Abstract	Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis . To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.
AbstractList	Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis . To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research. Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of representative sampling undermines a scientific field. This article therefore reports a critical review of the state of sampling in recent, high-quality software engineering research. The key findings are: (1) random sampling is rare; (2) sophisticated sampling strategies are very rare; (3) sampling, representativeness and randomness often appear misunderstood. These findings suggest that software engineering research has a generalizability crisis. To address these problems, this paper synthesizes existing knowledge of sampling into a succinct primer and proposes extensive guidelines for improving the conduct, presentation and evaluation of sampling in software engineering research. It is further recommended that while researchers should strive for more representative samples, disparaging non-probability sampling is generally capricious and particularly misguided for predominately qualitative research.
ArticleNumber	94
Author	Ralph, Paul Baltes, Sebastian
Author_xml	– sequence: 1 givenname: Sebastian orcidid: 0000-0002-2442-7522 surname: Baltes fullname: Baltes, Sebastian email: sebastian.baltes@adelaide.edu.au organization: University of Adelaide – sequence: 2 givenname: Paul surname: Ralph fullname: Ralph, Paul organization: Dalhousie University
BookMark	eNp9kE1Lw0AQhhepYFv9A54Cnlf3I9lsvEnxCwse1POymUzqlnRTd1OL_97ECIKHnmbeYZ53hndGJr71SMg5Z5ecsfwqcqZUSpngdNCC6iMy5Vkuaa64mvS91IJKkakTMotxzRgr8jSbkqcXu9k2zq8S55PY1t3eBkzQr5xHDMM8YEQb4P06sQkE1zmwTT_8dLhPrK-S1c5V2DtgPCXHtW0inv3WOXm7u31dPNDl8_3j4mZJQfKioxUAlDyrRK5smdYAtmSyYpnWgHWeo5WiKASksuSFErzMUCFAUWpWADJdyzm5GH23of3YYezMut0F3580QmUqTVOlsn5LjFsQ2hgD1mYb3MaGL8OZGUIyY2imD-1HC6N7SP-DwHW2c63vgnXNYVSOaNwOuWH4--oA9Q3EJYPv
CitedBy_id	crossref_primary_10_1016_j_apmrv_2024_09_004 crossref_primary_10_1016_j_infsof_2024_107549 crossref_primary_10_1016_j_infsof_2024_107425 crossref_primary_10_1007_s41060_024_00603_z crossref_primary_10_1109_TSE_2025_3586700 crossref_primary_10_1016_j_infsof_2025_107710 crossref_primary_10_1109_MS_2023_3267296 crossref_primary_10_1109_TSE_2021_3052177 crossref_primary_10_1016_j_infsof_2025_107672 crossref_primary_10_3390_app14114780 crossref_primary_10_1007_s10664_024_10582_1 crossref_primary_10_1007_s10664_024_10604_y crossref_primary_10_1007_s10664_022_10225_3 crossref_primary_10_1145_3747179 crossref_primary_10_1016_j_jss_2025_112579 crossref_primary_10_3390_app15020518 crossref_primary_10_3390_info16090774 crossref_primary_10_1016_j_infsof_2023_107201 crossref_primary_10_4102_curationis_v45i1_2372 crossref_primary_10_1016_j_jss_2025_112616 crossref_primary_10_1007_s11219_024_09698_w crossref_primary_10_1109_TSE_2025_3551664 crossref_primary_10_1109_TSE_2022_3154769 crossref_primary_10_20525_ijrbs_v14i5_4160 crossref_primary_10_37251_ijome_v1i2_1341 crossref_primary_10_37251_ijome_v1i2_1346 crossref_primary_10_1007_s10664_024_10442_y crossref_primary_10_1145_3729359 crossref_primary_10_1007_s00766_024_00428_z crossref_primary_10_1145_3649598 crossref_primary_10_1002_smr_2531 crossref_primary_10_1016_j_heliyon_2023_e17408 crossref_primary_10_1145_3712198 crossref_primary_10_1007_s10664_025_10659_5 crossref_primary_10_1007_s10664_025_10715_0 crossref_primary_10_1109_ACCESS_2023_3322934 crossref_primary_10_1007_s10664_025_10644_y crossref_primary_10_3390_risks13060105 crossref_primary_10_3390_su17052162 crossref_primary_10_1080_10447318_2022_2163566 crossref_primary_10_1016_j_jss_2023_111700 crossref_primary_10_1145_3571854 crossref_primary_10_1186_s12912_024_01922_z crossref_primary_10_1007_s10270_025_01273_6 crossref_primary_10_1007_s00500_023_09248_9 crossref_primary_10_1080_12460125_2025_2458874 crossref_primary_10_1109_TSE_2025_3552614 crossref_primary_10_1145_3765754 crossref_primary_10_1145_3698829 crossref_primary_10_1371_journal_pone_0308952 crossref_primary_10_1016_j_jss_2024_112120 crossref_primary_10_1145_3546943 crossref_primary_10_1080_13527266_2025_2509778 crossref_primary_10_1016_j_jss_2022_111397 crossref_primary_10_3390_vehicles5030057 crossref_primary_10_1007_s00521_024_10452_3 crossref_primary_10_1186_s40537_023_00733_4 crossref_primary_10_1016_j_datak_2024_102299 crossref_primary_10_1007_s10664_024_10546_5 crossref_primary_10_1109_TSE_2024_3474173 crossref_primary_10_1007_s10664_024_10565_2 crossref_primary_10_1177_87569728251374906 crossref_primary_10_1145_3757060 crossref_primary_10_1145_3680472 crossref_primary_10_3390_en16010112 crossref_primary_10_3390_foods12224089 crossref_primary_10_1016_j_jss_2024_112110 crossref_primary_10_1109_MS_2023_3257743 crossref_primary_10_1016_j_jss_2024_112196 crossref_primary_10_1007_s13369_023_08596_w crossref_primary_10_1109_TSE_2022_3225197 crossref_primary_10_1108_DPRG_06_2024_0131 crossref_primary_10_1007_s10664_023_10367_y crossref_primary_10_1109_TSE_2020_3019892 crossref_primary_10_1145_3640334 crossref_primary_10_1007_s10664_024_10550_9 crossref_primary_10_1145_3579643 crossref_primary_10_1109_ACCESS_2025_3584156 crossref_primary_10_3390_app13063437 crossref_primary_10_1109_TSE_2023_3339881 crossref_primary_10_1016_j_infsof_2024_107404 crossref_primary_10_1145_3635715 crossref_primary_10_1080_10447318_2023_2212219 crossref_primary_10_1108_CEMJ_02_2024_0057 crossref_primary_10_1007_s10664_024_10508_x crossref_primary_10_1016_j_jss_2025_112586 crossref_primary_10_1145_3626519 crossref_primary_10_1145_3603110 crossref_primary_10_1177_21582440231224780 crossref_primary_10_3390_software4030018 crossref_primary_10_1007_s00766_024_00425_2 crossref_primary_10_1016_j_csi_2024_103878 crossref_primary_10_1145_3643731 crossref_primary_10_61735_4j8w1r85 crossref_primary_10_1016_j_infsof_2024_107516 crossref_primary_10_1007_s10270_025_01294_1 crossref_primary_10_1016_j_infsof_2024_107512 crossref_primary_10_1016_j_infsof_2023_107337 crossref_primary_10_1016_j_jss_2025_112471 crossref_primary_10_1108_EMJB_11_2024_0290 crossref_primary_10_1002_sys_21663 crossref_primary_10_4236_ojbm_2025_132058 crossref_primary_10_1016_j_jss_2022_111562 crossref_primary_10_1108_IMDS_10_2022_0602 crossref_primary_10_1007_s10664_025_10661_x crossref_primary_10_1016_j_jss_2025_112596 crossref_primary_10_1145_3714468 crossref_primary_10_1109_TSE_2024_3369766 crossref_primary_10_31893_multirev_2025230 crossref_primary_10_1007_s10664_024_10557_2
Cites_doi	10.1145/3196398.3196430 10.1109/MSR.2013.6624034 10.1109/APSEC.2010.46 10.1007/s10651-005-6818-0 10.1145/2858036.2858498 10.1017/XPS.2015.19 10.1201/9780429029608 10.1109/TSE.2020.3003413 10.1145/3377811.3380333 10.4135/9781412985451 10.1007/BF00988249 10.4256/mio.2010.0017 10.1007/s10461-008-9421-1 10.4324/9781315422657 10.1145/3377811.3380395 10.1145/2901739.2901776 10.1109/CESI.2017.5 10.1145/3183440.3195001 10.1109/TSE.2013.12 10.1145/3338906.3338976 10.1037/0003-066X.63.7.602 10.2307/1403202 10.1109/ICSE.2019.00036 10.2307/3150766 10.1007/s10664-018-9655-0 10.1186/s12874-015-0089-8 10.1007/978-1-4614-3876-2_13 10.1145/2702123.2702549 10.1145/1134285.1134333 10.1046/j.1365-2648.1997.00371.x 10.1017/S0140525X0999152X 10.1145/2491411.2491415 10.1186/s40411-015-0023-0 10.1109/ISESE.2002.1166921 10.1007/978-1-84800-044-5_11 10.1177/1049732307308450 10.1007/978-3-319-93632-1_9 10.1007/s10664-020-09875-y 10.2307/3096941 10.4135/9781473920156 10.1007/978-1-84800-044-5_3 10.5281/zenodo.3666824 10.1016/j.infsof.2017.12.001 10.1145/3149485.3149496 10.4324/9780203793206 10.1073/pnas.1000261107 10.1109/TSE.2018.2796554 10.4135/9781452272047 10.1002/9781394260645 10.1080/01621459.1990.10474975 10.1287/isre.14.3.221.16560 10.1145/3241743 10.2307/1402647 10.1145/2961111.2962632 10.1007/s10664-017-9523-3 10.1023/A:1022908820784 10.1109/TSE.2004.1274044 10.1371/journal.pmed.1000097 10.1007/BF02765185 10.1016/j.infsof.2007.09.004 10.1037/h0055392 10.1145/3180155.3180235 10.1023/A:1024232915870 10.1145/2884781.2884833 10.1145/571681.571686 10.1145/2961111.2962628 10.1007/978-3-642-29044-2
ContentType	Journal Article
Copyright	The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022 The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.
Copyright_xml	– notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022 – notice: The Author(s), under exclusive licence to Springer Science+Business Media, LLC, part of Springer Nature 2022.
DBID	AAYXX CITATION 7SC 8FD 8FE 8FG ABJCF AFKRA ARAPS BENPR BGLVJ CCPQU DWQXO HCIFZ JQ2 L6V L7M L~C L~D M7S P5Z P62 PHGZM PHGZT PKEHL PQEST PQGLB PQQKQ PQUKI PRINS PTHSS S0W
DOI	10.1007/s10664-021-10072-8
DatabaseName	CrossRef Computer and Information Systems Abstracts Technology Research Database ProQuest SciTech Collection ProQuest Technology Collection Materials Science & Engineering Collection ProQuest Central UK/Ireland Advanced Technologies & Computer Science Collection ProQuest Central Technology Collection ProQuest One ProQuest Central SciTech Premium Collection ProQuest Computer Science Collection ProQuest Engineering Collection Advanced Technologies Database with Aerospace Computer and Information Systems Abstracts Academic Computer and Information Systems Abstracts Professional Engineering Database Advanced Technologies & Aerospace Database (ProQuest) ProQuest Advanced Technologies & Aerospace Collection ProQuest Central Premium ProQuest One Academic ProQuest One Academic Middle East (New) ProQuest One Academic Eastern Edition (DO NOT USE) One Applied & Life Sciences ProQuest One Academic (retired) ProQuest One Academic UKI Edition ProQuest Central China Engineering Collection DELNET Engineering & Technology Collection
DatabaseTitle	CrossRef Technology Collection Technology Research Database Computer and Information Systems Abstracts – Academic ProQuest One Academic Middle East (New) ProQuest Advanced Technologies & Aerospace Collection ProQuest Computer Science Collection Computer and Information Systems Abstracts SciTech Premium Collection ProQuest One Community College ProQuest Central China ProQuest Central ProQuest One Applied & Life Sciences ProQuest Engineering Collection ProQuest Central Korea ProQuest Central (New) Advanced Technologies Database with Aerospace Engineering Collection Advanced Technologies & Aerospace Collection Engineering Database ProQuest One Academic Eastern Edition ProQuest Technology Collection ProQuest SciTech Collection Computer and Information Systems Abstracts Professional Advanced Technologies & Aerospace Database ProQuest One Academic UKI Edition ProQuest DELNET Engineering and Technology Collection Materials Science & Engineering Collection ProQuest One Academic ProQuest One Academic (New)
DatabaseTitleList	Technology Collection
Database_xml	– sequence: 1 dbid: BENPR name: ProQuest Central url: https://www.proquest.com/central sourceTypes: Aggregation Database
DeliveryMethod	fulltext_linktorsrc
Discipline	Computer Science
EISSN	1573-7616
ExternalDocumentID	10_1007_s10664_021_10072_8
GroupedDBID	-4Z -59 -5G -BR -EM -Y2 -~C .86 .DC .VR 06D 0R~ 0VY 199 1N0 1SB 2.D 203 28- 29G 2J2 2JN 2JY 2KG 2LR 2P1 2VQ 2~H 30V 4.4 406 408 409 40D 40E 5GY 5QI 5VS 67Z 6NX 78A 8FE 8FG 8TC 8UJ 95- 95. 95~ 96X AABHQ AACDK AAHNG AAIAL AAJBT AAJKR AANZL AAOBN AARHV AARTL AASML AATNV AATVU AAUYE AAWCG AAYIU AAYOK AAYQN AAYTO AAYZH ABAKF ABBBX ABBXA ABDZT ABECU ABFTD ABFTV ABHLI ABHQN ABJCF ABJNI ABJOX ABKCH ABKTR ABMNI ABMQK ABNWP ABQBU ABQSL ABSXP ABTEG ABTHY ABTKH ABTMW ABULA ABWNU ABXPI ACAOD ACBXY ACDTI ACGFS ACHSB ACHXU ACIWK ACKNC ACMDZ ACMLO ACOKC ACOMO ACPIV ACSNA ACZOJ ADHHG ADHIR ADIMF ADINQ ADKNI ADKPE ADRFC ADTPH ADURQ ADYFF ADZKW AEBTG AEFIE AEFQL AEGAL AEGNC AEJHL AEJRE AEKMD AEMSY AENEX AEOHA AEPYU AESKC AETLH AEVLU AEXYK AFBBN AFEXP AFGCZ AFKRA AFLOW AFQWF AFWTZ AFZKB AGAYW AGDGC AGGDS AGJBK AGMZJ AGQEE AGQMX AGRTI AGWIL AGWZB AGYKE AHAVH AHBYD AHKAY AHSBF AHYZX AIAKS AIGIU AIIXL AILAN AITGF AJBLW AJRNO AJZVZ ALMA_UNASSIGNED_HOLDINGS ALWAN AMKLP AMXSW AMYLF AMYQR AOCGG ARAPS ARMRJ ASPBG AVWKF AXYYD AYJHY AZFZN B-. BA0 BBWZM BDATZ BENPR BGLVJ BGNMA BSONS CAG CCPQU COF CS3 CSCUP DDRTE DL5 DNIVK DPUIP DU5 EBLON EBS EIOEI EJD ESBYG FEDTE FERAY FFXSO FIGPU FINBP FNLPD FRRFC FSGXE FWDCC GGCAI GGRSB GJIRD GNWQR GQ6 GQ7 GQ8 GXS H13 HCIFZ HF~ HG5 HG6 HMJXF HQYDN HRMNR HVGLF HZ~ I09 IHE IJ- IKXTQ ITM IWAJR IXC IZIGR IZQ I~X I~Z J-C J0Z JBSCW JCJTX JZLTJ KDC KOV KOW L6V LAK LLZTM M4Y M7S MA- N2Q NB0 NDZJH NPVJJ NQJWS NU0 O9- O93 O9G O9I O9J OAM P19 P62 P9O PF0 PT4 PT5 PTHSS Q2X QOK QOS R4E R89 R9I RHV RNI RNS ROL RPX RSV RZC RZE RZK S0W S16 S1Z S26 S27 S28 S3B SAP SCJ SCLPG SCO SDH SDM SHX SISQX SJYHP SNE SNPRN SNX SOHCF SOJ SPISZ SRMVM SSLCW STPWE SZN T13 T16 TSG TSK TSV TUC U2A UG4 UOJIU UTJUX UZXMN VC2 VFIZW W23 W48 WK8 YLTOR Z45 Z7R Z7S Z7V Z7X Z7Z Z81 Z83 Z86 Z88 Z8M Z8N Z8P Z8R Z8T Z8U Z8W Z92 ZMTXR ~EX AAPKM AAYXX ABBRH ABDBE ABFSG ABRTQ ACSTC ADHKG AEZWR AFDZB AFFHD AFHIU AFOHR AGQPQ AHPBZ AHWEU AIXLP ATHPR AYFIA CITATION PHGZM PHGZT PQGLB 7SC 8FD DWQXO JQ2 L7M L~C L~D PKEHL PQEST PQQKQ PQUKI PRINS
ID	FETCH-LOGICAL-c319t-dcccb15d276ab4fccab03d0588cef77ea32992c43b19621b5e6ecc9b809ce08f3
IEDL.DBID	M7S
ISICitedReferencesCount	187
ISICitedReferencesURI	http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000788586200002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN	1382-3256
IngestDate	Tue Dec 02 15:57:16 EST 2025 Sat Nov 29 05:37:46 EST 2025 Tue Nov 18 22:01:37 EST 2025 Fri Feb 21 02:49:10 EST 2025
IsPeerReviewed	true
IsScholarly	true
Issue	4
Keywords	Representative sampling Random sampling Purposive sampling Sampling Probability sampling Research methodology
Language	English
LinkModel	DirectLink
MergedId	FETCHMERGED-LOGICAL-c319t-dcccb15d276ab4fccab03d0588cef77ea32992c43b19621b5e6ecc9b809ce08f3
Notes	ObjectType-Article-1 SourceType-Scholarly Journals-1 ObjectType-Feature-2 content type line 14
ORCID	0000-0002-2442-7522
PQID	2656444665
PQPubID	326341
ParticipantIDs	proquest_journals_2656444665 crossref_primary_10_1007_s10664_021_10072_8 crossref_citationtrail_10_1007_s10664_021_10072_8 springer_journals_10_1007_s10664_021_10072_8
PublicationCentury	2000
PublicationDate	2022-07-01
PublicationDateYYYYMMDD	2022-07-01
PublicationDate_xml	– month: 07 year: 2022 text: 2022-07-01 day: 01
PublicationDecade	2020
PublicationPlace	New York
PublicationPlace_xml	– name: New York – name: Dordrecht
PublicationSubtitle	An International Journal
PublicationTitle	Empirical software engineering : an international journal
PublicationTitleAbbrev	Empir Software Eng
PublicationYear	2022
Publisher	Springer US Springer Nature B.V
Publisher_xml	– name: Springer US – name: Springer Nature B.V
References	HenrichJHeineSJNorenzayanAThe weirdest people in the world?Behav Brain Sci2010332-3618310.1017/S0140525X0999152X GoelSSalganikMJAssessing respondent-driven samplingProceedings of the National Academy of Sciences2010107156743674710.1073/pnas.1000261107 Amir B, Ralph P (2018) There is no random sampling in software engineering research. In: Proceedings of the 40th international conference on software engineering: companion proceeedings, pp 344–345 TheisenCDunaiskiMWilliamsLVisserWSoftware engineering research at the international conference on software engineering in 2016ACM SIGSOFT Software Engineering Notes20184241710.1145/3149485.3149496 FeldtRZimmermannTBergersenGRFalessiDJedlitschkaAJuristoNMünchJOivoMRunesonPShepperdMSjøbergDIKTurhanBFour commentaries on the use of students and professionals in empirical software engineering experimentsEmpir Softw Eng20182363801382010.1007/s10664-018-9655-0 MoherDLiberatiATetzlaffJAltmanDGGroupPPreferred reporting items for systematic reviews and meta-analyses: the prisma statementPLoS Med200967e100009710.1371/journal.pmed.1000097 Sedano T, Ralph P, Péraire C (2019) The product backlog. In: 2019 IEEE/ACM 41St international conference on software engineering (ICSE), IEEE, pp 200–211 Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th working conference on Mining Software Repositories (MSR), IEEE, pp 137–141 DillmanDASmythJDChristianLMInternet, phone, mail and mixed-mode surveys: the tailored design method20144th edn.HobokenJohn Wiley & Sons GubaEGLincolnYSEpistemological and methodological bases of naturalistic inquiryEduc Commun Technol J198230423325210.1007/BF02765185 HeckathornDDRespondent-driven sampling: a new approach to the study of hidden populationsSoc Probl199744217419910.2307/3096941 ChecklandPHolwellSAction research: its nature and validitySyst Pract Action Res199811192110.1023/A:1022908820784 Caine K (2016) Local standards for sample size at chi. In: Proceedings of the 2016 CHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’16, pp 981–992 Ingram C, Drachen A (2020) How software practitioners use informal local meetups to share software engineering knowledge. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE FalessiDJuristoNWohlinCTurhanBMünchJJedlitschkaAOivoMEmpirical software engineering experts on the use of students and professionals in experimentsEmpir Softw Eng201823145248910.1007/s10664-017-9523-3 CohenJStatistical power analysis for the behavioral sciences1988HillsdaleLawrence Erlbaum Associates0747.62110 Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering, Springer, pp 63–92 LandonELJrBanksSKRelative efficiency and bias of Plus-One telephone samplingJ Mark Res197714329410.2307/3150766 FaugierJSargeantMSampling hard to reach populationsJ Adv Nurs199726479079710.1046/j.1365-2648.1997.00371.x YinRKCase study research: Design and methods20186th edn.CaliforniaSage, Thousand Oaks Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Genero M, Jedlitschka A, Jorgensen M (eds) 10th international symposium on Empirical Software Engineering and Measurement (ESEM 2016), ACM, Ciudad Real, Spain, pp 52:1–52:6, DOI https://doi.org/10.1145/2961111.2962628, (to appear in print) TrostJEStatistically nonrepresentative stratified sampling: a sampling technique for qualitative studiesQual Sociol198691545710.1007/BF00988249 KitchenhamBPfleegerSLPrinciples of survey research: Part 5: populations and samplesACM SIGSOFT Softw Eng Notes2002275172010.1145/571681.571686 PaulsonJWSucciGEberleinAAn empirical study of open-source and closed-source software productsIEEE Trans Softw Eng200430424625610.1109/TSE.2004.1274044 Huang X, Zhang H, Zhou X, Babar MA, Yang S (2018) Synthesizing qualitative research in software engineering: a critical review. In: Proceedings of the 40th international conference on software engineering, pp 1207–1218 de MelloRMDa SilvaPCTravassosGHInvestigating probabilistic sampling approaches for large-scale surveys in software engineeringJ Softw Eng Res Dev20153112610.1186/s40411-015-0023-0 TrochimWMDonnellyJPResearch methods knowledge base, vol 22001Cincinnati, OH, USAAtomic Dog Publishing Patton MQ (2014) Qualitative research & evaluation methods: integrating theory and practice. Sage Publications Ralph P, Baltes S, Adisaputri G, Torkar R, Kovalenko V, Kalinowski M, Novielli N, Yoo S, Devroey X, Tan X et al (2020b) Pandemic programming: how covid-19 affects software developers and how their organizations can help. Empir Softw Eng https://doi.org/10.1007/s10664-020-09875-y Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: 2017 IEEE/ACM 5th international workshop on Conducting Empirical Studies in Industry (CESI), IEEE, pp 33–39 CharmazKConstructing grounded theory2014LondonSage Thomas G, Myers K (2015) The anatomy of the case study. Sage De Mello RM, Travassos GH (2016) Surveys in software engineering: Identifying representative samples. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, pp 1–6 Daniel J (2011) Sampling essentials: practical guidelines for making sampling choices. Sage Publications Foster E (2014) Software engineering: a methodical approach apress. New York, USA van Manen M (2016) Phenomenology of practice: meaning-giving methods in phenomenological research and writing. Routledge Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Keele University and University of Durham BreckenridgeJJonesDDemystifying theoretical sampling in grounded theory researchGrounded Theory Rev200982112126 Zannier C, Melnik G, Maurer F (2006) On the success of empirical studies in the international conference on software engineering. In: Proceedings of the 28th international conference on software engineering, pp 341–350 StolKJFitzgeraldBThe abc of software engineering researchACM Transactions on Software Engineering and Methodology (TOSEM)20182731110.1145/3241743 Zhang H, Huang X, Zhou X, Huang H, Babar MA (2019) Ethnographic research in software engineering: a critical review and checklist. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 659–670 KruskalWMostellerFRepresentative sampling, i: mon-scientific literatureInt Stat Rev1979471132410.2307/1403202 Sjøberg D, Anda B, Arisholm E, Dyba T, Jørgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. In: 2002 international symposium on empirical software engineering. IEEE, Nara, Japan, pp 17–26, DOI https://doi.org/10.1109/ISESE.2002.1166921, (to appear in print) Glaser BG, Strauss AL (2017) Discovery of grounded theory: strategies for qualitative research. Routledge Duignan B (2019) Postmodernism. In: Encyclopedia Britannica, Encyclopedia Britannica, Inc.. https://www.britannica.com/topic/postmodernism-philosophy Easton G (2010) One case study is enough. Lancaster University technical report https://eprints.lancs.ac.uk/id/eprint/49016 MaalejWRobillardMPPatterns of knowledge in api reference documentationIEEE Trans Softw Eng20133991264128210.1109/TSE.2013.12 Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 9th joint meeting on foundations of software engineering, ACM, pp 466–476 DrauckerCBMartsolfDSRossRRuskTBTheoretical sampling and category development in grounded theoryQual Health Res20071781137114810.1177/1049732307308450 Gousios G (2013) The GHTorrent dataset and tool suite. In: Zimmermann T, Di Penta M, Kim S (eds) 10Th international working conference on Mining Software Repositories (MSR, vol 2013. IEEE, San Francisco, CA, USA, pp 233–236 Tempero E, Anslow C, Dietrich J, Han T, Li J, Lumpe M, Melton H, Noble J (2010) The qualitas corpus: a curated collection of java code for empirical studies. In: Proceedings of the 17th Asia Pacific software engineering conference. IEEE, Sydney, Australia, pp 336–345, DOI https://doi.org/10.1109/APSEC.2010.46, (to appear in print) FittsPMThe information capacity of the human motor system in controlling the amplitude of movementJ Exp Psychol195447638139110.1037/h0055392 MullinixKJLeeperTJDruckmanJNFreeseJThe generalizability of survey experimentsJ Exp Polit Sci20152210913810.1017/XPS.2015.19 Baltes S, Dumani L, Treude C, Diehl S (2018) SOTOrrent: reconstructing and analyzing the evolution stack overflow posts. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on Mining Software Repositories (MSR 2018), ACM, Gothenburg, Sweden, pp 319–330 Cochran WG (2007) Sampling techniques. John Wiley & Sons Russo D, Stol K (in press) Gender differences in personality traits of software engineers. IEEE Trans Softw Eng https://doi.org/10.1109/TSE.2020.3003413 KruskalWMostellerFRepresentative sampling, iii: the current statistical literatureInt Stat Rev197947324526510.2307/14026470471.62010 Baltes S, Ralph P (2020) Sampling in software engineering research supplementary material [data set]. https://doi.org/10.5281/zenodo.3666824 BeechamSBaddooNHallTRobinsonHSharpHMotivation in software engineering: a systematic literature reviewInf Softw Technol200850986087810.1016/j.infsof.2007.09.004 Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE MalekinejadMJohnstonLGKendallCKerrLRFSRifkinMRRutherfordGWUsing respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic reviewAIDS and Behavior200812110513010.1007/s10461-008-9421-1 Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2 10072_CR62 10072_CR61 10072_CR60 N Salleh (10072_CR63) 2018; 98 JW Paulson (10072_CR58) 2004; 30 R Feldt (10072_CR23) 2018; 23 P Turk (10072_CR77) 2005; 12 10072_CR19 10072_CR18 J Henrich (10072_CR32) 2010; 33 D Falessi (10072_CR21) 2018; 23 W Maalej (10072_CR46) 2013; 39 JE Trost (10072_CR76) 1986; 9 10072_CR15 DA Dillman (10072_CR16) 2014 KJ Mullinix (10072_CR55) 2015; 2 10072_CR14 S Beecham (10072_CR6) 2008; 50 10072_CR13 10072_CR57 10072_CR56 10072_CR11 EG Guba (10072_CR30) 1982; 30 10072_CR53 10072_CR50 P Ralph (10072_CR59) 2019; 45 C Theisen (10072_CR70) 2018; 42 10072_CR1 DD Heckathorn (10072_CR31) 1997; 44 S Goel (10072_CR28) 2010; 107 P Checkland (10072_CR10) 1998; 11 EL Landon Jr (10072_CR44) 1977; 14 10072_CR8 D Moher (10072_CR54) 2009; 6 10072_CR3 10072_CR4 W Kruskal (10072_CR43) 1979; 47 10072_CR5 10072_CR49 RK Yin (10072_CR81) 2018 10072_CR48 LR van Hoeven (10072_CR34) 2015; 15 10072_CR41 10072_CR83 SJ Gentles (10072_CR26) 2015; 20 WM Trochim (10072_CR75) 2001 10072_CR82 10072_CR80 K Charmaz (10072_CR9) 2014 KJ Stol (10072_CR67) 2018; 27 JJ Arnett (10072_CR2) 2008; 63 J Faugier (10072_CR22) 1997; 26 MB Miles (10072_CR52) 2014 SK Thompson (10072_CR72) 1990; 85 10072_CR39 CB Draucker (10072_CR17) 2007; 17 10072_CR37 10072_CR36 10072_CR35 10072_CR79 10072_CR78 10072_CR33 B Kitchenham (10072_CR40) 2002; 27 AS Lee (10072_CR45) 2003; 14 W Kruskal (10072_CR42) 1979; 47 J Breckenridge (10072_CR7) 2009; 8 10072_CR74 RM de Mello (10072_CR51) 2015; 3 10072_CR73 J Cohen (10072_CR12) 1988 LJ Sax (10072_CR64) 2003; 44 10072_CR71 M Malekinejad (10072_CR47) 2008; 12 PM Fitts (10072_CR24) 1954; 47 LG Johnston (10072_CR38) 2010; 5 10072_CR29 10072_CR27 10072_CR25 10072_CR69 10072_CR68 10072_CR66 10072_CR65 10072_CR20
References_xml	– reference: Gousios G (2013) The GHTorrent dataset and tool suite. In: Zimmermann T, Di Penta M, Kim S (eds) 10Th international working conference on Mining Software Repositories (MSR, vol 2013. IEEE, San Francisco, CA, USA, pp 233–236 – reference: Toepoel V (2012) Effects of incentives in surveys. In: Gideon L (ed) Handbook of survey methodology for the social sciences, springer, pp 209–223 – reference: KruskalWMostellerFRepresentative sampling, iii: the current statistical literatureInt Stat Rev197947324526510.2307/14026470471.62010 – reference: FalessiDJuristoNWohlinCTurhanBMünchJJedlitschkaAOivoMEmpirical software engineering experts on the use of students and professionals in experimentsEmpir Softw Eng201823145248910.1007/s10664-017-9523-3 – reference: TurkPBorkowskiJJA review of adaptive cluster sampling: 1990–2003Environ Ecol Stat20051215594214096810.1007/s10651-005-6818-0 – reference: van Manen M (2016) Phenomenology of practice: meaning-giving methods in phenomenological research and writing. Routledge – reference: Stol KJ, Ralph P, Fitzgerald B (2016) Grounded theory in software engineering research: a critical review and guidelines. In: Proceedings of the international conference on software engineering, IEEE, Austin, TX, USA, pp 120–131 – reference: ArnettJJThe neglected 95%: why American psychology needs to become less AmericanAm Psychol200863760210.1037/0003-066X.63.7.602 – reference: Henry GT (1990) Practical sampling. Sage – reference: SallehNHodaRSuMTKanijTGrundyJRecruitment, engagement and feedback in empirical software engineering studies in industrial contextsInform Software Technol20189816117210.1016/j.infsof.2017.12.001 – reference: Baltes S, Diehl S (2016) Worse than spam: issues in sampling software developers. In: Genero M, Jedlitschka A, Jorgensen M (eds) 10th international symposium on Empirical Software Engineering and Measurement (ESEM 2016), ACM, Ciudad Real, Spain, pp 52:1–52:6, DOI https://doi.org/10.1145/2961111.2962628, (to appear in print) – reference: FittsPMThe information capacity of the human motor system in controlling the amplitude of movementJ Exp Psychol195447638139110.1037/h0055392 – reference: HeckathornDDRespondent-driven sampling: a new approach to the study of hidden populationsSoc Probl199744217419910.2307/3096941 – reference: SaxLJGilmartinSKBryantANAssessing response rates and nonresponse bias in web and paper surveysRes High Educ200344440943210.1023/A:1024232915870 – reference: MalekinejadMJohnstonLGKendallCKerrLRFSRifkinMRRutherfordGWUsing respondent-driven sampling methodology for HIV biological and behavioral surveillance in international settings: a systematic reviewAIDS and Behavior200812110513010.1007/s10461-008-9421-1 – reference: StolKJFitzgeraldBThe abc of software engineering researchACM Transactions on Software Engineering and Methodology (TOSEM)20182731110.1145/3241743 – reference: RalphPToward methodological guidelines for process theories and taxonomies in software engineeringIEEE Trans Softw Eng201945771273510.1109/TSE.2018.2796554 – reference: FaugierJSargeantMSampling hard to reach populationsJ Adv Nurs199726479079710.1046/j.1365-2648.1997.00371.x – reference: BeechamSBaddooNHallTRobinsonHSharpHMotivation in software engineering: a systematic literature reviewInf Softw Technol200850986087810.1016/j.infsof.2007.09.004 – reference: MullinixKJLeeperTJDruckmanJNFreeseJThe generalizability of survey experimentsJ Exp Polit Sci20152210913810.1017/XPS.2015.19 – reference: YinRKCase study research: Design and methods20186th edn.CaliforniaSage, Thousand Oaks – reference: Foster E (2014) Software engineering: a methodical approach apress. New York, USA – reference: De Mello RM, Travassos GH (2016) Surveys in software engineering: Identifying representative samples. In: Proceedings of the 10th ACM/IEEE international symposium on empirical software engineering and measurement, pp 1–6 – reference: Vasilescu B, Posnett D, Ray B, van den Brand MG, Serebrenik A, Devanbu P, Filkov V (2015) Gender and Tenure Diversity in GitHub Teams. In: Proceedings of the 33rd annual ACM conference on human factors in computing systems - CHI ’15. ACM Press, Seoul, Republic of Korea, pp 3789–3798, DOI https://doi.org/10.1145/2702123.2702549, (to appear in print) – reference: Daniel J (2011) Sampling essentials: practical guidelines for making sampling choices. Sage Publications – reference: FeldtRZimmermannTBergersenGRFalessiDJedlitschkaAJuristoNMünchJOivoMRunesonPShepperdMSjøbergDIKTurhanBFour commentaries on the use of students and professionals in empirical software engineering experimentsEmpir Softw Eng20182363801382010.1007/s10664-018-9655-0 – reference: GoelSSalganikMJAssessing respondent-driven samplingProceedings of the National Academy of Sciences2010107156743674710.1073/pnas.1000261107 – reference: Duignan B (2019) Postmodernism. In: Encyclopedia Britannica, Encyclopedia Britannica, Inc.. https://www.britannica.com/topic/postmodernism-philosophy – reference: Huang X, Zhang H, Zhou X, Babar MA, Yang S (2018) Synthesizing qualitative research in software engineering: a critical review. In: Proceedings of the 40th international conference on software engineering, pp 1207–1218 – reference: Amir B, Ralph P (2018) There is no random sampling in software engineering research. In: Proceedings of the 40th international conference on software engineering: companion proceeedings, pp 344–345 – reference: Cochran WG (2007) Sampling techniques. John Wiley & Sons – reference: van HoevenLRJanssenMPRoesKCKoffijbergHAiming for a representative sample: simulating random versus purposive strategies for hospital selectionBMC Med Res Methodol20151519010.1186/s12874-015-0089-8 – reference: BreckenridgeJJonesDDemystifying theoretical sampling in grounded theory researchGrounded Theory Rev200982112126 – reference: Easterbrook S, Singer J, Storey MA, Damian D (2008) Selecting empirical methods for software engineering research. In: Guide to advanced empirical software engineering, Springer, pp 285–311 – reference: Cosentino V, Izquierdo JLC, Cabot J (2016) Findings from github: methods, datasets and limitations. In: 2016 IEEE/ACM 13th working conference on Mining Software Repositories (MSR), IEEE, pp 137–141 – reference: Sjøberg D, Anda B, Arisholm E, Dyba T, Jørgensen M, Karahasanovic A, Koren EF, Vokac M (2002) Conducting realistic experiments in software engineering. In: 2002 international symposium on empirical software engineering. IEEE, Nara, Japan, pp 17–26, DOI https://doi.org/10.1109/ISESE.2002.1166921, (to appear in print) – reference: de Mello RM, Travassos GH (2015) Characterizing sampling frames in software engineering surveys. In: Proceedings of the Ibero-American conference on sofware engineering (CibSE) – reference: Russo D, Stol K (in press) Gender differences in personality traits of software engineers. IEEE Trans Softw Eng https://doi.org/10.1109/TSE.2020.3003413 – reference: Torchiano M, Fernández DM, Travassos GH, de Mello RM (2017) Lessons learnt in conducting survey research. In: 2017 IEEE/ACM 5th international workshop on Conducting Empirical Studies in Industry (CESI), IEEE, pp 33–39 – reference: PaulsonJWSucciGEberleinAAn empirical study of open-source and closed-source software productsIEEE Trans Softw Eng200430424625610.1109/TSE.2004.1274044 – reference: ChecklandPHolwellSAction research: its nature and validitySyst Pract Action Res199811192110.1023/A:1022908820784 – reference: LeeASBaskervilleRLGeneralizing generalizability in information systems researchInf Syst Res200314322124310.1287/isre.14.3.221.16560 – reference: Valliant R, Dever JA, Kreuter F (2018) Designing multistage samples. In: Practical tools for designing and weighting survey samples, Springer, pp 209–264 – reference: LandonELJrBanksSKRelative efficiency and bias of Plus-One telephone samplingJ Mark Res197714329410.2307/3150766 – reference: MoherDLiberatiATetzlaffJAltmanDGGroupPPreferred reporting items for systematic reviews and meta-analyses: the prisma statementPLoS Med200967e100009710.1371/journal.pmed.1000097 – reference: Wohlin C, Runeson P, Höst M, Ohlsson MC, Regnell B, Wesslén A (2012) Experimentation in software engineering. Springer Science & Business Media – reference: Sedano T, Ralph P, Péraire C (2019) The product backlog. In: 2019 IEEE/ACM 41St international conference on software engineering (ICSE), IEEE, pp 200–211 – reference: Ralph P, Ali Nb, Baltes S, Bianculli D, Diaz J, Dittrich Y, Ernst N, Felderer M, Feldt R, Filieri A et al (2020a) Empirical standards for software engineering research. arXiv:201003525 – reference: Nagappan M, Zimmermann T, Bird C (2013) Diversity in software engineering research. In: Proceedings of the 9th joint meeting on foundations of software engineering, ACM, pp 466–476 – reference: Baltes S, Ralph P (2020) Sampling in software engineering research supplementary material [data set]. https://doi.org/10.5281/zenodo.3666824 – reference: DillmanDASmythJDChristianLMInternet, phone, mail and mixed-mode surveys: the tailored design method20144th edn.HobokenJohn Wiley & Sons – reference: Kitchenham BA, Pfleeger SL (2008) Personal opinion surveys. In: Guide to advanced empirical software engineering, Springer, pp 63–92 – reference: de MelloRMDa SilvaPCTravassosGHInvestigating probabilistic sampling approaches for large-scale surveys in software engineeringJ Softw Eng Res Dev20153112610.1186/s40411-015-0023-0 – reference: JohnstonLGSabinKSampling hard-to-reach populations with respondent driven samplingMethodological Innovations Online201052384810.4256/mio.2010.0017 – reference: Glaser BG, Strauss AL (2017) Discovery of grounded theory: strategies for qualitative research. Routledge – reference: HenrichJHeineSJNorenzayanAThe weirdest people in the world?Behav Brain Sci2010332-3618310.1017/S0140525X0999152X – reference: Ingram C, Drachen A (2020) How software practitioners use informal local meetups to share software engineering knowledge. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE – reference: MaalejWRobillardMPPatterns of knowledge in api reference documentationIEEE Trans Softw Eng20133991264128210.1109/TSE.2013.12 – reference: Baltes S, Dumani L, Treude C, Diehl S (2018) SOTOrrent: reconstructing and analyzing the evolution stack overflow posts. In: Zaidman A, Hill E, Kamei Y (eds) 15th international conference on Mining Software Repositories (MSR 2018), ACM, Gothenburg, Sweden, pp 319–330 – reference: GubaEGLincolnYSEpistemological and methodological bases of naturalistic inquiryEduc Commun Technol J198230423325210.1007/BF02765185 – reference: KruskalWMostellerFRepresentative sampling, i: mon-scientific literatureInt Stat Rev1979471132410.2307/1403202 – reference: KitchenhamBPfleegerSLPrinciples of survey research: Part 5: populations and samplesACM SIGSOFT Softw Eng Notes2002275172010.1145/571681.571686 – reference: TheisenCDunaiskiMWilliamsLVisserWSoftware engineering research at the international conference on software engineering in 2016ACM SIGSOFT Software Engineering Notes20184241710.1145/3149485.3149496 – reference: CharmazKConstructing grounded theory2014LondonSage – reference: GentlesSJCharlesCPloegJMcKibbonKASampling in qualitative research: insights from an overview of the methods literatureQual Rep2015201117721789 – reference: McElreath R (2020) Statistical rethinking: a Bayesian course with examples in R and Stan. CRC press – reference: Mohanani R, Turhan B, Ralph P (2019) Requirements framing affects design creativity. IEEE Trans Softw Eng – reference: CohenJStatistical power analysis for the behavioral sciences1988HillsdaleLawrence Erlbaum Associates0747.62110 – reference: Patton MQ (2014) Qualitative research & evaluation methods: integrating theory and practice. Sage Publications – reference: TrostJEStatistically nonrepresentative stratified sampling: a sampling technique for qualitative studiesQual Sociol198691545710.1007/BF00988249 – reference: Zannier C, Melnik G, Maurer F (2006) On the success of empirical studies in the international conference on software engineering. In: Proceedings of the 28th international conference on software engineering, pp 341–350 – reference: MilesMBHubermanAMSaldañaJQualitative data analysis: a methods sourcebook20144th edn.Thousand Oaks, California, USASage – reference: Thomas G, Myers K (2015) The anatomy of the case study. Sage – reference: Kitchenham B, Charters S (2007) Guidelines for performing systematic literature reviews in software engineering. Tech. rep., Keele University and University of Durham – reference: Tempero E, Anslow C, Dietrich J, Han T, Li J, Lumpe M, Melton H, Noble J (2010) The qualitas corpus: a curated collection of java code for empirical studies. In: Proceedings of the 17th Asia Pacific software engineering conference. IEEE, Sydney, Australia, pp 336–345, DOI https://doi.org/10.1109/APSEC.2010.46, (to appear in print) – reference: Ralph P, Baltes S, Adisaputri G, Torkar R, Kovalenko V, Kalinowski M, Novielli N, Yoo S, Devroey X, Tan X et al (2020b) Pandemic programming: how covid-19 affects software developers and how their organizations can help. Empir Softw Eng https://doi.org/10.1007/s10664-020-09875-y – reference: ThompsonSKAdaptive cluster samplingJ Am Stat Assoc19908541210501059113450110.1080/01621459.1990.10474975 – reference: Zhang H, Huang X, Zhou X, Huang H, Babar MA (2019) Ethnographic research in software engineering: a critical review and checklist. In: Proceedings of the 2019 27th ACM joint meeting on European software engineering conference and symposium on the foundations of software engineering, pp 659–670 – reference: Humbatova N, Jahangirova G, Bavota G, Riccio V, Stocco A, Tonella P (2020) Taxonomy of real faults in deep learning systems. In: 2020 IEEE/ACM 42nd international conference on software engineering (ICSE), IEEE – reference: TrochimWMDonnellyJPResearch methods knowledge base, vol 22001Cincinnati, OH, USAAtomic Dog Publishing – reference: DrauckerCBMartsolfDSRossRRuskTBTheoretical sampling and category development in grounded theoryQual Health Res20071781137114810.1177/1049732307308450 – reference: Easton G (2010) One case study is enough. Lancaster University technical report https://eprints.lancs.ac.uk/id/eprint/49016/ – reference: Caine K (2016) Local standards for sample size at chi. In: Proceedings of the 2016 CHI conference on human factors in computing systems, ACM, New York, NY, USA, CHI ’16, pp 981–992 – ident: 10072_CR5 doi: 10.1145/3196398.3196430 – ident: 10072_CR29 doi: 10.1109/MSR.2013.6624034 – ident: 10072_CR69 doi: 10.1109/APSEC.2010.46 – volume-title: Case study research: Design and methods year: 2018 ident: 10072_CR81 – volume: 12 start-page: 55 issue: 1 year: 2005 ident: 10072_CR77 publication-title: Environ Ecol Stat doi: 10.1007/s10651-005-6818-0 – ident: 10072_CR8 doi: 10.1145/2858036.2858498 – volume: 2 start-page: 109 issue: 2 year: 2015 ident: 10072_CR55 publication-title: J Exp Polit Sci doi: 10.1017/XPS.2015.19 – ident: 10072_CR49 doi: 10.1201/9780429029608 – ident: 10072_CR62 doi: 10.1109/TSE.2020.3003413 – ident: 10072_CR37 doi: 10.1145/3377811.3380333 – ident: 10072_CR33 doi: 10.4135/9781412985451 – volume: 9 start-page: 54 issue: 1 year: 1986 ident: 10072_CR76 publication-title: Qual Sociol doi: 10.1007/BF00988249 – volume: 5 start-page: 38 issue: 2 year: 2010 ident: 10072_CR38 publication-title: Methodological Innovations Online doi: 10.4256/mio.2010.0017 – volume: 12 start-page: 105 issue: 1 year: 2008 ident: 10072_CR47 publication-title: AIDS and Behavior doi: 10.1007/s10461-008-9421-1 – ident: 10072_CR48 doi: 10.4324/9781315422657 – volume-title: Constructing grounded theory year: 2014 ident: 10072_CR9 – ident: 10072_CR36 doi: 10.1145/3377811.3380395 – ident: 10072_CR13 doi: 10.1145/2901739.2901776 – ident: 10072_CR74 doi: 10.1109/CESI.2017.5 – ident: 10072_CR60 – ident: 10072_CR1 doi: 10.1145/3183440.3195001 – volume-title: Statistical power analysis for the behavioral sciences year: 1988 ident: 10072_CR12 – volume: 39 start-page: 1264 issue: 9 year: 2013 ident: 10072_CR46 publication-title: IEEE Trans Softw Eng doi: 10.1109/TSE.2013.12 – ident: 10072_CR83 doi: 10.1145/3338906.3338976 – volume: 63 start-page: 602 issue: 7 year: 2008 ident: 10072_CR2 publication-title: Am Psychol doi: 10.1037/0003-066X.63.7.602 – volume: 47 start-page: 13 issue: 1 year: 1979 ident: 10072_CR42 publication-title: Int Stat Rev doi: 10.2307/1403202 – ident: 10072_CR65 doi: 10.1109/ICSE.2019.00036 – volume: 14 start-page: 294 issue: 3 year: 1977 ident: 10072_CR44 publication-title: J Mark Res doi: 10.2307/3150766 – ident: 10072_CR50 – volume: 23 start-page: 3801 issue: 6 year: 2018 ident: 10072_CR23 publication-title: Empir Softw Eng doi: 10.1007/s10664-018-9655-0 – volume: 15 start-page: 90 issue: 1 year: 2015 ident: 10072_CR34 publication-title: BMC Med Res Methodol doi: 10.1186/s12874-015-0089-8 – ident: 10072_CR73 doi: 10.1007/978-1-4614-3876-2_13 – ident: 10072_CR79 doi: 10.1145/2702123.2702549 – ident: 10072_CR82 doi: 10.1145/1134285.1134333 – volume: 8 start-page: 112 issue: 2 year: 2009 ident: 10072_CR7 publication-title: Grounded Theory Rev – volume: 26 start-page: 790 issue: 4 year: 1997 ident: 10072_CR22 publication-title: J Adv Nurs doi: 10.1046/j.1365-2648.1997.00371.x – volume: 33 start-page: 61 issue: 2-3 year: 2010 ident: 10072_CR32 publication-title: Behav Brain Sci doi: 10.1017/S0140525X0999152X – ident: 10072_CR57 – ident: 10072_CR11 – ident: 10072_CR53 – ident: 10072_CR56 doi: 10.1145/2491411.2491415 – volume: 3 start-page: 1 issue: 1 year: 2015 ident: 10072_CR51 publication-title: J Softw Eng Res Dev doi: 10.1186/s40411-015-0023-0 – ident: 10072_CR66 doi: 10.1109/ISESE.2002.1166921 – ident: 10072_CR19 doi: 10.1007/978-1-84800-044-5_11 – volume: 17 start-page: 1137 issue: 8 year: 2007 ident: 10072_CR17 publication-title: Qual Health Res doi: 10.1177/1049732307308450 – ident: 10072_CR78 doi: 10.1007/978-3-319-93632-1_9 – volume-title: Qualitative data analysis: a methods sourcebook year: 2014 ident: 10072_CR52 – ident: 10072_CR61 doi: 10.1007/s10664-020-09875-y – volume: 20 start-page: 1772 issue: 11 year: 2015 ident: 10072_CR26 publication-title: Qual Rep – volume: 44 start-page: 174 issue: 2 year: 1997 ident: 10072_CR31 publication-title: Soc Probl doi: 10.2307/3096941 – ident: 10072_CR71 doi: 10.4135/9781473920156 – ident: 10072_CR41 doi: 10.1007/978-1-84800-044-5_3 – ident: 10072_CR4 doi: 10.5281/zenodo.3666824 – ident: 10072_CR25 – volume: 98 start-page: 161 year: 2018 ident: 10072_CR63 publication-title: Inform Software Technol doi: 10.1016/j.infsof.2017.12.001 – volume: 42 start-page: 1 issue: 4 year: 2018 ident: 10072_CR70 publication-title: ACM SIGSOFT Software Engineering Notes doi: 10.1145/3149485.3149496 – ident: 10072_CR27 doi: 10.4324/9780203793206 – volume: 107 start-page: 6743 issue: 15 year: 2010 ident: 10072_CR28 publication-title: Proceedings of the National Academy of Sciences doi: 10.1073/pnas.1000261107 – volume: 45 start-page: 712 issue: 7 year: 2019 ident: 10072_CR59 publication-title: IEEE Trans Softw Eng doi: 10.1109/TSE.2018.2796554 – ident: 10072_CR14 doi: 10.4135/9781452272047 – volume-title: Internet, phone, mail and mixed-mode surveys: the tailored design method year: 2014 ident: 10072_CR16 doi: 10.1002/9781394260645 – volume: 85 start-page: 1050 issue: 412 year: 1990 ident: 10072_CR72 publication-title: J Am Stat Assoc doi: 10.1080/01621459.1990.10474975 – volume: 14 start-page: 221 issue: 3 year: 2003 ident: 10072_CR45 publication-title: Inf Syst Res doi: 10.1287/isre.14.3.221.16560 – volume: 27 start-page: 11 issue: 3 year: 2018 ident: 10072_CR67 publication-title: ACM Transactions on Software Engineering and Methodology (TOSEM) doi: 10.1145/3241743 – volume: 47 start-page: 245 issue: 3 year: 1979 ident: 10072_CR43 publication-title: Int Stat Rev doi: 10.2307/1402647 – ident: 10072_CR15 doi: 10.1145/2961111.2962632 – volume: 23 start-page: 452 issue: 1 year: 2018 ident: 10072_CR21 publication-title: Empir Softw Eng doi: 10.1007/s10664-017-9523-3 – volume: 11 start-page: 9 issue: 1 year: 1998 ident: 10072_CR10 publication-title: Syst Pract Action Res doi: 10.1023/A:1022908820784 – ident: 10072_CR39 – volume: 30 start-page: 246 issue: 4 year: 2004 ident: 10072_CR58 publication-title: IEEE Trans Softw Eng doi: 10.1109/TSE.2004.1274044 – volume-title: Research methods knowledge base, vol 2 year: 2001 ident: 10072_CR75 – volume: 6 start-page: e1000097 issue: 7 year: 2009 ident: 10072_CR54 publication-title: PLoS Med doi: 10.1371/journal.pmed.1000097 – volume: 30 start-page: 233 issue: 4 year: 1982 ident: 10072_CR30 publication-title: Educ Commun Technol J doi: 10.1007/BF02765185 – ident: 10072_CR20 – volume: 50 start-page: 860 issue: 9 year: 2008 ident: 10072_CR6 publication-title: Inf Softw Technol doi: 10.1016/j.infsof.2007.09.004 – volume: 47 start-page: 381 issue: 6 year: 1954 ident: 10072_CR24 publication-title: J Exp Psychol doi: 10.1037/h0055392 – ident: 10072_CR35 doi: 10.1145/3180155.3180235 – volume: 44 start-page: 409 issue: 4 year: 2003 ident: 10072_CR64 publication-title: Res High Educ doi: 10.1023/A:1024232915870 – ident: 10072_CR68 doi: 10.1145/2884781.2884833 – ident: 10072_CR18 – volume: 27 start-page: 17 issue: 5 year: 2002 ident: 10072_CR40 publication-title: ACM SIGSOFT Softw Eng Notes doi: 10.1145/571681.571686 – ident: 10072_CR3 doi: 10.1145/2961111.2962628 – ident: 10072_CR80 doi: 10.1007/978-3-642-29044-2
SSID	ssj0009745
Score	2.6808794
Snippet	Representative sampling appears rare in empirical software engineering research. Not all studies need representative samples, but a general lack of...
SourceID	proquest crossref springer
SourceType	Aggregation Database Enrichment Source Index Database Publisher
SubjectTerms	Compilers Computer Science Engineering research Guidelines Interpreters Programming Languages Qualitative research Random sampling Software engineering Software Engineering/Programming and Operating Systems
SummonAdditionalLinks	– databaseName: SpringerLINK Contemporary 1997-Present dbid: RSV link: http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwnV3NS8MwFH_o9ODF-YnTKTl400DbpE3qTcQhCEOcym4lX50D6WTd9N836VKrooIeQ5NQ3sv7gPfe7wdwTLl1vIYJHNJYYkq0sn6QKpwLJajO0yQ1qiKbYP0-Hw7TGz8UVtbd7nVJsvLUH4bdkoRi11Lg1taOl2HFhjvuzPF28NBA7bKKmtiB62FiI7oflfn-js_hqMkxv5RFq2jTa__vPzdg3WeX6HzxHDZhyRRb0K6ZG5A35G24HgjXSV6M0LhApfXEr2JqkGmwCZEHAXo8QwIpT4eAFnMuSBQajeYOH8v1zO_Afe_y7uIKe1oFrKy9zbBWSskw1hFLhKS5VaEMiA5izpXJGTOC2BAVKUqktc4olLFJrJ5TyYNUmYDnZBdaxaQwe4BIkFpxy0SHIqGKEx4Qqm0SaViYx4KZDoS1dDPlMccd9cVT1qAlO2llVlrVOsp4B07ezzwvEDd-3d2tlZZ56yuzyCap1BWq4w6c1kpqPv982_7fth_AWuSmIaru3S60ZtO5OYRV9TIbl9Oj6lW-ATpq26Q priority: 102 providerName: Springer Nature
Title	Sampling in software engineering research: a critical review and guidelines
URI	https://link.springer.com/article/10.1007/s10664-021-10072-8 https://www.proquest.com/docview/2656444665
Volume	27
WOSCitedRecordID	wos000788586200002&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText	1
inHoldings	1
isFullTextHit
isPrint
journalDatabaseRights	– providerCode: PRVAVX databaseName: SpringerLink Contemporary customDbUrl: eissn: 1573-7616 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0009745 issn: 1382-3256 databaseCode: RSV dateStart: 19970101 isFulltext: true titleUrlDefault: https://link.springer.com/search?facet-content-type=%22Journal%22 providerName: Springer Nature
link	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV3NS8MwFH_o5sGL8xOnc-TgTYNtk7apF1HZEIQxNpXhpaRJOgfSzXXTf9-kSy0KevESCG1Dycv7IO-93w_glDJteFXIsUv9BFMihbaDVOCUC05lGgWREgXZRNjrsdEo6tsLt9yWVZY2sTDUcirMHfmFpwMPapKP_tXsDRvWKJNdtRQa61A3KAleUbo3rEB3w4Kk2MDsYaJ9u22asa1zQUCxKVAwc20VvjumKtr8kSAt_E638d8_3oYtG3Gi69UR2YE1le1Co2RzQFa59-B-yE11eTZGkwzl2jp_8LlCqsIrRBYY6OUScSQsRQJa9b4gnkk0XhrMLFNHvw-P3c7D7R22VAtYaB1cYCmESFxfemHAE5pqsSYOkY7PmFBpGCpOtNvyBCWJ1ljPTXwVaNlHCXMioRyWkgOoZdNMHQIiTqQ3PgmkywMqGGEOoVIHlip0U5-Hqgluuc-xsDjkhg7jNa4QlI1sYi2bYu7FrAlnX9_MVigcf77dKgUSW43M40oaTTgvRVo9_n21o79XO4ZNz3REFBW8Lagt5kt1AhvifTHJ522o33R6_UG7OJd67PvPehwMnz4B8cjpiA
linkProvider	ProQuest
linkToHtml	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3LSsQwFL34At34FsdnFrrSYNukbSqIiA-U0UFQwV1Nk3QUpOrMqPhTfqM3ndSioDsXLkPbQJqTk9vm3nMA1rhA4jWxpD4PM8qZVsiDXNFcKsl1nkSJUaXZRNxqievr5HwA3qtaGJtWWXFiSdT6Qdl_5FsBBh7cHj6Gu49P1LpG2dPVykKjD4umeXvFT7buzskBzu96EBwdXu4fU-cqQBXCrUe1UirzQx3Ekcx4jiPIPKa9UAhl8jg2kiFDB4qzDMEZ-FloIhxmkgkvUcYTOcN-B2GYW_YvUwUvapHfuDRFtrJ-lGEs4Yp0XKleFHFqEyJsG1no60ZYR7ffDmTLfe5o4r-9oUkYdxE12esvgSkYMMU0TFRuFcSR1ww0L6TNni_a5K4gXdx9XmXHEFPrMRInfHS7TSRRzgKC9Gt7iCw0aT9bTTBbJzALV38ypDkYKh4KMw-EeQlOdBZpX0ZcCSY8xjUGzib281DGpgF-Na-pcjrr1u7jPq0Voi0WUsRC2Q5S0YCNz2ce-yojv969VAEgdYzTTevZb8BmBaH68s-9Lfze2yqMHl-enaanJ63mIowFtvqjzFZegqFe59ksw4h66d11OyvlWiBw89fQ-gBVREXY
linkToPdf	http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1NT9wwEB1RqKpegH4gtlDqA5xaiyR2YgcJVVWXVdGi1UptpVUvqWM7gFQFullA_DV-HePEIQKpe9tDj1ESS46fn8fxzHsAu1wi8VqhaMjjnHJmNPIg17RQWnFTpElqdW02IUYjOZmk4yW4a2thXFply4k1UZsL7f6R70cYeHB3-BjvFz4tYtwffL78S52DlDtpbe00GogM7e0Nbt-qw-M-jvVeFA2Ofnz9Rr3DANUIvRk1Wus8jE0kEpXzAnuTB8wEsZTaFkJYxZCtI81ZjkCNwjy2CXY5zWWQahvIgmG7z2BF4B7TbfzG8a9O8FfUBslO4o8yjCt8wY4v20sSTl1yhLtGRnq8KHaR7pPD2XrNG6z9z19rHVZ9pE2-NFPjFSzZ8jWstS4WxJPaGxh-Vy6rvjwl5yWpcFW6UVNLbKfTSLwg0tkBUUR7awjS1PwQVRpyeuW0wlz9wFv4uZAubcByeVHaTSAsSHHQ88SEKuFaMhkwbjCgtiIsYiVsD8J2jDPt9dedDcifrFOOdrjIEBf1dZTJHnx8eOeyUR-Z-_R2C4bMM1GVdUjowacWTt3tf7f2bn5rH-AFIio7OR4Nt-Bl5IpC6iTmbVieTa_se3iur2fn1XSnnhYEfi8aWfeCuk67
openUrl	ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Sampling+in+software+engineering+research%3A+a+critical+review+and+guidelines&rft.jtitle=Empirical+software+engineering+%3A+an+international+journal&rft.au=Baltes%2C+Sebastian&rft.au=Ralph%2C+Paul&rft.date=2022-07-01&rft.pub=Springer+Nature+B.V&rft.issn=1382-3256&rft.eissn=1573-7616&rft.volume=27&rft.issue=4&rft_id=info:doi/10.1007%2Fs10664-021-10072-8&rft.externalDBID=HAS_PDF_LINK
thumbnail_l	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=1382-3256&client=summon
thumbnail_m	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=1382-3256&client=summon
thumbnail_s	http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=1382-3256&client=summon