Identifying, categorizing and mitigating threats to validity in software engineering secondary studies
Secondary studies are vulnerable to threats to validity. Although, mitigating these threats is crucial for the credibility of these studies, we currently lack a systematic approach to identify, categorize and mitigate threats to validity for secondary studies. In this paper, we review the corpus of...
Saved in:
| Published in: | Information and software technology Vol. 106; pp. 201 - 230 |
|---|---|
| Main Authors: | , , , , |
| Format: | Journal Article |
| Language: | English |
| Published: |
Elsevier B.V
01.02.2019
|
| Subjects: | |
| ISSN: | 0950-5849, 1873-6025 |
| Online Access: | Get full text |
| Tags: |
Add Tag
No Tags, Be the first to tag this record!
|
| Abstract | Secondary studies are vulnerable to threats to validity. Although, mitigating these threats is crucial for the credibility of these studies, we currently lack a systematic approach to identify, categorize and mitigate threats to validity for secondary studies.
In this paper, we review the corpus of secondary studies, with the aim to identify: (a) the trend of reporting threats to validity, (b) the most common threats to validity and corresponding mitigation actions, and (c) possible categories in which threats to validity can be classified.
To achieve this goal we employ the tertiary study research method that is used for synthesizing knowledge from existing secondary studies. In particular, we collected data from more than 100 studies, published until December 2016 in top quality software engineering venues (both journals and conference).
Our results suggest that in recent years, secondary studies are more likely to report their threats to validity. However, the presentation of such threats is rather ad hoc, e.g., the same threat may be presented with a different name, or under a different category. To alleviate this problem, we propose a classification schema for reporting threats to validity and possible mitigation actions. Both the classification of threats and the associated mitigation actions have been validated by an empirical study, i.e., Delphi rounds with experts.
Based on the proposed schema, we provide a checklist, which authors of secondary studies can use for identifying and categorizing threats to validity and corresponding mitigation actions, while readers of secondary studies can use the checklist for assessing the validity of the reported results. |
|---|---|
| AbstractList | Secondary studies are vulnerable to threats to validity. Although, mitigating these threats is crucial for the credibility of these studies, we currently lack a systematic approach to identify, categorize and mitigate threats to validity for secondary studies.
In this paper, we review the corpus of secondary studies, with the aim to identify: (a) the trend of reporting threats to validity, (b) the most common threats to validity and corresponding mitigation actions, and (c) possible categories in which threats to validity can be classified.
To achieve this goal we employ the tertiary study research method that is used for synthesizing knowledge from existing secondary studies. In particular, we collected data from more than 100 studies, published until December 2016 in top quality software engineering venues (both journals and conference).
Our results suggest that in recent years, secondary studies are more likely to report their threats to validity. However, the presentation of such threats is rather ad hoc, e.g., the same threat may be presented with a different name, or under a different category. To alleviate this problem, we propose a classification schema for reporting threats to validity and possible mitigation actions. Both the classification of threats and the associated mitigation actions have been validated by an empirical study, i.e., Delphi rounds with experts.
Based on the proposed schema, we provide a checklist, which authors of secondary studies can use for identifying and categorizing threats to validity and corresponding mitigation actions, while readers of secondary studies can use the checklist for assessing the validity of the reported results. |
| Author | Avgeriou, Paris Chatzigeorgiou, Alexander Verbeek, Marijn Bibi, Stamatia Ampatzoglou, Apostolos |
| Author_xml | – sequence: 1 givenname: Apostolos surname: Ampatzoglou fullname: Ampatzoglou, Apostolos email: apostolos.ampatzoglou@gmail.com organization: Department of Applied Informatics, University of Macedonia, Thessaloniki, Greece – sequence: 2 givenname: Stamatia orcidid: 0000-0003-4248-3752 surname: Bibi fullname: Bibi, Stamatia organization: Department of Informatics and Telecommunications, University of Western Macedonia, Kozani, Greece – sequence: 3 givenname: Paris surname: Avgeriou fullname: Avgeriou, Paris organization: Department of Mathematics and Computer Science, University of Groningen, the Netherlands – sequence: 4 givenname: Marijn surname: Verbeek fullname: Verbeek, Marijn organization: Department of Mathematics and Computer Science, University of Groningen, the Netherlands – sequence: 5 givenname: Alexander surname: Chatzigeorgiou fullname: Chatzigeorgiou, Alexander organization: Department of Applied Informatics, University of Macedonia, Thessaloniki, Greece |
| BookMark | eNqFUMtOwzAQtFCRaAt_wMEfQMI6idOEAxKqeFSqxAXOkWuvy1atjWxTVL6eROXEAU6rGe3M7syEjZx3yNilgFyAqK83OTkbvc0LEE1P5QD1CRuLZlZmNRRyxMbQSshkU7VnbBLjBkDMoIQxswuDLpE9kFtfca0Srn2grx5x5QzfUaK1SgNMbwFVijx5vldbMpQOnBzvz6ZPFZCjW5NDDMNuRO2dUeHAY_owhPGcnVq1jXjxM6fs9eH-Zf6ULZ8fF_O7ZaYrkCkrZbkSLZSqqXXRrEDXTWUKW7ZqJq2oQKtKFUVVV1hY3UoDKyUKaG1jjNVCQjllN0dfHXyMAW2nKfX_e5eCom0noBsa6zbdsbFuaGxg-8Z6cfVL_B5o16f4T3Z7lGEfbE8YuqgJnUZDAXXqjKe_Db4BfM-MlQ |
| CitedBy_id | crossref_primary_10_1155_2020_8830683 crossref_primary_10_1016_j_cose_2024_103988 crossref_primary_10_1016_j_infsof_2023_107303 crossref_primary_10_1109_ACCESS_2022_3223130 crossref_primary_10_1016_j_jss_2020_110892 crossref_primary_10_1016_j_jss_2021_111189 crossref_primary_10_1016_j_jss_2021_111187 crossref_primary_10_1145_3487043 crossref_primary_10_3390_land13111758 crossref_primary_10_1016_j_csi_2025_104073 crossref_primary_10_1016_j_jss_2022_111592 crossref_primary_10_1007_s10462_024_10902_3 crossref_primary_10_1109_ACCESS_2023_3321666 crossref_primary_10_1109_ACCESS_2021_3089100 crossref_primary_10_1134_S0361768822080175 crossref_primary_10_2478_cait_2024_0037 crossref_primary_10_1016_j_datak_2024_102374 crossref_primary_10_1108_IJPDLM_04_2020_0103 crossref_primary_10_1016_j_infsof_2025_107796 crossref_primary_10_1016_j_infsof_2024_107551 crossref_primary_10_1145_3643889 crossref_primary_10_1145_3520480 crossref_primary_10_1145_3659098 crossref_primary_10_1016_j_infsof_2022_106964 crossref_primary_10_1007_s10639_022_11281_4 crossref_primary_10_1007_s10664_024_10480_6 crossref_primary_10_1016_j_cose_2021_102366 crossref_primary_10_1145_3747178 crossref_primary_10_1155_2022_9701739 crossref_primary_10_1016_j_cosrev_2024_100634 crossref_primary_10_1080_14413523_2024_2372122 crossref_primary_10_1016_j_infsof_2019_106176 crossref_primary_10_1016_j_jss_2025_112613 crossref_primary_10_1145_3616372 crossref_primary_10_1145_3708527 crossref_primary_10_1002_smr_2464 crossref_primary_10_1016_j_jss_2025_112334 crossref_primary_10_1109_ACCESS_2019_2949905 crossref_primary_10_1016_j_infsof_2023_107169 crossref_primary_10_1007_s10664_021_10084_4 crossref_primary_10_1007_s10664_023_10345_4 crossref_primary_10_1134_S0361768822080229 crossref_primary_10_15388_25_INFOR605 crossref_primary_10_1016_j_infsof_2024_107607 crossref_primary_10_1016_j_scico_2021_102652 crossref_primary_10_1016_j_infsof_2024_107568 crossref_primary_10_1177_21582440251314671 crossref_primary_10_1016_j_infsof_2025_107722 crossref_primary_10_1016_j_jss_2020_110752 crossref_primary_10_1016_j_jss_2021_111004 crossref_primary_10_1109_TSE_2022_3174092 crossref_primary_10_1007_s13369_023_07945_z crossref_primary_10_1007_s11356_023_30752_w crossref_primary_10_1016_j_infsof_2024_107609 crossref_primary_10_1002_smr_2370 crossref_primary_10_1016_j_jss_2023_111771 crossref_primary_10_1080_23744731_2020_1757327 crossref_primary_10_1145_3649598 crossref_primary_10_1007_s10270_020_00790_w crossref_primary_10_1007_s10664_022_10285_5 crossref_primary_10_1002_stvr_1748 crossref_primary_10_1016_j_infsof_2022_107131 crossref_primary_10_1016_j_infsof_2023_107375 crossref_primary_10_1016_j_pmcj_2024_101931 crossref_primary_10_1016_j_jss_2023_111899 crossref_primary_10_1109_ACCESS_2022_3169785 crossref_primary_10_7717_peerj_cs_1097 crossref_primary_10_1016_j_scico_2023_102961 crossref_primary_10_3390_app10217749 crossref_primary_10_1007_s00766_025_00444_7 crossref_primary_10_3390_app112412121 crossref_primary_10_1007_s10207_024_00939_4 crossref_primary_10_1016_j_cogsys_2022_10_012 crossref_primary_10_1016_j_infsof_2022_106946 crossref_primary_10_1016_j_jss_2024_112011 crossref_primary_10_1145_3508359 crossref_primary_10_1007_s10270_020_00839_w crossref_primary_10_1145_3709357 crossref_primary_10_1049_sfw2_12064 crossref_primary_10_1007_s11831_023_09981_2 crossref_primary_10_1016_j_infsof_2021_106771 crossref_primary_10_1016_j_jss_2023_111829 crossref_primary_10_1016_j_infsof_2024_107469 crossref_primary_10_1016_j_compag_2025_110847 crossref_primary_10_1145_3702974 crossref_primary_10_1016_j_jss_2021_111148 crossref_primary_10_1016_j_infsof_2021_106814 crossref_primary_10_1007_s10207_022_00610_w crossref_primary_10_1016_j_jss_2024_112241 crossref_primary_10_1016_j_cie_2025_111042 crossref_primary_10_1145_3639060 crossref_primary_10_1145_3563395 crossref_primary_10_1145_3510460 crossref_primary_10_1007_s00766_023_00407_w crossref_primary_10_1016_j_ipm_2025_104263 crossref_primary_10_1049_sfw2_12011 crossref_primary_10_1007_s10664_023_10332_9 crossref_primary_10_1177_07356331231203251 crossref_primary_10_1016_j_jss_2023_111679 crossref_primary_10_1016_j_infsof_2024_107472 crossref_primary_10_1016_j_infsof_2025_107870 crossref_primary_10_1016_j_jss_2020_110608 crossref_primary_10_1016_j_jss_2020_110848 crossref_primary_10_1145_3654440 crossref_primary_10_1016_j_infsof_2024_107459 crossref_primary_10_1109_ACCESS_2024_3394732 crossref_primary_10_1016_j_infsof_2019_04_006 crossref_primary_10_1016_j_atech_2025_100889 crossref_primary_10_1007_s40964_025_01262_7 crossref_primary_10_1007_s42979_024_02956_5 crossref_primary_10_1016_j_infsof_2024_107586 crossref_primary_10_1080_08993408_2025_2543095 crossref_primary_10_5753_jserd_2025_4892 crossref_primary_10_1109_ACCESS_2022_3171803 crossref_primary_10_1016_j_infsof_2022_106890 crossref_primary_10_1145_3626522 crossref_primary_10_1007_s00607_025_01432_z crossref_primary_10_1016_j_infsof_2023_107329 crossref_primary_10_1016_j_infsof_2025_107802 crossref_primary_10_1007_s11227_024_06465_3 crossref_primary_10_1177_23970022231226316 crossref_primary_10_1002_smr_2697 crossref_primary_10_1016_j_jss_2024_112187 crossref_primary_10_3390_s23198056 crossref_primary_10_1016_j_jss_2024_112063 crossref_primary_10_1002_smr_2294 crossref_primary_10_1109_ACCESS_2024_3364676 crossref_primary_10_3390_en13112881 crossref_primary_10_1145_3731597 crossref_primary_10_3390_s25133951 crossref_primary_10_3390_electronics13081452 crossref_primary_10_1016_j_jbi_2025_104809 crossref_primary_10_1016_j_infsof_2023_107174 crossref_primary_10_1016_j_jss_2025_112508 crossref_primary_10_1016_j_jksuci_2024_102094 crossref_primary_10_3390_electronics13152989 crossref_primary_10_1016_j_infsof_2023_107338 crossref_primary_10_1109_ACCESS_2021_3101963 crossref_primary_10_1134_S0361768822080084 crossref_primary_10_1145_3690652 crossref_primary_10_1007_s00766_023_00396_w crossref_primary_10_1145_3734216 crossref_primary_10_1145_3572905 crossref_primary_10_1007_s10009_025_00794_1 crossref_primary_10_1007_s10270_022_01049_2 crossref_primary_10_1016_j_datak_2024_102367 crossref_primary_10_1016_j_adhoc_2025_103897 crossref_primary_10_1007_s10664_024_10516_x crossref_primary_10_3390_fi16100353 |
| Cites_doi | 10.1136/jech.52.6.377 10.1016/S0749-3797(99)00122-1 10.1353/etc.2012.0009 10.1016/j.infsof.2011.01.004 10.1016/j.infsof.2013.07.010 10.1016/j.infsof.2015.03.007 10.1007/978-1-4615-4625-2 10.1016/0197-2456(95)00134-4 10.1186/1471-2288-7-10 10.1109/TSE.2013.56 10.1016/j.infsof.2017.10.017 10.1007/s10664-005-1290-x 10.1016/S0895-4356(98)00131-0 10.1016/j.jss.2007.08.028 10.1371/journal.pone.0003081 10.17763/haer.62.3.8323320856251826 10.1016/j.infsof.2011.04.004 10.1186/2046-4053-4-1 10.1177/0193841X16665199 10.1016/j.jss.2013.04.076 10.1016/j.infsof.2008.09.009 10.1007/s10664-008-9102-8 10.1016/j.jss.2010.09.036 10.1016/j.infsof.2010.03.006 10.1016/0951-8320(91)90053-A 10.1007/s11292-007-9046-9 10.1109/ESEM.2009.5314238 |
| ContentType | Journal Article |
| Copyright | 2018 |
| Copyright_xml | – notice: 2018 |
| DBID | AAYXX CITATION |
| DOI | 10.1016/j.infsof.2018.10.006 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Business |
| EISSN | 1873-6025 |
| EndPage | 230 |
| ExternalDocumentID | 10_1016_j_infsof_2018_10_006 S0950584918302106 |
| GroupedDBID | --K --M -~X .DC .~1 0R~ 1B1 1~. 1~5 29I 4.4 457 4G. 5GY 5VS 7-5 71M 77K 8P~ 9JN AABNK AACTN AAEDT AAEDW AAIAV AAIKJ AAKOC AALRI AAOAW AAQFI AAQXK AAXUO AAYFN AAYOK ABBOA ABFNM ABFRF ABJNI ABMAC ABTAH ABXDB ABYKQ ACDAQ ACGFO ACGFS ACGOD ACNNM ACRLP ACZNC ADBBV ADEZE ADJOM ADMUD AEBSH AEFWE AEKER AENEX AFKWA AFTJW AGHFR AGUBO AGYEJ AHHHB AHZHX AIALX AIEXJ AIKHN AITUG AJBFU AJOXV ALMA_UNASSIGNED_HOLDINGS AMFUW AMRAJ AOUOD ASPBG AVWKF AXJTR AZFZN BKOJK BKOMP BLXMC CS3 DU5 EBS EFJIC EFLBG EJD EO8 EO9 EP2 EP3 FDB FEDTE FGOYB FIRID FNPLU FYGXN G-Q G8K GBLVA GBOLZ HLZ HVGLF HZ~ IHE J1W KOM LG9 M41 MO0 MS~ N9A O-L O9- OAUVE OZT P-8 P-9 P2P PC. PQQKQ Q38 R2- RIG ROL RPZ SBC SDF SDG SDP SES SEW SPC SPCBC SSV SSZ T5K TWZ UHS UNMZH WH7 WUQ XFK ZY4 ~G- 77I 9DU AATTM AAXKI AAYWO AAYXX ABDPE ABWVN ACLOT ACRPL ACVFH ADCNI ADNMO AEIPS AEUPX AFJKZ AFPUW AGQPQ AIGII AIIUN AKBMS AKRWK AKYEP ANKPU APXCP CITATION EFKBS ~HD |
| ID | FETCH-LOGICAL-c405t-353b1903a86c28b0c684d2f39a75f140ca4a22464e2fc95d0ba1209f8ddfc1503 |
| ISICitedReferencesCount | 196 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=000454381300012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 0950-5849 |
| IngestDate | Tue Nov 18 21:50:03 EST 2025 Sat Nov 29 07:07:27 EST 2025 Fri Feb 23 02:23:55 EST 2024 |
| IsDoiOpenAccess | false |
| IsOpenAccess | true |
| IsPeerReviewed | true |
| IsScholarly | true |
| Keywords | Empirical software engineering Literature Review Threats to Validity Secondary studies |
| Language | English |
| LinkModel | OpenURL |
| MergedId | FETCHMERGED-LOGICAL-c405t-353b1903a86c28b0c684d2f39a75f140ca4a22464e2fc95d0ba1209f8ddfc1503 |
| ORCID | 0000-0003-4248-3752 |
| OpenAccessLink | https://www.sciencedirect.com/science/article/pii/S0950584918302106 |
| PageCount | 30 |
| ParticipantIDs | crossref_citationtrail_10_1016_j_infsof_2018_10_006 crossref_primary_10_1016_j_infsof_2018_10_006 elsevier_sciencedirect_doi_10_1016_j_infsof_2018_10_006 |
| PublicationCentury | 2000 |
| PublicationDate | 2019-02-01 |
| PublicationDateYYYYMMDD | 2019-02-01 |
| PublicationDate_xml | – month: 02 year: 2019 text: 2019-02-01 day: 01 |
| PublicationDecade | 2010 |
| PublicationTitle | Information and software technology |
| PublicationYear | 2019 |
| Publisher | Elsevier B.V |
| Publisher_xml | – name: Elsevier B.V |
| References | Galster, Weyns, Tofan, Michalik, Avgeriou (bib0013) 2014; 40 Nickerson, Muntermann, Varshney, Isaac (bib0030) 2009 Petersen, Gencel. (bib0035) Oct. 2013 Maxwell (bib0029) 1992; 62 C. Wohlin, M. Host, P. Runeson, M. Ohlsson, B. Regnell, and A. Wesslen, “Experimentation in software engineering: an introduction”, Kluwer Academic Publishers, 2000 Siegmund, Siegmund, Apel (bib0040) 2015 B. Kitchenham et al. "The impact of limited search procedures for systematic literature reviews: a participant-observer case study", Proc. Third Int'l Symp. Empirical Software Eng. and Measurement, pp. 336–345 Moher, Shamseer, Clarke, Ghersi, Liberati, Petticrew, Shekelle, Stewart (bib0031) 2015; 54 Sjoberg, Dyba, Jorgensen (bib0042) 2007 Wohlin, Runeson, Anselmo da Mota Silveira Neto, Engström, do Carmo Machado, Santana de Almeida (bib0048) 2013; 86 Verhagen, de Vet, de Bie, Kessels, Boers, Bouter, Knipschild (bib0045) 1998; 51 Kitchenham, Dybå, Jørgensen (bib0022) May , 2004 Kitchenham, Brereton, Budgen, Turner, Bailey, Linkman (bib0020) January 2009; 51 Shea, Grimshaw, Wells, Boers, Andersson, Hamel, Porter, Tugwell, Moher, Bouter (bib0039) 2007; 7 Marques, Rodrigues, Conte (bib0028) August 2012 da Silva, Santos, Soares, França, Monteiro, Maciel (bib0041) 2011; 53 Downs, Black (bib0010) 1998; 52 Basili, Selby (bib0003) 1991; 32 Jedlitschka, Pfahl (bib0017) 2005 Cai, Card (bib0006) June 2008; 81 Kitchenham, Pretorius, Budgen, Pearl Brereton, Turner, Niazi, Linkman (bib0019) August 2010; 52 Petersen, Vakkalanka, Kuzniarz (bib0034) August 2015; 64 Lethbridge, Sim, Singer (bib0026) Jul. 2005; 10 Zaza, Wright-De Agüero, Briss, Truman, Hopkins, Hennessy, Sosin, Anderson, Carande-Kulis, Teutsch, Pappaioanou (bib0052) 2000; 18 Lincoln, Guba (bib0027) 1985 Feldt, Magazinius (bib0012) July 2010 Zhou, Jin, Zhang, Li, Huang (bib0053) 2016 Bano, Zowghi, Ikram (bib0002) August 2014 (bib0007) 2002; 6 Avellar, Thomas, Kleinman, Sama-Miller, S.E, Coughlin, Westbrook (bib0001) 2017; 41 Verner, Brereton, Kitchenham, Turner (bib0046) 14-15 May 2012 Slocum, Detrich, Spencer (bib0043) 2012; 35 Wong, Tse, Glass, Basili, Chen (bib0050) January 2011; 84 Dwan, Altman, Arnaiz, Bloom, Chan, Cronin, Decullier, Easterbrook, Von Elm, Gamble, Ghersi, Ioannidis, Simes, Williamson, Ghersi (bib0011) 2008; 3 Perry, Porter, Votta (bib0032) 2000 Pfleeger, Kitchenham (bib0036) 2001 Cruzes, Dybå (bib0009) May 2011; 53 Biffl, Kalinowski, Ekaputra, Neto, Conte, Winkler (bib0004) 2014 Imtiaz, Bano, Ikram, Niazi (bib0015) 2013 Cook, Campbell (bib0008) 1979 Jadad, Moore, Carroll, Jenkinson, Reynolds, Gavaghan, McQuay (bib0016) 1999; 17 Kitchenham, Budgen, Brereton (bib0021) 2010 Kitchenham, Charters (bib0018) 2007 Kitchenham, Bereton (bib0024) 2013; 55 Rothstein (bib0037) 2008 Budgen, Brereton, Drummond, Williams (bib0005) 2018; 95 Kitchenham, Budgen, Brereton (bib0025) 2015 Petersen, Feldt, Mujtaba, Mattsson (bib0033) 2008; 8 Wohlin (bib0049) 2014 Greenwood, Levin (bib0014) 2007 Yin (bib0051) 2009 Runeson, Höst (bib0038) December 2009; 14 Tichy, Padberg (bib0044) 2007 Yin (10.1016/j.infsof.2018.10.006_bib0051) 2009 Greenwood (10.1016/j.infsof.2018.10.006_bib0014) 2007 Runeson (10.1016/j.infsof.2018.10.006_bib0038) 2009; 14 Budgen (10.1016/j.infsof.2018.10.006_bib0005) 2018; 95 Sjoberg (10.1016/j.infsof.2018.10.006_bib0042) 2007 Wong (10.1016/j.infsof.2018.10.006_bib0050) 2011; 84 Shea (10.1016/j.infsof.2018.10.006_bib0039) 2007; 7 Kitchenham (10.1016/j.infsof.2018.10.006_bib0022) 2004 Feldt (10.1016/j.infsof.2018.10.006_bib0012) 2010 Jedlitschka (10.1016/j.infsof.2018.10.006_bib0017) 2005 Cai (10.1016/j.infsof.2018.10.006_bib0006) 2008; 81 Dwan (10.1016/j.infsof.2018.10.006_bib0011) 2008; 3 (10.1016/j.infsof.2018.10.006_bib0007) 2002; 6 Bano (10.1016/j.infsof.2018.10.006_bib0002) 2014 Wohlin (10.1016/j.infsof.2018.10.006_bib0049) 2014 Pfleeger (10.1016/j.infsof.2018.10.006_bib0036) 2001 Basili (10.1016/j.infsof.2018.10.006_bib0003) 1991; 32 Zaza (10.1016/j.infsof.2018.10.006_bib0052) 2000; 18 Galster (10.1016/j.infsof.2018.10.006_bib0013) 2014; 40 Verner (10.1016/j.infsof.2018.10.006_bib0046) 2012 Siegmund (10.1016/j.infsof.2018.10.006_bib0040) 2015 Marques (10.1016/j.infsof.2018.10.006_bib0028) 2012 10.1016/j.infsof.2018.10.006_bib0047 Cook (10.1016/j.infsof.2018.10.006_bib0008) 1979 Wohlin (10.1016/j.infsof.2018.10.006_bib0048) 2013; 86 Biffl (10.1016/j.infsof.2018.10.006_bib0004) 2014 Kitchenham (10.1016/j.infsof.2018.10.006_bib0025) 2015 Petersen (10.1016/j.infsof.2018.10.006_bib0034) 2015; 64 Petersen (10.1016/j.infsof.2018.10.006_bib0035) 2013 Moher (10.1016/j.infsof.2018.10.006_bib0031) 2015; 54 Imtiaz (10.1016/j.infsof.2018.10.006_bib0015) 2013 Lethbridge (10.1016/j.infsof.2018.10.006_bib0026) 2005; 10 Petersen (10.1016/j.infsof.2018.10.006_bib0033) 2008; 8 Cruzes (10.1016/j.infsof.2018.10.006_bib0009) 2011; 53 Jadad (10.1016/j.infsof.2018.10.006_bib0016) 1999; 17 Kitchenham (10.1016/j.infsof.2018.10.006_bib0018) 2007 Kitchenham (10.1016/j.infsof.2018.10.006_bib0021) 2010 Rothstein (10.1016/j.infsof.2018.10.006_bib0037) 2008 Avellar (10.1016/j.infsof.2018.10.006_bib0001) 2017; 41 Kitchenham (10.1016/j.infsof.2018.10.006_bib0024) 2013; 55 Perry (10.1016/j.infsof.2018.10.006_bib0032) 2000 Lincoln (10.1016/j.infsof.2018.10.006_bib0027) 1985 da Silva (10.1016/j.infsof.2018.10.006_bib0041) 2011; 53 Kitchenham (10.1016/j.infsof.2018.10.006_bib0019) 2010; 52 Nickerson (10.1016/j.infsof.2018.10.006_bib0030) 2009 Verhagen (10.1016/j.infsof.2018.10.006_bib0045) 1998; 51 Slocum (10.1016/j.infsof.2018.10.006_bib0043) 2012; 35 Downs (10.1016/j.infsof.2018.10.006_bib0010) 1998; 52 Maxwell (10.1016/j.infsof.2018.10.006_bib0029) 1992; 62 Zhou (10.1016/j.infsof.2018.10.006_bib0053) 2016 Kitchenham (10.1016/j.infsof.2018.10.006_bib0020) 2009; 51 Tichy (10.1016/j.infsof.2018.10.006_bib0044) 2007 10.1016/j.infsof.2018.10.006_bib0023 |
| References_xml | – year: 2014 ident: bib0004 article-title: Towards a semantic knowledge base on threats to validity and control actions in controlled experiments publication-title: 8th ACM/IEEE International Symposium on Empirical Software Engineering and Measurement (ESEM) – volume: 54 year: 2015 ident: bib0031 article-title: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement publication-title: Systematic Reviews – year: 2015 ident: bib0025 article-title: Evidence Base Software Engineering and Systematic Reviews – reference: B. Kitchenham et al. "The impact of limited search procedures for systematic literature reviews: a participant-observer case study", Proc. Third Int'l Symp. Empirical Software Eng. and Measurement, pp. 336–345 – volume: 86 start-page: 2594 year: 2013 end-page: 2610 ident: bib0048 article-title: On the reliability of mapping studies in software engineering publication-title: J. Syst. Softw. – year: 2007 ident: bib0018 article-title: Guidelines For Performing Systematic Literature Reviews in Software Engineering – year: 14-15 May 2012 ident: bib0046 article-title: Systematic literature reviews in global software development: a tertiary study publication-title: Evaluation and Assessment in Software Engineering (EASE '12) – volume: 41 start-page: 283 year: 2017 end-page: 325 ident: bib0001 article-title: External validity: the next step for systematic reviews? – volume: 52 start-page: 377 year: 1998 end-page: 384 ident: bib0010 article-title: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions publication-title: Journal of Epidemiology & Community Health – volume: 55 start-page: 2049 year: 2013 end-page: 2075 ident: bib0024 article-title: A systematic review of systematic review process research in software engineering – volume: 14 start-page: 131 year: December 2009 end-page: 164 ident: bib0038 article-title: Guidelines for conducting and reporting case study research in software engineering publication-title: Emp. Softw. Eng. – start-page: 358 year: 2007 end-page: 378 ident: bib0042 article-title: The future of empirical methods in software engineering research publication-title: Proceedings of 2007 Future of Software Engineering – volume: 10 start-page: 311 year: Jul. 2005 end-page: 341 ident: bib0026 article-title: Studying software engineers: data collection techniques for software field studies publication-title: Empirical Softw. Eng. – year: 1985 ident: bib0027 article-title: Naturalistic Inquiry – volume: 53 start-page: 899 year: 2011 end-page: 913 ident: bib0041 article-title: Six years of systematic literature reviews in software engineering: an updated tertiary study publication-title: Inf. Softw. Technol. – volume: 35 start-page: 201 year: 2012 end-page: 233 ident: bib0043 article-title: "Evaluating the validity of systematic reviews to indentify empirically supported treatments publication-title: Educ. Treat. Child. – volume: 18 start-page: 44 year: 2000 end-page: 74 ident: bib0052 article-title: Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services publication-title: Am. J. Prevent. Med. – start-page: 345 year: 2000 end-page: 355 ident: bib0032 article-title: Empirical studies of software engineering: a roadmap publication-title: Proceedings of the conference on the future of Software engineering – year: 2001 ident: bib0036 article-title: Principles of survey research: part 1: turning lemons into lemonade publication-title: , 6 November 2001 – start-page: 374 year: July 2010 end-page: 379 ident: bib0012 article-title: Validity Threats in Empirical Software Engineering Research - An Initial Survey publication-title: Proceedings of the 22nd Int. Conf. on Software Engineering and Knowledge Engineering (SEKE) – start-page: 9 year: August 2014 end-page: 16 ident: bib0002 article-title: Systematic reviews in requirements engineering: a tertiary study”, Empirical Requirements Engineering (EmpiRE) publication-title: 2014 IEEE Fourth International Workshop on – start-page: 10 year: 2005 end-page: 17 ident: bib0017 article-title: Reporting guidelines for controlled experiments in software engineering publication-title: Proceedings of the International Symposium on Empirical Software Engineering – volume: 7 start-page: 10 year: 2007 ident: bib0039 article-title: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews publication-title: BMC Med. Res. Methodol. – volume: 6 start-page: 1 year: 2002 end-page: 4 ident: bib0007 article-title: The Database of Abstracts of Reviews of Effects (DARE) publication-title: Effect. Matters – volume: 64 start-page: 1 year: August 2015 end-page: 18 ident: bib0034 article-title: Guidelines for conducting systematic mapping studies in software engineering: an update publication-title: Information and Software Technology – volume: 32 start-page: 171 year: 1991 end-page: 191 ident: bib0003 article-title: Paradigms for experimentation and empirical studies in software engineering publication-title: Reliab. Eng. Syst. Saf. – volume: 84 start-page: 162 year: January 2011 end-page: 168 ident: bib0050 article-title: An assessment of systems and software engineering scholars and institutions (2003–2007 and 2004–2008) publication-title: J. Syst. Softw. – volume: 8 start-page: 68 year: 2008 end-page: 77 ident: bib0033 article-title: Systematic mapping studies in software engineering publication-title: In the proceedings of Evaluation and Assessment in Software Engineering – start-page: 153 year: 2016 end-page: 160 ident: bib0053 article-title: A map of threats to validity of systematic literature reviews in software engineering publication-title: 23rd Asia-Pacific Software Engineering Conference (APSEC) – year: 1979 ident: bib0008 article-title: Quasi-experimentation: design & analysis issues for field settings – volume: 52 start-page: 792 year: August 2010 end-page: 805 ident: bib0019 article-title: Systematic literature reviews in software engineering – A tertiary study publication-title: Inf. Softw. Technol. – volume: 3 start-page: e3081 year: 2008 ident: bib0011 article-title: "Systematic review of the empirical evidence of study publication bias and outcome reporting bias publication-title: PLoS One – volume: 62 start-page: 279 year: 1992 end-page: 301 ident: bib0029 article-title: Understanding and validity in qualitative research publication-title: Harvard educational review – year: 2013 ident: bib0015 article-title: A tertiary study: experiences of conducting systematic literature reviews in software engineering publication-title: Proceedings of the 17th International Conference on Evaluation and Assessment in Software Engineering (EASE '13) – volume: 17 start-page: 1 year: 1999 end-page: 12 ident: bib0016 article-title: Assessing the quality of reports of randomized clinical trials: is blinding necessary? publication-title: Controlled Clin. Trials – start-page: 1 year: 2010 end-page: 9 ident: bib0021 article-title: The value of mapping studies – A participant-observer case study publication-title: Evaluation and Assessment in Software Engineering (EASE '10) – start-page: 273 year: May , 2004 end-page: 281 ident: bib0022 article-title: Evidence-based software engineering publication-title: Proceedings of the 26th International Conference on Software Engineering (ICSE '04) – year: 2009 ident: bib0051 article-title: Case Study research: Design and Methods – reference: C. Wohlin, M. Host, P. Runeson, M. Ohlsson, B. Regnell, and A. Wesslen, “Experimentation in software engineering: an introduction”, Kluwer Academic Publishers, 2000 – volume: 81 start-page: 1051 year: June 2008 end-page: 1058 ident: bib0006 article-title: An analysis of research topics in software engineering – 2006 publication-title: J. Syst. Softw. – start-page: 163 year: 2007 end-page: 164 ident: bib0044 article-title: Empirical methods in software engineering research publication-title: ICSE Companion – start-page: 81 year: Oct. 2013 end-page: 89 ident: bib0035 article-title: Worldviews, research methods, and their relationship to validity in empirical software engineering research publication-title: , 2013 – volume: 51 start-page: 7 year: January 2009 end-page: 15 ident: bib0020 article-title: Systematic literature reviews in software engineering – a systematic literature review publication-title: Inf. Softw. Technol. – year: 2007 ident: bib0014 article-title: Introduction to Action research: Social Research For Social Change – start-page: 9 year: 2015 end-page: 19 ident: bib0040 article-title: Views on internal and external validity in empirical software engineering publication-title: 37th IEEE International Conference on Software Engineering (ICSE) – year: 2014 ident: bib0049 article-title: Guidelines for snowballing in systematic literature studies and a replication in software engineering publication-title: Proceedings of the 18th International Conference on Evaluation and Assessment in Software Engineering (EASE '14) – year: August 2012 ident: bib0028 article-title: Systematic literature reviews in distributed software development: a tertiary study publication-title: Global Software Engineering (ICGSE '12) – volume: 51 start-page: 1235 year: 1998 end-page: 1241 ident: bib0045 article-title: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus publication-title: J. Clin. Epidemiol. – year: 2009 ident: bib0030 article-title: Taxonomy development in information systems: developing a taxonomy of mobile applications publication-title: European Conference in Information Systems – start-page: 61 year: 2008 end-page: 81 ident: bib0037 article-title: Publication bias as a threat to the validity of meta-analytic results publication-title: J. Exp. Criminol. – volume: 53 start-page: 440 year: May 2011 end-page: 455 ident: bib0009 article-title: Research synthesis in software engineering: a tertiary study publication-title: Inf. Softw. Technol. – volume: 40 start-page: 282 year: 2014 end-page: 306 ident: bib0013 article-title: Variability in software systems - A systematic literature review publication-title: IEEE Trans. Softw. Eng. – volume: 95 start-page: 62 year: 2018 end-page: 74 ident: bib0005 article-title: Reporting systematic reviews: some lessons from a tertiary study – year: 2015 ident: 10.1016/j.infsof.2018.10.006_bib0025 – year: 2012 ident: 10.1016/j.infsof.2018.10.006_bib0028 article-title: Systematic literature reviews in distributed software development: a tertiary study – year: 2009 ident: 10.1016/j.infsof.2018.10.006_bib0051 – volume: 52 start-page: 377 issue: 6 year: 1998 ident: 10.1016/j.infsof.2018.10.006_bib0010 article-title: The feasibility of creating a checklist for the assessment of the methodological quality both of randomised and non-randomised studies of health care interventions publication-title: Journal of Epidemiology & Community Health doi: 10.1136/jech.52.6.377 – volume: 18 start-page: 44 issue: 1 year: 2000 ident: 10.1016/j.infsof.2018.10.006_bib0052 article-title: Data collection instrument and procedure for systematic reviews in the Guide to Community Preventive Services publication-title: Am. J. Prevent. Med. doi: 10.1016/S0749-3797(99)00122-1 – start-page: 1 year: 2010 ident: 10.1016/j.infsof.2018.10.006_bib0021 article-title: The value of mapping studies – A participant-observer case study – volume: 35 start-page: 201 issue: 2 year: 2012 ident: 10.1016/j.infsof.2018.10.006_bib0043 article-title: "Evaluating the validity of systematic reviews to indentify empirically supported treatments publication-title: Educ. Treat. Child. doi: 10.1353/etc.2012.0009 – volume: 53 start-page: 440 issue: 5 year: 2011 ident: 10.1016/j.infsof.2018.10.006_bib0009 article-title: Research synthesis in software engineering: a tertiary study publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2011.01.004 – year: 2014 ident: 10.1016/j.infsof.2018.10.006_bib0049 article-title: Guidelines for snowballing in systematic literature studies and a replication in software engineering – volume: 55 start-page: 2049 issue: 12 year: 2013 ident: 10.1016/j.infsof.2018.10.006_bib0024 article-title: A systematic review of systematic review process research in software engineering publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2013.07.010 – volume: 64 start-page: 1 year: 2015 ident: 10.1016/j.infsof.2018.10.006_bib0034 article-title: Guidelines for conducting systematic mapping studies in software engineering: an update publication-title: Information and Software Technology doi: 10.1016/j.infsof.2015.03.007 – year: 2013 ident: 10.1016/j.infsof.2018.10.006_bib0015 article-title: A tertiary study: experiences of conducting systematic literature reviews in software engineering – ident: 10.1016/j.infsof.2018.10.006_bib0047 doi: 10.1007/978-1-4615-4625-2 – year: 1979 ident: 10.1016/j.infsof.2018.10.006_bib0008 article-title: Quasi-experimentation: design & analysis issues for field settings publication-title: Boston – year: 2012 ident: 10.1016/j.infsof.2018.10.006_bib0046 article-title: Systematic literature reviews in global software development: a tertiary study – volume: 17 start-page: 1 issue: 1 year: 1999 ident: 10.1016/j.infsof.2018.10.006_bib0016 article-title: Assessing the quality of reports of randomized clinical trials: is blinding necessary? publication-title: Controlled Clin. Trials doi: 10.1016/0197-2456(95)00134-4 – volume: 7 start-page: 10 issue: 1 year: 2007 ident: 10.1016/j.infsof.2018.10.006_bib0039 article-title: Development of AMSTAR: a measurement tool to assess the methodological quality of systematic reviews publication-title: BMC Med. Res. Methodol. doi: 10.1186/1471-2288-7-10 – volume: 40 start-page: 282 issue: 3 year: 2014 ident: 10.1016/j.infsof.2018.10.006_bib0013 article-title: Variability in software systems - A systematic literature review publication-title: IEEE Trans. Softw. Eng. doi: 10.1109/TSE.2013.56 – volume: 95 start-page: 62 year: 2018 ident: 10.1016/j.infsof.2018.10.006_bib0005 article-title: Reporting systematic reviews: some lessons from a tertiary study publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2017.10.017 – volume: 10 start-page: 311 issue: 3 year: 2005 ident: 10.1016/j.infsof.2018.10.006_bib0026 article-title: Studying software engineers: data collection techniques for software field studies publication-title: Empirical Softw. Eng. doi: 10.1007/s10664-005-1290-x – volume: 51 start-page: 1235 issue: 12 year: 1998 ident: 10.1016/j.infsof.2018.10.006_bib0045 article-title: The Delphi list: a criteria list for quality assessment of randomized clinical trials for conducting systematic reviews developed by Delphi consensus publication-title: J. Clin. Epidemiol. doi: 10.1016/S0895-4356(98)00131-0 – volume: 81 start-page: 1051 issue: 6 year: 2008 ident: 10.1016/j.infsof.2018.10.006_bib0006 article-title: An analysis of research topics in software engineering – 2006 publication-title: J. Syst. Softw. doi: 10.1016/j.jss.2007.08.028 – year: 2009 ident: 10.1016/j.infsof.2018.10.006_bib0030 article-title: Taxonomy development in information systems: developing a taxonomy of mobile applications publication-title: European Conference in Information Systems – year: 2014 ident: 10.1016/j.infsof.2018.10.006_bib0004 article-title: Towards a semantic knowledge base on threats to validity and control actions in controlled experiments – volume: 3 start-page: e3081 issue: 8 year: 2008 ident: 10.1016/j.infsof.2018.10.006_bib0011 article-title: "Systematic review of the empirical evidence of study publication bias and outcome reporting bias publication-title: PLoS One doi: 10.1371/journal.pone.0003081 – start-page: 163 year: 2007 ident: 10.1016/j.infsof.2018.10.006_bib0044 article-title: Empirical methods in software engineering research publication-title: ICSE Companion – start-page: 374 year: 2010 ident: 10.1016/j.infsof.2018.10.006_bib0012 article-title: Validity Threats in Empirical Software Engineering Research - An Initial Survey – start-page: 273 year: 2004 ident: 10.1016/j.infsof.2018.10.006_bib0022 article-title: Evidence-based software engineering – volume: 62 start-page: 279 issue: 3 year: 1992 ident: 10.1016/j.infsof.2018.10.006_bib0029 article-title: Understanding and validity in qualitative research publication-title: Harvard educational review doi: 10.17763/haer.62.3.8323320856251826 – year: 2001 ident: 10.1016/j.infsof.2018.10.006_bib0036 article-title: Principles of survey research: part 1: turning lemons into lemonade – start-page: 153 year: 2016 ident: 10.1016/j.infsof.2018.10.006_bib0053 article-title: A map of threats to validity of systematic literature reviews in software engineering – volume: 53 start-page: 899 issue: 9 year: 2011 ident: 10.1016/j.infsof.2018.10.006_bib0041 article-title: Six years of systematic literature reviews in software engineering: an updated tertiary study publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2011.04.004 – volume: 54 issue: 1 year: 2015 ident: 10.1016/j.infsof.2018.10.006_bib0031 article-title: Preferred reporting items for systematic review and meta-analysis protocols (PRISMA-P) 2015 statement publication-title: Systematic Reviews doi: 10.1186/2046-4053-4-1 – start-page: 345 year: 2000 ident: 10.1016/j.infsof.2018.10.006_bib0032 article-title: Empirical studies of software engineering: a roadmap – start-page: 358 year: 2007 ident: 10.1016/j.infsof.2018.10.006_bib0042 article-title: The future of empirical methods in software engineering research – volume: 41 start-page: 283 issue: 4 year: 2017 ident: 10.1016/j.infsof.2018.10.006_bib0001 article-title: External validity: the next step for systematic reviews? publication-title: Eval. Rev. doi: 10.1177/0193841X16665199 – start-page: 9 year: 2015 ident: 10.1016/j.infsof.2018.10.006_bib0040 article-title: Views on internal and external validity in empirical software engineering – volume: 86 start-page: 2594 issue: 10 year: 2013 ident: 10.1016/j.infsof.2018.10.006_bib0048 article-title: On the reliability of mapping studies in software engineering publication-title: J. Syst. Softw. doi: 10.1016/j.jss.2013.04.076 – start-page: 10 year: 2005 ident: 10.1016/j.infsof.2018.10.006_bib0017 article-title: Reporting guidelines for controlled experiments in software engineering – start-page: 81 year: 2013 ident: 10.1016/j.infsof.2018.10.006_bib0035 article-title: Worldviews, research methods, and their relationship to validity in empirical software engineering research – volume: 51 start-page: 7 issue: 1 year: 2009 ident: 10.1016/j.infsof.2018.10.006_bib0020 article-title: Systematic literature reviews in software engineering – a systematic literature review publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2008.09.009 – volume: 6 start-page: 1 issue: 2 year: 2002 ident: 10.1016/j.infsof.2018.10.006_bib0007 article-title: The Database of Abstracts of Reviews of Effects (DARE) publication-title: Effect. Matters – volume: 14 start-page: 131 issue: 2 year: 2009 ident: 10.1016/j.infsof.2018.10.006_bib0038 article-title: Guidelines for conducting and reporting case study research in software engineering publication-title: Emp. Softw. Eng. doi: 10.1007/s10664-008-9102-8 – volume: 84 start-page: 162 issue: 1 year: 2011 ident: 10.1016/j.infsof.2018.10.006_bib0050 article-title: An assessment of systems and software engineering scholars and institutions (2003–2007 and 2004–2008) publication-title: J. Syst. Softw. doi: 10.1016/j.jss.2010.09.036 – start-page: 9 year: 2014 ident: 10.1016/j.infsof.2018.10.006_bib0002 article-title: Systematic reviews in requirements engineering: a tertiary study”, Empirical Requirements Engineering (EmpiRE) – volume: 52 start-page: 792 issue: 8 year: 2010 ident: 10.1016/j.infsof.2018.10.006_bib0019 article-title: Systematic literature reviews in software engineering – A tertiary study publication-title: Inf. Softw. Technol. doi: 10.1016/j.infsof.2010.03.006 – volume: 8 start-page: 68 year: 2008 ident: 10.1016/j.infsof.2018.10.006_bib0033 article-title: Systematic mapping studies in software engineering – year: 2007 ident: 10.1016/j.infsof.2018.10.006_bib0014 – volume: 32 start-page: 171 issue: 1–2 year: 1991 ident: 10.1016/j.infsof.2018.10.006_bib0003 article-title: Paradigms for experimentation and empirical studies in software engineering publication-title: Reliab. Eng. Syst. Saf. doi: 10.1016/0951-8320(91)90053-A – year: 1985 ident: 10.1016/j.infsof.2018.10.006_bib0027 – start-page: 61 year: 2008 ident: 10.1016/j.infsof.2018.10.006_bib0037 article-title: Publication bias as a threat to the validity of meta-analytic results publication-title: J. Exp. Criminol. doi: 10.1007/s11292-007-9046-9 – year: 2007 ident: 10.1016/j.infsof.2018.10.006_bib0018 – ident: 10.1016/j.infsof.2018.10.006_bib0023 doi: 10.1109/ESEM.2009.5314238 |
| SSID | ssj0017030 |
| Score | 2.6348767 |
| SecondaryResourceType | review_article |
| Snippet | Secondary studies are vulnerable to threats to validity. Although, mitigating these threats is crucial for the credibility of these studies, we currently lack... |
| SourceID | crossref elsevier |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 201 |
| SubjectTerms | Empirical software engineering Literature Review Secondary studies Threats to Validity |
| Title | Identifying, categorizing and mitigating threats to validity in software engineering secondary studies |
| URI | https://dx.doi.org/10.1016/j.infsof.2018.10.006 |
| Volume | 106 |
| WOSCitedRecordID | wos000454381300012&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVESC databaseName: Elsevier SD Freedom Collection Journals 2021 customDbUrl: eissn: 1873-6025 dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0017030 issn: 0950-5849 databaseCode: AIEXJ dateStart: 19950101 isFulltext: true titleUrlDefault: https://www.sciencedirect.com providerName: Elsevier |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwtV3NT9swFLcqQNMuEzCmsQHygVsXVOKkcY4VAgEHxAGm3iLHsat0XVK1oUP8Tfsj915sp4FOfBy4RJUbvzh5P_t9-L1nQg6PmeQgh7QXKi68gB8HXqpg4vVDEQWZz9NMyfqwiejqig-H8XWn89flwiwmUVHw-_t4-q6shjZgNqbOvoHdDVFogN_AdLgC2-H6Ksab1Ns6fQm_H0Y8jcpZ_uCyEX_npqxGnSaFKmNd4wFGkGeokedFdw5L8x-MCFPLYoXdOVrOGcbYzVuhh2MXCN8kQXaNJ95SqFY89wNYf6qHcjQp7-p1aYqVPSZly1uf5ib8TCDBRmoMFiMYiOl0jScnuj9-AjCU-mUTj_Jx0XZkYO5UExTiPJI9D_Sh-NHi3Hu0vNr7jaT2zY7OihAw_ogxWi7wuhi-x49MBN9S6LmN_ieysIlQdMFv48RQSZAKNCV1ffd1PwpjEAPrg4vT4WWza4Wrp6ntaF7EpWrW8YSro_m_KtRSb242ySdrl9CBwdMW6ahim3xwaRGfiW7B6gdtg4oCw-kSVNSCilYldaCieUEdJGgLVLQBFbWg2iG3Z6c3J-eePaPDk6DqVx4LWQo6JRO8L2Fm92SfwxTXLBZRqMF4lyIQWLMwUL6WcZj1UoHZ2ppnmZZgjLAvZK0oC_WV0NTP-iHTERc-mNEaSGIxyhBr3LEYxNIuYe5zJdIWsMdzVCbJc8zaJV7Ta2oKuLxwf-Q4kVgl1CiXCcDr2Z7f3vik7-TjchrskbVqdqf2yYZcVPl8dmCx9Q-V0bC_ |
| linkProvider | Elsevier |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Identifying%2C+categorizing+and+mitigating+threats+to+validity+in+software+engineering+secondary+studies&rft.jtitle=Information+and+software+technology&rft.au=Ampatzoglou%2C+Apostolos&rft.au=Bibi%2C+Stamatia&rft.au=Avgeriou%2C+Paris&rft.au=Verbeek%2C+Marijn&rft.date=2019-02-01&rft.issn=0950-5849&rft.volume=106&rft.spage=201&rft.epage=230&rft_id=info:doi/10.1016%2Fj.infsof.2018.10.006&rft.externalDBID=n%2Fa&rft.externalDocID=10_1016_j_infsof_2018_10_006 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=0950-5849&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=0950-5849&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=0950-5849&client=summon |