The Relationship between Intercoder Reliability of Data Extraction and Effect Measure Calculation in Single-Case Meta-Analysis
Meta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from linear graphs in primary studies, prior to calculating and aggregating single-case effect measures. Existing research suggests data extraction yie...
Gespeichert in:
| Veröffentlicht in: | School psychology review Jg. 54; H. 3; S. 291 - 298 |
|---|---|
| Hauptverfasser: | , , |
| Format: | Journal Article |
| Sprache: | Englisch |
| Veröffentlicht: |
Routledge
04.05.2025
|
| Schlagworte: | |
| ISSN: | 2372-966X, 2372-966X |
| Online-Zugang: | Volltext |
| Tags: |
Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
|
| Abstract | Meta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from linear graphs in primary studies, prior to calculating and aggregating single-case effect measures. Existing research suggests data extraction yields reliable and valid data; however, we have an incomplete understanding of the downstream effects of relying on data extracted by two or more people. This study was undertaken to enhance that understanding in the context of SCEDs published in school psychology journals. Data for 91 unique outcomes across 67 cases in 20 SCEDs were extracted by two data extractors. Four different single-case effect measures were calculated using data extracted by each data extractor and then compared to determine the similarity of the effect measures. Overall, intercoder reliability metrics suggested a high degree of agreement, and there were minimal differences in single-case effect measures calculated from data extracted by different researchers. Intercoder reliability metrics and differences in single-case effect measures were generally negatively related, though the strength varied depending on the single-case effect measure. Hence, it is unlikely that the small differences in effect measure estimates due to the slight unreliability of the data extraction process would have a considerable impact on the interpretation of single-case effect measures.
Impact Statement
Two people extracted highly similar numerical data from the same linear graphs using plot digitizing software. Differences in calculations across data extracted by two people were trivial. Results suggest researchers can likely have confidence in the calculation of effect measures aggregated in meta-analyses of single-case experimental designs, provided they achieve comparable levels of agreement amongst data extractors. |
|---|---|
| AbstractList | Meta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from linear graphs in primary studies, prior to calculating and aggregating single-case effect measures. Existing research suggests data extraction yields reliable and valid data; however, we have an incomplete understanding of the downstream effects of relying on data extracted by two or more people. This study was undertaken to enhance that understanding in the context of SCEDs published in school psychology journals. Data for 91 unique outcomes across 67 cases in 20 SCEDs were extracted by two data extractors. Four different single-case effect measures were calculated using data extracted by each data extractor and then compared to determine the similarity of the effect measures. Overall, intercoder reliability metrics suggested a high degree of agreement, and there were minimal differences in single-case effect measures calculated from data extracted by different researchers. Intercoder reliability metrics and differences in single-case effect measures were generally negatively related, though the strength varied depending on the single-case effect measure. Hence, it is unlikely that the small differences in effect measure estimates due to the slight unreliability of the data extraction process would have a considerable impact on the interpretation of single-case effect measures.
Impact Statement
Two people extracted highly similar numerical data from the same linear graphs using plot digitizing software. Differences in calculations across data extracted by two people were trivial. Results suggest researchers can likely have confidence in the calculation of effect measures aggregated in meta-analyses of single-case experimental designs, provided they achieve comparable levels of agreement amongst data extractors. |
| Author | Koval, Elizabeth T. Drevon, Daniel D. Peart, Allison M. |
| Author_xml | – sequence: 1 givenname: Daniel D. surname: Drevon fullname: Drevon, Daniel D. organization: Central Michigan University – sequence: 2 givenname: Allison M. surname: Peart fullname: Peart, Allison M. organization: Central Michigan University – sequence: 3 givenname: Elizabeth T. surname: Koval fullname: Koval, Elizabeth T. organization: Central Michigan University |
| BookMark | eNqFkEFLAzEQhYNUsK3-BCF_YGs22Xa3eLHUqoWKoBW8LbPJxEbSbElSai_-dru2gnjQ0wzz5nvwXoe0XO2QkPOU9VJWsAsucj4cDF56nHHR4zwXBedHpN3ck0Zo_dhPSCeEN8aYEBlvk4_5AukjWoimdmFhVrTCuEF0dOoielkr9I1uoDLWxC2tNb2GCHTyHj3IhqLgFJ1ojTLSe4Sw9kjHYOV6b0qNo0_GvVpMxhBw9xIhGTmw22DCKTnWYAOeHWaXPN9M5uO7ZPZwOx2PZokUKY-JEKqfcZCQ5kpAwYpMc82LLM8ESt1XfUSWs2GuhQQ14ExWHNOqUgVqpSrNRJf0977S1yF41OXKmyX4bZmysimx_C6xbEosDyXuuMtfnDTxK9YuvbH_0ld72jhd-yVsam9VGWFra689OGlCKf62-ATlEJB0 |
| CitedBy_id | crossref_primary_10_1080_2372966X_2025_2522589 crossref_primary_10_1080_2372966X_2025_2497709 |
| Cites_doi | 10.4324/9781315517698-15 10.1177/0145445516673998 10.1016/j.beth.2010.08.006 10.1037/met0000019 10.3758/BRM.41.1.177 10.1002/pits.22012 10.1016/j.chb.2015.09.008 10.1177/1098300715573630 10.1002/pits.22359 10.1016/j.jsp.2020.11.002 10.1016/j.beth.2008.10.006 10.1177/0145445515616105 10.1080/17489539.2021.1886412 10.1016/j.chb.2012.07.031 10.1177/0741932512452794 10.1080/02796015.2014.12087427 10.1016/j.ridd.2018.02.001 10.1080/00273171.2014.973989 10.1177/0145445516645763 10.1037/spq0000305 10.1007/s40614-021-00284-0 |
| ContentType | Journal Article |
| Copyright | 2023 National Association of School Psychologists 2023 |
| Copyright_xml | – notice: 2023 National Association of School Psychologists 2023 |
| DBID | AAYXX CITATION |
| DOI | 10.1080/2372966X.2023.2273822 |
| DatabaseName | CrossRef |
| DatabaseTitle | CrossRef |
| DatabaseTitleList | |
| DeliveryMethod | fulltext_linktorsrc |
| Discipline | Education Psychology |
| EISSN | 2372-966X |
| EndPage | 298 |
| ExternalDocumentID | 10_1080_2372966X_2023_2273822 2273822 |
| Genre | Research Article |
| GrantInformation_xml | – fundername: Faculty Research and Creative Endeavors committee at Central Michigan University |
| GroupedDBID | --Z -~X 0-V 0BK 0R~ 8A4 8R4 8R5 AAGDL AAHIA AAHSB AAMFJ AAMIU AAPUL AAYJJ AAZJI AAZMC ABIVO ABLIJ ABOPQ ABPPZ ABXUL ABXYU ACHQT ACIHN ACNCT ADAHI ADBBV ADKVQ AEAQA AEFOU AEISY AEZRU AFKRA AFRVT AGDLA AGRBW AHDZW AIKWM AIYEW AKBVH ALMA_UNASSIGNED_HOLDINGS ALQZU ALSLI AQTUD ARALO AZQEC BENPR BMOTO BOHLJ BPHCQ BVXVI CJNVE DGFLZ EBS EJD EMG F5P FEDTE FYUFA H13 HF~ HVGLF IAO ICJ ICO IEA IER INH IOF IPC IPO IPY ITC L7B M0P M2M M4Z P2P PQQKQ PROAC PZZ Q2X RNANH ROSJB RSYQP RWL RXW S0X SJN TAE TASJS TBQAZ TDBHL TED TFH TFL TFW TN5 TNTFI TUROJ TZEDD UKHRP WH7 AAYXX CITATION |
| ID | FETCH-LOGICAL-c312t-33d542aca17d3a8084f2f284743ecf5d5ee07097f3cad620cb2e1bbd8efddbf03 |
| IEDL.DBID | TFW |
| ISICitedReferencesCount | 2 |
| ISICitedReferencesURI | http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001088399800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| ISSN | 2372-966X |
| IngestDate | Sat Nov 29 07:51:34 EST 2025 Tue Nov 18 22:14:23 EST 2025 Mon Oct 20 23:45:13 EDT 2025 |
| IsPeerReviewed | true |
| IsScholarly | true |
| Issue | 3 |
| Language | English |
| LinkModel | DirectLink |
| MergedId | FETCHMERGED-LOGICAL-c312t-33d542aca17d3a8084f2f284743ecf5d5ee07097f3cad620cb2e1bbd8efddbf03 |
| PageCount | 8 |
| ParticipantIDs | informaworld_taylorfrancis_310_1080_2372966X_2023_2273822 crossref_primary_10_1080_2372966X_2023_2273822 crossref_citationtrail_10_1080_2372966X_2023_2273822 |
| PublicationCentury | 2000 |
| PublicationDate | 2025-05-04 |
| PublicationDateYYYYMMDD | 2025-05-04 |
| PublicationDate_xml | – month: 05 year: 2025 text: 2025-05-04 day: 04 |
| PublicationDecade | 2020 |
| PublicationTitle | School psychology review |
| PublicationYear | 2025 |
| Publisher | Routledge |
| Publisher_xml | – name: Routledge |
| References | e_1_3_2_27_1 e_1_3_2_20_1 e_1_3_2_21_1 e_1_3_2_22_1 e_1_3_2_23_1 e_1_3_2_24_1 e_1_3_2_25_1 e_1_3_2_26_1 Card N. A. (e_1_3_2_5_1) 2015 e_1_3_2_16_1 e_1_3_2_9_1 e_1_3_2_17_1 e_1_3_2_8_1 e_1_3_2_18_1 e_1_3_2_7_1 e_1_3_2_19_1 e_1_3_2_2_1 e_1_3_2_10_1 e_1_3_2_11_1 e_1_3_2_6_1 e_1_3_2_12_1 e_1_3_2_13_1 e_1_3_2_4_1 e_1_3_2_14_1 e_1_3_2_3_1 e_1_3_2_15_1 |
| References_xml | – ident: e_1_3_2_19_1 doi: 10.4324/9781315517698-15 – ident: e_1_3_2_6_1 doi: 10.1177/0145445516673998 – ident: e_1_3_2_17_1 doi: 10.1016/j.beth.2010.08.006 – ident: e_1_3_2_18_1 doi: 10.1037/met0000019 – ident: e_1_3_2_24_1 doi: 10.3758/BRM.41.1.177 – ident: e_1_3_2_26_1 doi: 10.1002/pits.22012 – ident: e_1_3_2_10_1 – ident: e_1_3_2_22_1 doi: 10.1016/j.chb.2015.09.008 – ident: e_1_3_2_13_1 doi: 10.1177/1098300715573630 – ident: e_1_3_2_21_1 doi: 10.1002/pits.22359 – volume-title: Applied meta-analysis for social science research year: 2015 ident: e_1_3_2_5_1 – ident: e_1_3_2_14_1 doi: 10.1016/j.jsp.2020.11.002 – ident: e_1_3_2_16_1 doi: 10.1016/j.beth.2008.10.006 – ident: e_1_3_2_8_1 doi: 10.1177/0145445515616105 – ident: e_1_3_2_25_1 doi: 10.1080/17489539.2021.1886412 – ident: e_1_3_2_4_1 doi: 10.1016/j.chb.2012.07.031 – ident: e_1_3_2_11_1 doi: 10.1177/0741932512452794 – ident: e_1_3_2_3_1 doi: 10.1080/02796015.2014.12087427 – ident: e_1_3_2_23_1 – ident: e_1_3_2_20_1 – ident: e_1_3_2_27_1 doi: 10.1016/j.ridd.2018.02.001 – ident: e_1_3_2_9_1 doi: 10.1080/00273171.2014.973989 – ident: e_1_3_2_15_1 doi: 10.1177/0145445516645763 – ident: e_1_3_2_7_1 – ident: e_1_3_2_12_1 doi: 10.1037/spq0000305 – ident: e_1_3_2_2_1 doi: 10.1007/s40614-021-00284-0 |
| SSID | ssj0003342 |
| Score | 2.4323668 |
| Snippet | Meta-analyzing data from single-case experimental designs (SCEDs) usually requires data extraction, a process by which numerical values are obtained from... |
| SourceID | crossref informaworld |
| SourceType | Enrichment Source Index Database Publisher |
| StartPage | 291 |
| SubjectTerms | Jorge E. Gonzalez meta-analysis research methods single subject designs |
| Title | The Relationship between Intercoder Reliability of Data Extraction and Effect Measure Calculation in Single-Case Meta-Analysis |
| URI | https://www.tandfonline.com/doi/abs/10.1080/2372966X.2023.2273822 |
| Volume | 54 |
| WOSCitedRecordID | wos001088399800001&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D |
| hasFullText | 1 |
| inHoldings | 1 |
| isFullTextHit | |
| isPrint | |
| journalDatabaseRights | – providerCode: PRVAWR databaseName: Taylor & Francis customDbUrl: eissn: 2372-966X dateEnd: 99991231 omitProxy: false ssIdentifier: ssj0003342 issn: 2372-966X databaseCode: TFW dateStart: 19720101 isFulltext: true titleUrlDefault: https://www.tandfonline.com providerName: Taylor & Francis |
| link | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV07T8MwELZQxdCFRwFRXvLA6pLYbh4jKq0YoEKiQLfI8UNEqtIqDYgu_Hbs2InaARhgTi6y7Mvd2f7u-wC45FQGoacU0tUqRlRxH6VxGiGhpJChzyitOryf78LxOJpO4weHJlw6WKXZQytLFFHFavNzs3RZI-KusLlqCoJpz0h_97BpLsEmCuvK3vj4ZPTSxGJCKK77dr6z3MhIG3yla5lmtPsPY9wDO67MhNfWL_bBlsw7RqHZoTk6oN1EvtUB-NTOAhtY3Gu2gA6-BasDQ9P2XpjnmSX1XsG5gjesZHD4URa2MwLqkUJLhQzv7bkjHLAZd-pgMMvho86SM4kGOm_qV0qGakKUQ_A0Gk4Gt8gJMyBOfFwiQkSfYsaZHwrCIi-iCiuT5yiRXPVFX0odSeJQEc5EgD2eYumnqYikEiJVHjkCrXyey2MAPW4uHhmjMec0YrqcZFJvsjATRH_Nk11A68VJuGMtN-IZs8R35Kb1TCdmphM3013Qa8wWlrbjN4N4feWTsjovUVbcJCE_2p78wfYUtLFRFDYQSnoGWmXxJs_BNn8vs2VxUTnzF1xa9Cw |
| linkProvider | Taylor & Francis |
| linkToHtml | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV1LTwIxEG4MmsjFB2rEZw9ei7tt2cfRIAQjcBGV26bbRyQhQNbVyMXfbmcfBA7qQc-7s2na2Zl2-s33IXQlufZ8xxhid6uUcCNdEodxQJTRSvuu4Dzr8H7q-YNBMBqFq70wAKuEM7TJiSKyWA0_NxSjS0jcNYW7Js8bNUD7u0Ghu4TaMLzZtLkWYH3DzvMyGjPGadm5853pWk5aYyxdyTWd3f8Y5R7aKXaa-CZ3jX20oac1EGkuAB01VF0Gv8UB-rT-gpfIuJfxHBcILpzVDKHzPYHn45zXe4FnBt-KVOD2R5rkzRHYDhXnbMi4n5cecUtMZCEQhsdT_GAT5USTlk2d9pVUkJIT5RA9dtrDVpcU2gxEMpemhDHV5FRI4fqKicAJuKEGUh1nWpqmamptg0noGyaF8qgjY6rdOFaBNkrFxmFHqDKdTfUxwo6Eu0cheCglD4TdUQptz1lUKGa_5ug64uXqRLIgLgf9jEnkFvym5UxHMNNRMdN11FiazXPmjt8MwtWlj9KsZGJyfZOI_Wh78gfbS7TdHfZ7Ue9ucH-KqhQEhgFRyc9QJU3e9Dnaku_p-DW5yDz7C7lo-E0 |
| linkToPdf | http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwrV3NT8IwFG8MGsPFD9SInz14LW5tYdvRAItGJCSiclu6fsQlBAhMIxf_dtutW-CgHvS8vaVp3957bX_v9wPgilPZ8hylkK5WMaKKuygOYh8JJYX0XEZp1uH93PP6fX80CgYWTbiwsEqzh1Y5UUQWq83PPROqQMRdY3PV1GqNGkb6u4FNcwnWUXgzI8fSLj0MX8pgTAjFRePOd6ZrKWmNsHQl1YS7_zDIPbBj60x4kzvGPtiQk5qRaLZwjhqolqFveQA-tbfAEhf3msygxW_B7MTQ9L3PzfMkZ_VewqmCHZYy2P1I53lrBNQjhTkXMnzIDx5hm425lQeDyQQ-6jQ5lqitE6d-JWWoYEQ5BE9hd9i-RVaZAXHi4hQRIpoUM85cTxDmOz5VWJlER4nkqimaUupQEniKcCZa2OExlm4cC18qIWLlkCNQmUwn8hhAh5ubR8ZowDn1ma4nmdS7LMwE0V9zZB3QYnEibmnLjXrGOHItu2kx05GZ6cjOdB00SrNZztvxm0GwuvJRmh2YqFzdJCI_2p78wfYSbA86YdS769-fgio26sIGTknPQCWdv8lzsMXf02Qxv8j8-gupgvbx |
| openUrl | ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=The+Relationship+between+Intercoder+Reliability+of+Data+Extraction+and+Effect+Measure+Calculation+in+Single-Case+Meta-Analysis&rft.jtitle=School+psychology+review&rft.au=Drevon%2C+Daniel+D.&rft.au=Peart%2C+Allison+M.&rft.au=Koval%2C+Elizabeth+T.&rft.date=2025-05-04&rft.pub=Routledge&rft.eissn=2372-966X&rft.volume=54&rft.issue=3&rft.spage=291&rft.epage=298&rft_id=info:doi/10.1080%2F2372966X.2023.2273822&rft.externalDocID=2273822 |
| thumbnail_l | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2372-966X&client=summon |
| thumbnail_m | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2372-966X&client=summon |
| thumbnail_s | http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2372-966X&client=summon |