Zobraziť v EDS

Data Extractions Using a Large Language Model (Elicit) and Human Reviewers in Randomized Controlled Trials: A Systematic Comparison.

Uložené v:
Podrobná bibliografia
Názov: Data Extractions Using a Large Language Model (Elicit) and Human Reviewers in Randomized Controlled Trials: A Systematic Comparison.
Autori: Bianchi, Joleen, Hirt, Julian, Vogt, Magdalena, Vetsch, Janine
Zdroj: Cochrane Evidence Synthesis & Methods; Jul2025, Vol. 3 Issue 4, p1-6, 6p
Predmety: LANGUAGE models, ARTIFICIAL intelligence, DATA extraction, RANDOMIZED controlled trials, SAMPLE size (Statistics)
Abstrakt: Aim: We aimed at comparing data extractions from randomized controlled trials by using Elicit and human reviewers. Background: Elicit is an artificial intelligence tool which may automate specific steps in conducting systematic reviews. However, the tool's performance and accuracy have not been independently assessed. Methods: For comparison, we sampled 20 randomized controlled trials of which data were extracted manually from a human reviewer. We assessed the variables study objectives, sample characteristics and size, study design, interventions, outcome measured, and intervention effects and classified the results into "more," "equal to," "partially equal," and "deviating" extractions. STROBE checklist was used to report the study. Results: We analysed 20 randomized controlled trials from 11 countries. The studies covered diverse healthcare topics. Across all seven variables, Elicit extracted "more" data in 29.3% of cases, "equal" in 20.7%, "partially equal" in 45.7%, and "deviating" in 4.3%. Elicit provided "more" information for the variable study design (100%) and sample characteristics (45%). In contrast, for more nuanced variables, such as "intervention effects," Elicit's extractions were less detailed, with 95% rated as "partially equal." Conclusions: Elicit was capable of extracting data partly correct for our predefined variables. Variables like "intervention effect" or "intervention" may require a human reviewer to complete the data extraction. Our results suggest that verification by human reviewers is necessary to ensure that all relevant information is captured completely and correctly by Elicit. Implications: Systematic reviews are labor‐intensive. Data extraction process may be facilitated by artificial intelligence tools. Use of Elicit may require a human reviewer to double‐check the extracted data. [ABSTRACT FROM AUTHOR]
Copyright of Cochrane Evidence Synthesis & Methods is the property of Wiley-Blackwell and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Databáza: Complementary Index
Popis
Abstrakt:Aim: We aimed at comparing data extractions from randomized controlled trials by using Elicit and human reviewers. Background: Elicit is an artificial intelligence tool which may automate specific steps in conducting systematic reviews. However, the tool's performance and accuracy have not been independently assessed. Methods: For comparison, we sampled 20 randomized controlled trials of which data were extracted manually from a human reviewer. We assessed the variables study objectives, sample characteristics and size, study design, interventions, outcome measured, and intervention effects and classified the results into "more," "equal to," "partially equal," and "deviating" extractions. STROBE checklist was used to report the study. Results: We analysed 20 randomized controlled trials from 11 countries. The studies covered diverse healthcare topics. Across all seven variables, Elicit extracted "more" data in 29.3% of cases, "equal" in 20.7%, "partially equal" in 45.7%, and "deviating" in 4.3%. Elicit provided "more" information for the variable study design (100%) and sample characteristics (45%). In contrast, for more nuanced variables, such as "intervention effects," Elicit's extractions were less detailed, with 95% rated as "partially equal." Conclusions: Elicit was capable of extracting data partly correct for our predefined variables. Variables like "intervention effect" or "intervention" may require a human reviewer to complete the data extraction. Our results suggest that verification by human reviewers is necessary to ensure that all relevant information is captured completely and correctly by Elicit. Implications: Systematic reviews are labor‐intensive. Data extraction process may be facilitated by artificial intelligence tools. Use of Elicit may require a human reviewer to double‐check the extracted data. [ABSTRACT FROM AUTHOR]
ISSN:28329023
DOI:10.1002/cesm.70033