Diagnostic Accuracy and Failure Mode Analysis of a Deep Learning Algorithm for the Detection of Intracranial Hemorrhage

To determine the institutional diagnostic accuracy of an artificial intelligence (AI) decision support systems (DSS), Aidoc, in diagnosing intracranial hemorrhage (ICH) on noncontrast head CTs and to assess the potential generalizability of an AI DSS. This retrospective study included 3,605 consecut...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Journal of the American College of Radiology Ročník 18; číslo 8; s. 1143
Hlavní autori: Voter, Andrew F, Meram, Ece, Garrett, John W, Yu, John-Paul J
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: United States 01.08.2021
Predmet:
generalizability
noncontrast head CT
intracranial hemorrhage
Artificial intelligence
decision support systems
ISSN:1558-349X, 1558-349X
On-line prístup:Zistit podrobnosti o prístupe
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:To determine the institutional diagnostic accuracy of an artificial intelligence (AI) decision support systems (DSS), Aidoc, in diagnosing intracranial hemorrhage (ICH) on noncontrast head CTs and to assess the potential generalizability of an AI DSS. This retrospective study included 3,605 consecutive, emergent, adult noncontrast head CT scans performed between July 1, 2019, and December 30, 2019, at our institution (51% female subjects, mean age of 61 ± 21 years). Each scan was evaluated for ICH by both a certificate of added qualification certified neuroradiologist and Aidoc. We determined the diagnostic accuracy of the AI model and performed a failure mode analysis with quantitative CT radiomic image characterization. Of the 3,605 scans, 349 cases of ICH (9.7% of studies) were identified. The neuroradiologist and Aidoc interpretations were concordant in 96.9% of cases and the overall sensitivity, specificity, positive predictive value, and negative predictive value were 92.3%, 97.7%, 81.3%, and 99.2%, respectively, with positive predictive values unexpectedly lower than in previously reported studies. Prior neurosurgery, type of ICH, and number of ICHs were significantly associated with decreased model performance. Quantitative image characterization with CT radiomics failed to reveal significant differences between concordant and discordant studies. This study revealed decreased diagnostic accuracy of an AI DSS at our institution. Despite extensive evaluation, we were unable to identify the source of this discrepancy, raising concerns about the generalizability of these tools with indeterminate failure modes. These results further highlight the need for standardized study design to allow for rigorous and reproducible site-to-site comparison of emerging deep learning technologies.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1558-349X
1558-349X
DOI:10.1016/j.jacr.2021.03.005