Evaluation of crowdsourced mortality prediction models as a framework for assessing artificial intelligence in medicine

Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community c...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Journal of the American Medical Informatics Association : JAMIA Ročník 31; číslo 1; s. 35
Hlavní autori: Bergquist, Timothy, Schaffter, Thomas, Yan, Yao, Yu, Thomas, Prosser, Justin, Gao, Jifan, Chen, Guanhua, Charzewski, Łukasz, Nawalany, Zofia, Brugere, Ivan, Retkute, Renata, Prusokas, Alidivinas, Prusokas, Augustinas, Choi, Yonghwa, Lee, Sanghoon, Choe, Junseok, Lee, Inggeol, Kim, Sunkyu, Kang, Jaewoo, Mooney, Sean D, Guinney, Justin
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: England 22.12.2023
Predmet:
Algorithms
Artificial Intelligence
Crowdsourcing
Humans
Machine Learning
Medicine
evaluation
machine learning
health informatics
ISSN:1527-974X, 1527-974X
On-line prístup:Zistit podrobnosti o prístupe
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Applications of machine learning in healthcare are of high interest and have the potential to improve patient care. Yet, the real-world accuracy of these models in clinical practice and on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate methods that predict healthcare outcomes. We focused on the prediction of all-cause mortality as the community challenge question. Using a Model-to-Data framework, 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries, generated 25 accurate models all trained on a dataset of over 1.1 million patients and evaluated on patients prospectively collected over a 1-year observation of a large health system. The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI, 0.942-0.951) and an area under the precision-recall curve of 0.487 (95% CI, 0.458-0.499) on a prospectively collected patient cohort. Post hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data. This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
ISSN:1527-974X
1527-974X
DOI:10.1093/jamia/ocad159