Understanding the evidence for artificial intelligence in healthcare

The entire evaluation sequence—technical performance, usability and workflow, and impact—should also be repeated whenever conditions change, especially if the models may learn and change their performance over time.7 Model performance can vary for a wide variety of reasons such as changes in the und...

Celý popis

Uloženo v:
Podrobná bibliografie
Vydáno v:BMJ quality & safety Ročník 34; číslo 7; s. 421
Hlavní autoři: Jackson, Gretchen Purcell, Shortliffe, Edward H
Médium: Journal Article
Jazyk:angličtina
Vydáno: England BMJ Publishing Group LTD 01.07.2025
Témata:
Algorithms
Artificial intelligence
Clinical outcomes
Health care
Health care delivery
Infectious diseases
Informatics
Medical errors
Ophthalmology
R&D
Research & development
Usability
Performance measures
Decision support, computerized
Information technology
Evaluation methodology
Patient Safety
ISSN:2044-5415, 2044-5423, 2044-5423
On-line přístup:Získat plný text
Tagy: Přidat tag
Žádné tagy, Buďte první, kdo vytvoří štítek k tomuto záznamu!
Abstract The entire evaluation sequence—technical performance, usability and workflow, and impact—should also be repeated whenever conditions change, especially if the models may learn and change their performance over time.7 Model performance can vary for a wide variety of reasons such as changes in the underlying data used for prediction or behavioural changes from use of the model itself. In general, most AI algorithms either predict or classify, so their performance is measured in a manner similar to the evaluation of diagnostic tests, using metrics such as sensitivity, specificity and area under the curve.12 13 Healthcare providers should pay particular attention to rates of false positives and false negatives, as well as their consequences, as clinical judgement is often needed in selecting performance thresholds. Studies of healthcare AI tools should explicitly report false positive and false negative rates rather than composite measures such as F1 scores (an evaluation metric that combines precision and recall) so that medical practitioners can determine their suitability for practice. The phases of clinical research used for drugs and devices are useful for framing AI evaluation, but there are important nuances, as articulated by Park and colleagues.16 AI is similar to drugs and devices, as early-phase ‘laboratory’ studies are needed to demonstrate proof-of-concept technical performance, usability of prototypes, and potential for impact.
AbstractList The entire evaluation sequence—technical performance, usability and workflow, and impact—should also be repeated whenever conditions change, especially if the models may learn and change their performance over time.7 Model performance can vary for a wide variety of reasons such as changes in the underlying data used for prediction or behavioural changes from use of the model itself. In general, most AI algorithms either predict or classify, so their performance is measured in a manner similar to the evaluation of diagnostic tests, using metrics such as sensitivity, specificity and area under the curve.12 13 Healthcare providers should pay particular attention to rates of false positives and false negatives, as well as their consequences, as clinical judgement is often needed in selecting performance thresholds. Studies of healthcare AI tools should explicitly report false positive and false negative rates rather than composite measures such as F1 scores (an evaluation metric that combines precision and recall) so that medical practitioners can determine their suitability for practice. The phases of clinical research used for drugs and devices are useful for framing AI evaluation, but there are important nuances, as articulated by Park and colleagues.16 AI is similar to drugs and devices, as early-phase ‘laboratory’ studies are needed to demonstrate proof-of-concept technical performance, usability of prototypes, and potential for impact.
Author Shortliffe, Edward H
Jackson, Gretchen Purcell
Author_xml – sequence: 1
  givenname: Gretchen Purcell
  orcidid: 0000-0002-3242-8058
  surname: Jackson
  fullname: Jackson, Gretchen Purcell
  email: gretchenpurcell@stanfordalumni.org
  organization: Pediatric Surgery, Pediatrics, and Biomedical Informatics, Vanderbilt University Medical Center, Nashville, Tennessee, USA
– sequence: 2
  givenname: Edward H
  surname: Shortliffe
  fullname: Shortliffe, Edward H
  organization: Population Health Sciences, Weill Cornell Medical College, New York, New York, USA
BackLink https://www.ncbi.nlm.nih.gov/pubmed/40246317$$D View this record in MEDLINE/PubMed
BookMark eNpdkEtPwzAQhC1UREvpD-CCInHhEvAjTuwjKk-pEhd6jhx707hKnNZ2kPj3hFI4sJddaT7tjOYcTVzvAKFLgm8JYfld1W33IaWY8hQTwbk8QTOKsyzlGWWTv5vwKVqEsMXjMCklZmdommGa5YwUM_SwdgZ8iMoZ6zZJbCCBD2vAaUjq3ifKR1tbbVWbWBehbe3moFmXNKDa2Gjl4QKd1qoNsDjuOVo_Pb4vX9LV2_Pr8n6VVkzmMTU8Z1Qb4LSmXGEoKokxlwyUEUITLkwuaKGxUloUo06Y4lJLY-pKClJlbI5ufv7ufL8fIMSys0GPoZSDfgglI5KMTqKgI3r9D932g3djugP1bUv4SF0dqaHqwJQ7bzvlP8vfftgXv7hqJA
CitedBy_id crossref_primary_10_1016_j_ijmedinf_2025_106103
ContentType Journal Article
Copyright 2025 Author(s) (or their employer(s)) 2025. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ Group. http://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ . Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
Copyright_xml – notice: 2025 Author(s) (or their employer(s)) 2025. Re-use permitted under CC BY-NC. No commercial re-use. See rights and permissions. Published by BMJ Group. http://creativecommons.org/licenses/by-nc/4.0/ This is an open access article distributed in accordance with the Creative Commons Attribution Non Commercial (CC BY-NC 4.0) license, which permits others to distribute, remix, adapt, build upon this work non-commercially, and license their derivative works on different terms, provided the original work is properly cited, appropriate credit is given, any changes made indicated, and the use is non-commercial. See: http://creativecommons.org/licenses/by-nc/4.0/ . Notwithstanding the ProQuest Terms and Conditions, you may use this content in accordance with the terms of the License.
DBID NPM
3V.
7RV
7X7
7XB
88E
8FI
8FJ
8FK
ABUWG
AEUYN
AFKRA
AN0
BENPR
BTHHO
CCPQU
FYUFA
GHDGH
K9.
KB0
M0S
M1P
NAPCQ
PHGZM
PHGZT
PJZUB
PKEHL
PPXIY
PQEST
PQQKQ
PQUKI
PRINS
7X8
DOI 10.1136/bmjqs-2025-018559
DatabaseName PubMed
ProQuest Central (Corporate)
Nursing & Allied Health Database
Health & Medical Collection
ProQuest Central (purchase pre-March 2016)
Medical Database (Alumni Edition)
Hospital Premium Collection
Hospital Premium Collection (Alumni Edition)
ProQuest Central (Alumni) (purchase pre-March 2016)
ProQuest Central (Alumni)
ProQuest One Sustainability
ProQuest Central UK/Ireland
British Nursing Database
ProQuest Central
BMJ Journals
ProQuest One
Health Research Premium Collection
Health Research Premium Collection (Alumni)
ProQuest Health & Medical Complete (Alumni)
Nursing & Allied Health Database (Alumni Edition)
ProQuest Health & Medical Collection
Medical Database
Nursing & Allied Health Premium
ProQuest Central Premium
ProQuest One Academic (New)
ProQuest Health & Medical Research Collection
ProQuest One Academic Middle East (New)
ProQuest One Health & Nursing
ProQuest One Academic Eastern Edition (DO NOT USE)
ProQuest One Academic (retired)
ProQuest One Academic UKI Edition
ProQuest Central China
MEDLINE - Academic
DatabaseTitle PubMed
ProQuest One Academic Middle East (New)
ProQuest Health & Medical Complete (Alumni)
ProQuest Central (Alumni Edition)
ProQuest One Community College
ProQuest One Health & Nursing
ProQuest Central China
ProQuest Central
ProQuest One Sustainability
ProQuest Health & Medical Research Collection
Health Research Premium Collection
Health and Medicine Complete (Alumni Edition)
Health & Medical Research Collection
ProQuest Central (New)
ProQuest Medical Library (Alumni)
ProQuest One Academic Eastern Edition
British Nursing Index with Full Text
ProQuest Nursing & Allied Health Source
ProQuest Hospital Collection
Health Research Premium Collection (Alumni)
ProQuest Hospital Collection (Alumni)
Nursing & Allied Health Premium
ProQuest Health & Medical Complete
ProQuest Medical Library
ProQuest One Academic UKI Edition
BMJ Journals
ProQuest Nursing & Allied Health Source (Alumni)
ProQuest One Academic
ProQuest One Academic (New)
ProQuest Central (Alumni)
MEDLINE - Academic
DatabaseTitleList ProQuest One Academic Middle East (New)
Database_xml – sequence: 1
  dbid: NPM
  name: PubMed
  url: http://www.ncbi.nlm.nih.gov/entrez/query.fcgi?db=PubMed
  sourceTypes: Index Database
– sequence: 2
  dbid: 7RV
  name: Nursing & Allied Health Database
  url: https://search.proquest.com/nahs
  sourceTypes: Aggregation Database
DeliveryMethod fulltext_linktorsrc
EISSN 2044-5423
ExternalDocumentID 40246317
Genre Editorial
Commentary
GroupedDBID ---
.VT
0R~
53G
5VS
7RV
7~S
AAHLL
AAOJX
AAUVZ
AAWJN
AAWTL
ABAAH
ABJNI
ABKDF
ABMQD
ABTFR
ABVAJ
ACGFS
ACGTL
ACHQT
ACHTP
ACMFJ
ACOFX
ACQHZ
ACTZY
ADBBV
ADCEG
ADUGQ
AEKJL
AENEX
AERUA
AFWFF
AHMBA
AHNKE
AHQMW
AJYBZ
AKKEP
ALIPV
ALMA_UNASSIGNED_HOLDINGS
BENPR
BLJBA
BOMFT
BTHHO
C45
CXRWF
EBS
EX3
HAJ
HZ~
NAPCQ
NPM
O9-
OVD
RHI
RMJ
RV8
TEORI
UYXKK
V24
WOW
ZY1
3V.
7X7
7XB
88E
8FI
8FJ
8FK
ABUWG
AEUYN
AFKRA
AN0
BPHCQ
BVXVI
CCPQU
FYUFA
HMCUK
K9.
M1P
PHGZM
PHGZT
PJZUB
PKEHL
PPXIY
PQEST
PQQKQ
PQUKI
PRINS
PROAC
PSQYO
UKHRP
7X8
ID FETCH-LOGICAL-b396t-d5632cde52f25a0e7b900593ead88c158d6827c0aac87a0e13a59c9ddfb981b43
IEDL.DBID 7RV
ISICitedReferencesCount 3
ISICitedReferencesURI http://www.webofscience.com/api/gateway?GWVersion=2&SrcApp=Summon&SrcAuth=ProQuest&DestLinkType=CitingArticles&DestApp=WOS_CPL&KeyUT=001524054100006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
ISSN 2044-5415
2044-5423
IngestDate Fri Sep 05 17:33:29 EDT 2025
Tue Oct 07 06:57:10 EDT 2025
Thu Jul 10 06:16:31 EDT 2025
IsDoiOpenAccess false
IsOpenAccess true
IsPeerReviewed true
IsScholarly true
Issue 7
Keywords Performance measures
Decision support, computerized
Information technology
Evaluation methodology
Patient Safety
Language English
LinkModel DirectLink
MergedId FETCHMERGED-LOGICAL-b396t-d5632cde52f25a0e7b900593ead88c158d6827c0aac87a0e13a59c9ddfb981b43
Notes SourceType-Scholarly Journals-1
content type line 14
ObjectType-Editorial-2
ObjectType-Commentary-1
content type line 23
ORCID 0000-0002-3242-8058
OpenAccessLink https://qualitysafety.bmj.com/content/34/7/421.full
PMID 40246317
PQID 3191059315
PQPubID 2041042
ParticipantIDs proquest_miscellaneous_3191396872
proquest_journals_3191059315
pubmed_primary_40246317
PublicationCentury 2000
PublicationDate 2025-07-01
PublicationDateYYYYMMDD 2025-07-01
PublicationDate_xml – month: 07
  year: 2025
  text: 2025-07-01
  day: 01
PublicationDecade 2020
PublicationPlace England
PublicationPlace_xml – name: England
– name: London
PublicationTitle BMJ quality & safety
PublicationTitleAlternate BMJ Qual Saf
PublicationYear 2025
Publisher BMJ Publishing Group LTD
Publisher_xml – name: BMJ Publishing Group LTD
SSID ssj0000399903
Score 2.4723063
Snippet The entire evaluation sequence—technical performance, usability and workflow, and impact—should also be repeated whenever conditions change, especially if the...
SourceID proquest
pubmed
SourceType Aggregation Database
Index Database
StartPage 421
SubjectTerms Algorithms
Artificial intelligence
Clinical outcomes
Health care
Health care delivery
Infectious diseases
Informatics
Medical errors
Ophthalmology
R&D
Research & development
Usability
Title Understanding the evidence for artificial intelligence in healthcare
URI https://www.ncbi.nlm.nih.gov/pubmed/40246317
https://www.proquest.com/docview/3191059315
https://www.proquest.com/docview/3191396872
Volume 34
WOSCitedRecordID wos001524054100006&url=https%3A%2F%2Fcvtisr.summon.serialssolutions.com%2F%23%21%2Fsearch%3Fho%3Df%26include.ft.matches%3Dt%26l%3Dnull%26q%3D
hasFullText 1
inHoldings 1
isFullTextHit
isPrint
link http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMwpV3JTsMwELWg5cCFRWyFUhmJq9VsTuwTYmnFAaKqoqi3yFtEkUiXtHw_48SUnnrhYslyFFnjF8-zZzIPoVtuYl-qUBHOBLMHFEOYnwdEakNlzLzcU1Wd2ZckTdl4zAfuwq10aZW_e2K1UeupsnfkXYCKpQKhT-9mc2JVo2x01Ulo7KKmb7kx4DkZvq_vWDzwvrwSRw68KCIUnJULbPph3JVfn_MSUBLY7DVG6RaSWTmb_uF_p3mEDhzNxPc1Lo7RjilO0NNo808WDNQPGycqioG7YouiuqAEnmxU6oQO_liniZ2iUb_39vhMnIoCkSGPl0TTOAwUmD7IAyo8k0he6fgBhBhTPmU6ZkGiPCEUS2DcDwXlimudSw6cNgrPUKOYFuYCYeGHBg5oVGshwJaeVFrxKBJAIS2zyVuo_WuUzH0KZfZnkRa6WQ8DiG1kQhRmuqqfgcmyJGih89rw2ayutpEBfqIYWM7l9pdfof16KW0mbRs1louVuUZ76ns5KRedChe2HSdVyzqo-dBLB0PopYPXH_dLxa4
linkProvider ProQuest
linkToHtml http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V3JTsMwEB2xSXBhEVtZjQTHqIkTJ_YBIURBrVoqDiD1FhzbEUWiWwqIn-IbGSdp6YlbDxwjR9bIs73xjGcAzoUJvUT5yhFcchugGId7KXUSbVgScjd1Vd5nthW127zTEQ8L8D15C2PLKic2MTfUuq_sHXkVRcVCAd9jV4OhY6dG2ezqZIRGIRZN8_WJIVt22aghfy8ovbt9vKk75VQBJ_FFOHY0C32qkBSaUiZdEyUin2uHR8q58hjXIaeRcqVUPMJ1z5dMKKF1mgjEeIGP-y7CMtrxyAZ7USea3um46O1FPoyZukHgMHSOZSLV88Nq8vY6zFAqqa2W44z9AWpz53a38d-OZRPWSxhNrgu534IF09uG2tPsSx2C0JaYcmgqQWxOrJYUDTNId6YTKX6Ql2kZ3A48zYXwXVjq9XtmH4j0fIMBKNNaSuSdmyitRBBIhMgWuaUVOJowIS5VPYt_OVCBs-kyKqnNvMie6b8X_yCxPKIV2CsYHQ-KbiIx6kcQIoo7-HvzU1itP9634laj3TyEtUKMbNXwESyNR-_mGFbUx7ibjU5ymSTwPG9u_wDAyB7r
linkToPdf http://cvtisr.summon.serialssolutions.com/2.0.0/link/0/eLvHCXMw1V1LSwMxEB60injxga9q1Qh6XLqb3ewmBxG1FkUpRSx4W7NJFivYdxX_mr_Oye62euqtB49LljDkm8eXZDIDcCpM6CXKV47gktsNinG4l1In0YYlIXdTV2V1Zh-iRoM_P4vmAnxP3sLYtMqJT8wcte4qe0ZeRVWxVMD3WDUt0iKatfpFr-_YDlL2pnXSTiNXkXvz9Ynbt-H5XQ2xPqO0fvN0fesUHQacxBfhyNEs9KlCsWhKmXRNlIisxx0uL-fKY1yHnEbKlVLxCMc9XzKhhNZpIpDvBT7OuwhLEZKMoARLVzeN5uP0hMfF2C-y1szUDQKHYagsrlU9P6wm72_9IeootblznLEZFDcLdfX1_7xIG7BWEGxymVvEJiyYzhbUWn_f8BAkvcQU7VQJsnZi7ScvpUHaf2qU4gd5nSbIbUNrLoLvQKnT7Zg9INLzDW5NmdZSIo5uorQSQSCRPFtOl5ahMgEkLpzAMP5Fowwn02E0X3snIzumO87_QWF5RMuwm4Me9_I6IzFaThAiv9ufPfkxrCDI8cNd4_4AVnONsunEFSiNBmNzCMvqY9QeDo4KBSXwMm-4fwBeHykL
openUrl ctx_ver=Z39.88-2004&ctx_enc=info%3Aofi%2Fenc%3AUTF-8&rfr_id=info%3Asid%2Fsummon.serialssolutions.com&rft_val_fmt=info%3Aofi%2Ffmt%3Akev%3Amtx%3Ajournal&rft.genre=article&rft.atitle=Understanding+the+evidence+for+artificial+intelligence+in+healthcare&rft.jtitle=BMJ+quality+%26+safety&rft.au=Gretchen+Purcell+Jackson&rft.au=Shortliffe%2C+Edward+H&rft.date=2025-07-01&rft.pub=BMJ+Publishing+Group+LTD&rft.issn=2044-5415&rft.eissn=2044-5423&rft.spage=bmjqs-2025-018559&rft_id=info:doi/10.1136%2Fbmjqs-2025-018559&rft.externalDBID=HAS_PDF_LINK
thumbnail_l http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/lc.gif&issn=2044-5415&client=summon
thumbnail_m http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/mc.gif&issn=2044-5415&client=summon
thumbnail_s http://covers-cdn.summon.serialssolutions.com/index.aspx?isbn=/sc.gif&issn=2044-5415&client=summon