View in EDS

Criteria and Protocol: Assessing Generative AI Efficacy in Perceiving EULAR 2019 Lupus Classification.

Saved in:
Bibliographic Details
Title: Criteria and Protocol: Assessing Generative AI Efficacy in Perceiving EULAR 2019 Lupus Classification.
Authors: Lushington, Gerald H., Nair, Sandeep, Jupe, Eldon R., Rubin, Bernard, Purushothaman, Mohan
Source: Diagnostics (2075-4418); Sep2025, Vol. 15 Issue 18, p2409, 17p
Subject Terms: SYSTEMIC lupus erythematosus, GENERATIVE artificial intelligence, MEDICAL records, AUTOIMMUNE diseases, MEDICAL informatics, AUTOMATIC summarization
Abstract: Background/Objectives: In clinical informatics, the term 'information overload' is increasingly used to describe the operational impediments of excessive documentation. While electronic health records (EHRs) are growing in abundance, many medical records (MRs) remain in legacy formats that impede efficient, systematic processing, contributing to the extenuating challenges of care fragmentation. Thus, there is a growing interest in using generative AI (genAI) for automated MR summarization and characterization. Methods: MRs for a set of 78 individuals were digitized. Some were known systemic lupus erythematosus (SLE) cases, while others were under evaluation for possible SLE classification. A two-pass genAI assessment strategy was implemented using the Claude 3.5 large language model (LLM) to mine MRs for information relevant to classifying SLE vs. undifferentiated connective tissue disorder (UCTD) vs. neither via the 22-criteria EULAR 2019 model. Results: Compared to clinical determination, the antinuclear antibody (ANA) criterion (whose results are crucial for classifying SLE-negative cases) exhibited favorable sensitivity 0.78 ± 0.09 (95% confidence interval) and a positive predictive value 0.85 ± 0.08 but a marginal performance for specificity 0.60 ± 0.11 and uncertain predictivity for the negative predictive value 0.48 ± 0.11. Averaged over the remaining 21 criteria, these four performance metrics were 0.69 ± 0.11, 0.87 ± 0.04, 0.54 ± 0.10, and 0.93 ± 0.03. Conclusions: ANA performance statistics imply that genAI yields confident assessments of SLE negativity (per high sensitivity) but weaker positivity. The remaining genAI criterial determinations support (per specificity) confident assertions of SLE-positivity but tend to misclassify a significant fraction of clinical positives as UCTD. [ABSTRACT FROM AUTHOR]
Copyright of Diagnostics (2075-4418) is the property of MDPI and its content may not be copied or emailed to multiple sites without the copyright holder's express written permission. Additionally, content may not be used with any artificial intelligence tools or machine learning technologies. However, users may print, download, or email articles for individual use. This abstract may be abridged. No warranty is given about the accuracy of the copy. Users should refer to the original published version of the material for the full abstract. (Copyright applies to all Abstracts.)
Database: Biomedical Index
Description
Abstract:Background/Objectives: In clinical informatics, the term 'information overload' is increasingly used to describe the operational impediments of excessive documentation. While electronic health records (EHRs) are growing in abundance, many medical records (MRs) remain in legacy formats that impede efficient, systematic processing, contributing to the extenuating challenges of care fragmentation. Thus, there is a growing interest in using generative AI (genAI) for automated MR summarization and characterization. Methods: MRs for a set of 78 individuals were digitized. Some were known systemic lupus erythematosus (SLE) cases, while others were under evaluation for possible SLE classification. A two-pass genAI assessment strategy was implemented using the Claude 3.5 large language model (LLM) to mine MRs for information relevant to classifying SLE vs. undifferentiated connective tissue disorder (UCTD) vs. neither via the 22-criteria EULAR 2019 model. Results: Compared to clinical determination, the antinuclear antibody (ANA) criterion (whose results are crucial for classifying SLE-negative cases) exhibited favorable sensitivity 0.78 ± 0.09 (95% confidence interval) and a positive predictive value 0.85 ± 0.08 but a marginal performance for specificity 0.60 ± 0.11 and uncertain predictivity for the negative predictive value 0.48 ± 0.11. Averaged over the remaining 21 criteria, these four performance metrics were 0.69 ± 0.11, 0.87 ± 0.04, 0.54 ± 0.10, and 0.93 ± 0.03. Conclusions: ANA performance statistics imply that genAI yields confident assessments of SLE negativity (per high sensitivity) but weaker positivity. The remaining genAI criterial determinations support (per specificity) confident assertions of SLE-positivity but tend to misclassify a significant fraction of clinical positives as UCTD. [ABSTRACT FROM AUTHOR]
ISSN:20754418
DOI:10.3390/diagnostics15182409