Reference Standards, Judges, and Comparison Subjects: Roles for Experts in Evaluating System Performance

Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even whe...

Celý popis

Uložené v:
Podrobná bibliografia
Vydané v:Journal of the American Medical Informatics Association : JAMIA Ročník 9; číslo 1; s. 1 - 15
Hlavní autori: Hripcsak, G., Wilcox, A.
Médium: Journal Article
Jazyk:English
Vydavateľské údaje: England American Medical Informatics Association 01.01.2002
Predmet:
Evaluation Studies as Topic
Information Systems - standards
Methods
Reference Standards
Reproducibility of Results
The Practice of Informatics
ISSN:1067-5027, 1527-974X
On-line prístup:Získať plný text
Tagy: Pridať tag
Žiadne tagy, Buďte prvý, kto otaguje tento záznam!
Popis
Shrnutí:Medical informatics systems are often designed to perform at the level of human experts. Evaluation of the performance of these systems is often constrained by lack of reference standards, either because the appropriate response is not known or because no simple appropriate response exists. Even when performance can be assessed, it is not always clear whether the performance is sufficient or reasonable. These challenges can be addressed if an evaluator enlists the help of clinical domain experts. 1) The experts can carry out the same tasks as the system, and then their responses can be combined to generate a reference standard. 2)The experts can judge the appropriateness of system output directly. 3) The experts can serve as comparison subjects with which the system can be compared. These are separate roles that have different implications for study design, metrics, and issues of reliability and validity. Diagrams help delineate the roles of experts in complex study designs.
Bibliografia:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 23
This work was supported by grants R01 LM06910 and R01 LM06274 from the National Library of Medicine and grant NY01-002153889 from Pfizer, Inc.
ISSN:1067-5027
1527-974X
DOI:10.1136/jamia.2002.0090001