External validation of new risk prediction models is infrequent and reveals worse prognostic discrimination

To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations. We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discrimi...

Full description

Saved in:
Bibliographic Details
Published in:Journal of clinical epidemiology Vol. 68; no. 1; pp. 25 - 34
Main Authors: Siontis, George C.M., Tzoulaki, Ioanna, Castaldi, Peter J., Ioannidis, John P.A.
Format: Journal Article
Language:English
Published: United States Elsevier Inc 01.01.2015
Elsevier Limited
Subjects:
Area Under Curve
Area under the receiver operating characteristics curve
Biomarkers
Clinical medicine
Derivation study
Discrimination
Disease
Epidemiology
External validation
Forecasting
Humans
Internal Medicine
Models, Statistical
Mortality
Prediction models
Probability
Prognosis
Prognostic models
Reproducibility of Results
Risk
Risk prediction model
Validation studies
Risk prediction model
Area under the receiver operating characteristics curve
Derivation study
Discrimination
External validation
Prognostic models
ISSN:0895-4356, 1878-5921, 1878-5921
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To evaluate how often newly developed risk prediction models undergo external validation and how well they perform in such validations. We reviewed derivation studies of newly proposed risk models and their subsequent external validations. Study characteristics, outcome(s), and models' discriminatory performance [area under the curve, (AUC)] in derivation and validation studies were extracted. We estimated the probability of having a validation, change in discriminatory performance with more stringent external validation by overlapping or different authors compared to the derivation estimates. We evaluated 127 new prediction models. Of those, for 32 models (25%), at least an external validation study was identified; in 22 models (17%), the validation had been done by entirely different authors. The probability of having an external validation by different authors within 5 years was 16%. AUC estimates significantly decreased during external validation vs. the derivation study [median AUC change: −0.05 (P < 0.001) overall; −0.04 (P = 0.009) for validation by overlapping authors; −0.05 (P < 0.001) for validation by different authors]. On external validation, AUC decreased by at least 0.03 in 19 models and never increased by at least 0.03 (P < 0.001). External independent validation of predictive models in different studies is uncommon. Predictive performance may worsen substantially on external validation.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:0895-4356
1878-5921
1878-5921
DOI:10.1016/j.jclinepi.2014.09.007