Stability of clinical prediction models developed using statistical or machine learning methods

Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model‐building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We rai...

Ausführliche Beschreibung

Gespeichert in:
Bibliographische Detailangaben
Veröffentlicht in:Biometrical journal Jg. 65; H. 8; S. e2200302 - n/a
Hauptverfasser: Riley, Richard D., Collins, Gary S.
Format: Journal Article
Sprache:Englisch
Veröffentlicht: Germany Wiley - VCH Verlag GmbH & Co. KGaA 01.12.2023
John Wiley and Sons Inc
Schlagworte:
Bias
Calibration
Case studies
Computer Simulation
Datasets
fairness
Humans
Instability
Learning algorithms
Machine Learning
Mathematical models
Models, Statistical
prediction model
Prediction models
Prognosis
Regression and Prediction Modeling
Stability
Stability analysis
Statistical analysis
uncertainty
fairness
prediction model
uncertainty
calibration
stability
ISSN:0323-3847, 1521-4036, 1521-4036
Online-Zugang:Volltext
Tags: Tag hinzufügen
Keine Tags, Fügen Sie den ersten Tag hinzu!
Beschreibung
Zusammenfassung:Clinical prediction models estimate an individual's risk of a particular health outcome. A developed model is a consequence of the development dataset and model‐building strategy, including the sample size, number of predictors, and analysis method (e.g., regression or machine learning). We raise the concern that many models are developed using small datasets that lead to instability in the model and its predictions (estimated risks). We define four levels of model stability in estimated risks moving from the overall mean to the individual level. Through simulation and case studies of statistical and machine learning approaches, we show instability in a model's estimated risks is often considerable, and ultimately manifests itself as miscalibration of predictions in new data. Therefore, we recommend researchers always examine instability at the model development stage and propose instability plots and measures to do so. This entails repeating the model‐building steps (those used to develop the original prediction model) in each of multiple (e.g., 1000) bootstrap samples, to produce multiple bootstrap models, and deriving (i) a prediction instability plot of bootstrap model versus original model predictions; (ii) the mean absolute prediction error (mean absolute difference between individuals’ original and bootstrap model predictions), and (iii) calibration, classification, and decision curve instability plots of bootstrap models applied in the original sample. A case study illustrates how these instability assessments help reassure (or not) whether model predictions are likely to be reliable (or not), while informing a model's critical appraisal (risk of bias rating), fairness, and further validation requirements.
Bibliographie:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
content type line 23
ISSN:0323-3847
1521-4036
1521-4036
DOI:10.1002/bimj.202200302