Internal-external cross-validation helped to evaluate the generalizability of prediction models in large clustered datasets

To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predi...

Full description

Saved in:
Bibliographic Details
Published in:JOURNAL OF CLINICAL EPIDEMIOLOGY Vol. 137; pp. 83 - 91
Main Authors: Takada, Toshihiko, Nijman, Steven, Denaxas, Spiros, Snell, Kym I.E., Uijl, Alicia, Nguyen, Tri-Long, Asselbergs, Folkert W., Debray, Thomas P.A.
Format: Journal Article Publication
Language:English
Published: United States Elsevier Inc 01.09.2021
Elsevier Limited
Subjects:
Body mass index
Calibration
Cluster Analysis
Congestive heart failure
Datasets
Datasets as Topic - statistics & numerical data
Discrimination
Electronic health records
Epidemiology
Ethnicity
Forecasting
Heterogeneity
Humans
Internal Medicine
Maximum likelihood estimation
Model comparison
Models, Statistical
Population
Prediction model
Prediction models
Primary care
Regression analysis
Regression models
Statistical analysis
Validation
Variables
Validation
Heterogeneity
Model comparison
Discrimination
Calibration
Prediction model
ISSN:0895-4356, 1878-5921, 1878-5921
Online Access:Get full text
Tags: Add Tag
No Tags, Be the first to tag this record!
Description
Summary:To illustrate how to evaluate the need of complex strategies for developing generalizable prediction models in large clustered datasets. We developed eight Cox regression models to estimate the risk of heart failure using a large population-level dataset. These models differed in the number of predictors, the functional form of the predictor effects (non-linear effects and interaction) and the estimation method (maximum likelihood and penalization). Internal-external cross-validation was used to evaluate the models’ generalizability across the included general practices. Among 871,687 individuals from 225 general practices, 43,987 (5.5%) developed heart failure during a median follow-up time of 5.8 years. For discrimination, the simplest prediction model yielded a good concordance statistic, which was not much improved by adopting complex strategies. Between-practice heterogeneity in discrimination was similar in all models. For calibration, the simplest model performed satisfactorily. Although accounting for non-linear effects and interaction slightly improved the calibration slope, it also led to more heterogeneity in the observed/expected ratio. Similar results were found in a second case study involving patients with stroke. In large clustered datasets, prediction model studies may adopt internal-external cross-validation to evaluate the generalizability of competing models, and to identify promising modelling strategies.
Bibliography:ObjectType-Article-1
SourceType-Scholarly Journals-1
ObjectType-Feature-2
content type line 14
ObjectType-Article-2
ObjectType-Feature-1
content type line 23
ObjectType-Undefined-3
ISSN:0895-4356
1878-5921
1878-5921
DOI:10.1016/j.jclinepi.2021.03.025