Abstract
Background Risk prediction models are used in healthcare settings to tailor therapies to individuals most likely to benefit. Despite appropriate external validation, difference in local characteristics (e.g. patient mix) may attenuate model performance. Prior to any implementation it is therefore advisable to explore local performance, typically requiring a modest amount of historic data. Depending on model performance, model adjustments might be necessary which often require large amounts of data. Here we explore a small sample size approach approximating de novo derivation, by combining model stacking and transfer learning, referred to as stacked transfer learning. As an example we focus on stacking previously trained risk prediction models for cardiovascular disease (CVD), stroke, (chronic) kidney disease, and diabetes.
Methods We leverage data from the UK biobank to illustrate the benefits of stacking previously trained risk prediction models, predicting the risk of incident CVD, chronic kidney disease (CKD) or diabetes. To mimic sample sizes available in local settings, such as a small to large healthcare trust, we iterated the number of training cases between 10 and 1000. Model stacking was performed using a LASSO penalized logistic regression model, and compared performance of a de novo model estimating the local association of 33 variables used in the aforementioned risk prediction models.
Results We found that stacked models require roughly one-tenths of the training sample size compared to de novo derivation of a prediction model. For example, predicting CVD the stacked model required 30 cases to reach a area under the curve (AUC) value (with 95% CI) of 0.732 (0.728, 0.735), while the de novo model required 300 cases to reach approximately the same performance. As expected, the absolute performance depended on the predicted outcome, where for example the difference between de novo and stacked modelling was smaller for CKD prediction.
Conclusion We show that our proposed ”stacked transfer learning” approach closely approximated the predictive performance of a de novo model, often requiring only a fraction of the data. As such, this approach should be considered when tailoring a model to a local setting.
Competing Interest Statement
AFS has received funding from New Amsterdam Pharma for unrelated work.
Funding Statement
Folkert Asselbergs and A. Floriaan Schmidt are supported by the UCL Hospital NIHR Biomedical Research Centre grant. A. Floriaan Schmidt is supported by British Heart Foundation (BHF) grant PG/22/10989 and the UCL BHF Research Accelerator AA/18/6/34223. This work received support from the Dutch Research Council (MyDigiTwin 628.011.213). This work was supported by the EXPANSE and EXPOSOME-NL projects. The EXPANSE project is funded by the European Union Horizon 2020 research and innovation programme under grant agreement No. 874627. The EXPOSOME-NL project is funded through the Gravitation program of the Dutch Ministry of Education, Culture, and Science and the Netherlands Organization for Scientific Research (NWO grant number 024.004.017). Jasmine Gratton was supported by the BHF studentship FS/17/70/33482
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
This research has been conducted using the UK Biobank Resource under Application Numbers 12113 and 24711.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
1. Dataset obtained from the UK Biobank. 2. All data produced for this submission as figures, and tables are available in the main manuscript and the supplementary file. 3. All code scripts used for running experiments, and producing tables and figures will shortly be made available via GitLab.
