RT Journal Article
SR Electronic
T1 Evaluation of crowdsourced mortality prediction models as a framework for assessing AI in medicine
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2021.01.18.21250072
DO 10.1101/2021.01.18.21250072
A1 Bergquist, Timothy
A1 Schaffter, Thomas
A1 Yan, Yao
A1 Yu, Thomas
A1 Prosser, Justin
A1 Gao, Jifan
A1 Chen, Guanhua
A1 Charzewski, Łukasz
A1 Nawalany, Zofia
A1 Brugere, Ivan
A1 Retkute, Renata
A1 Prusokas, Alidivinas
A1 Prusokas, Augustinas
A1 Choi, Yonghwa
A1 Lee, Sanghoon
A1 Choe, Junseok
A1 Lee, Inggeol
A1 Kim, Sunkyu
A1 Kang, Jaewoo
A1 Patient Mortality Prediction DREAM Challenge Consortium
A1 Mooney, Sean D.
A1 Guinney, Justin
YR 2021
UL http://medrxiv.org/content/early/2021/01/20/2021.01.18.21250072.abstract
AB Applications of machine learning in healthcare are of high interest and have the potential to significantly improve patient care. Yet, the real-world accuracy and performance of these models on different patient subpopulations remains unclear. To address these important questions, we hosted a community challenge to evaluate different methods that predict healthcare outcomes. To overcome patient privacy concerns, we employed a Model-to-Data approach, allowing citizen scientists and researchers to train and evaluate machine learning models on private health data without direct access to that data. We focused on the prediction of all-cause mortality as the community challenge question. In total, we had 345 registered participants, coalescing into 25 independent teams, spread over 3 continents and 10 countries. The top performing team achieved a final area under the receiver operator curve of 0.947 (95% CI 0.942, 0.951) and an area under the precision-recall curve of 0.487 (95% CI 0.458, 0.499) on patients prospectively collected over a one year observation of a large health system. Post-hoc analysis after the challenge revealed that models differ in accuracy on subpopulations, delineated by race or gender, even when they are trained on the same data and have similar accuracy on the population. This is the largest community challenge focused on the evaluation of state-of-the-art machine learning methods in a healthcare system performed to date, revealing both opportunities and pitfalls of clinical AI.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis work was supported by the Clinical and Translational Science Awards Program National Center for Data to Health funding by the National Center for Advancing Translational Sciences at the National Institutes of Health (grant numbers U24TR002306 and UL1 TR002319). Any opinions expressed in this document are those of the Center for Data to Health community and the Institute for Translational Health Sciences and do not necessarily reflect the views of the National Center for Advancing Translational Sciences, team members, or affiliated organizations and institutions. TB, YY, SM, JG, TY, TS, and JP were supported by grant number U24TR002306. TB, JP, and SM were supported by grant number UL1 TR002319.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:We received an institutional review board (IRB) nonhuman subjects research designation from the University of Washington Human Subjects Research Division to construct a dataset derived from all patient records from the University of Washington Enterprise Data Warehouse.All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.YesDue to privacy concerns, the clinical data used in this study can not be made available.