Abstract
Background Liver transplantation (LT) is a treatment for acute-on-chronic liver failure (ACLF) but up to 40% mortality post-LT has been reported. Existing post-LT models in ACLF have been limited by small samples. In this study, we developed a novel Expert-Augmented Machine Learning (EAML) model to predict post-LT outcomes.
Methods We identified ACLF patients in the University of California Health Data Warehouse (UCHDW). We used EAML, which uses the RuleFit machine learning (ML) algorithm to extract rules from decision-trees that are then evaluated by human experts, to predict post-LT outcomes. We compared EAML/RuleFit’s performances versus other popular models.
Results We identified 1,384 ACLF patients. For death at one-year: areas-under-the-receiver-operating characteristic curve (AUROCs) were 0.707 (Confidence Interval [CI] 0.625-0.793) for EAML and 0.719 (CI 0.640-0.800) for RuleFit. For death at 90-days: AUROCs were 0.678 (CI 0.581-0.776) for EAML and 0.707 (CI 0.615-0.800) for RuleFit. In pairwise comparisons, EAML/RuleFit models outperformed cross-sectional models. Divergences between experts and ML in rankings revealed biases and artifacts in the underlying data.
Conclusions EAML/RuleFit outperformed cross-sectional models. Significant discrepancies between experts and ML occurred in rankings of biomarkers used in clinical practice. EAML may serve as a method for ML-guided hypothesis generation in further ACLF research.
Competing Interest Statement
Dr. Jin Ge receives research support from Merck and Co.
Funding Statement
The authors of this study were supported in part by the KL2TR001870 (National Center for Advancing Translational Sciences, Ge and McCulloch), AASLD Anna S. Lok Advanced/Transplant Hepatology Award AHL21-104606 (AASLD Foundation, Ge), American Society of Transplantation LICOP Grant Award CA-0182782 (American Society of Transplantation, Ge), P30DK026743 (UCSF Liver Center Grant, Ge and Lai), F31HL156498 (National Heart, Lung, and Blood Institute, Digitale), UL1TR001872 (National Center for Advancing Translational Sciences, Pletcher), and R01AG059183/K24AG080021 (National Institute on Aging, Lai). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other funding agencies. The funding agencies played no role in the analysis of the data or the preparation of this manuscript.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The Institutional Review Board at the University of California, San Francisco gave ethical approval for this work under #20-32717 for model generation and #22-37555 for expert input.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Financial/Grant Support: The authors of this study were supported in part by the KL2TR001870 (National Center for Advancing Translational Sciences, Ge and McCulloch), AASLD Anna S. Lok Advanced/Transplant Hepatology Award AHL21-104606 (AASLD Foundation, Ge), American Society of Transplantation LICOP Grant Award CA-0182782 (American Society of Transplantation, Ge), P30DK026743 (UCSF Liver Center Grant, Ge and Lai), F31HL156498 (National Heart, Lung, and Blood Institute, Digitale), UL1TR001872 (National Center for Advancing Translational Sciences, Pletcher), and R01AG059183/K24AG080021 (National Institute on Aging, Lai). The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health or any other funding agencies. The funding agencies played no role in the analysis of the data or the preparation of this manuscript.
Disclosures: Dr. Jin Ge receives research support from Merck and Co.
Writing Assistance: None.
Data Availability
The authors thank the Center for Data-driven Insights and Innovation (CDI2) at the University of California Health (UCH) for its analytical and technical support related to the use of the UC Health Data Warehouse (UCHDW) and related data assets, including the UC COVID Research Data Set (CORDS).
Abbreviations
- ACLF
- acute-on-chronic liver failure;
- APASL ACLF
- Asian Pacific Association for the Study of the Liver ACLF Research Consortium;
- AUROC
- area-under-the-receiver-operating characteristic curve;
- CDI2
- Center for Data-Driven Insights and Innovation;
- CDS
- clinical decision support;
- CI
- confidence interval;
- CORDS
- UC COVID Research Data Set;
- CPT4
- Current Procedural Terminology version 4;
- EAML
- Expert-Augmented Machine Learning;
- EF-CLIF
- European Association for the Study of the Liver-Chronic Liver Failure Consortium;
- EHR
- electronic health record;
- ESLD
- end-stage liver disease;
- FiO2
- fraction of inspired oxygen;
- GBM
- Gradient Boosting Machine;
- GLMNET
- Elastic-Net Regularized Generalized Linear Model;
- ICD-10-CM
- International Classification of Diseases, Tenth Revision, Clinical Modification;
- LASSO
- Least Absolute Shrinkage and Selection Operator;
- LT
- liver transplantation;
- ML
- machine learning;
- N3C
- National COVID Cohort Collaborative;
- NACSELD
- North American Consortium for the Study of End-Stage Liver Disease;
- OMOP
- Observational Medical Outcomes Partnership;
- PaO2
- arterial partial pressure of oxygen;
- RF
- Random Forest;
- SNOMED
- Standard Nomenclature of Medicine;
- SpO2
- partial oxygen saturation;
- TAM
- Transplantation for ACLF-3 Model;
- UCH
- University of California Health;
- UCHDW
- University of California Health Data Warehouse;
- VHACDW
- Veterans Health Administration Corporate Data Warehouse
