Skip to main content
medRxiv
  • Home
  • About
  • Submit
  • ALERTS / RSS
Advanced Search

Automating Responses to Patient Portal Messages Using Generative AI

Amarpreet Kaur, Alex Budko, Katrina Liu, Eric Eaton, Bryan Steitz, Kevin B. Johnson
doi: https://doi.org/10.1101/2024.04.25.24306183
Amarpreet Kaur
1Clinical Research Coordinator, Department of Biostatistics, Epidemiology, and Informatics Perelman School of Medicine, University of Pennsylvania B202 Richards Building, 3700 Hamilton Walk University of Pennsylvania, Philadelphia, PA 19104-6116
MHS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • For correspondence: amarpreet.kaur{at}pennmedicine.upenn.edu
Alex Budko
2School of Engineering and Applied Science, University of Pennsylvania
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Katrina Liu
2School of Engineering and Applied Science, University of Pennsylvania
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Eric Eaton
2School of Engineering and Applied Science, University of Pennsylvania
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Bryan Steitz
3Vanderbilt University Medical Center, Vanderbilt University
PhD
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
Kevin B. Johnson
4Perelman School of Medicine, University of Pennsylvania
MD, MS
  • Find this author on Google Scholar
  • Find this author on PubMed
  • Search for this author on this site
  • Abstract
  • Full Text
  • Info/History
  • Metrics
  • Data/Code
  • Preview PDF
Loading

ABSTRACT

Background Patient portals serve as vital bridges between patients and providers, playing an increasing role in healthcare communication. The rising volume and complexity of these messages is exacerbating physician and nursing burnout. Recent studies have demonstrated that AI chatbots can generate message responses that are viewed favorably by healthcare professionals; however, these studies have not included the diverse range of messages typically found in patient portals. Our goal is to investigate the quality of GPT-generated message responses across the spectrum of message types within a patient portal.

Methods We used novel prompt engineering techniques to craft synthetic responses tailored to adult primary care patients. We enrolled a sample of primary care providers in a cross-sectional study to compare authentic with synthetic patient portal message responses, generated by GPT-4. The survey assessed each message’s empathy, relevance, medical accuracy, and readability on a scale from 0 to 5. Respondents were asked to identify messages that were GPT-generated vs. provider-generated. Mean scores for all metrics were computed for subsequent analysis.

Results A total of 49 health care providers participated in the survey (59% completion rate), comprising 16 physicians and 32 advanced practice providers (APPs). When presented with GPT vs. authentic message response pairs, participants correctly identified GPT-generated responses 73% of the time and correctly identified authentic responses 50% of the time. In comparison to messages generated by physicians, GPT-4 generated messages exhibited higher mean scores for empathy (3.57 vs. 3.07, p < 0.001), relevance (3.94 vs. 3.81, p = 0.08) accuracy (4.05 vs. 3.95, p= 0.12) and readability (4.5 vs. 4.13, p < 0.001).

Limitations The study is a single site, single-specialty study, limited due to the use of synthetic data.

Conclusion Our findings affirm the potential of GPT-generated patient portal message responses to achieve comparable levels of empathy, relevance, and readability to those found in typical responses according to the health care providers and indicates promising prospects for their integration in the healthcare sector. Additional studies should be done within provider workflows and with careful evaluation of patient attitudes and concerns related to the ethics as well as the quality of generated patient portal message responses in all settings.

Competing Interest Statement

The authors have declared no competing interest.

Funding Statement

This study was funded by the National Institutes of Health, specifically the National Library Of Medicine.

Author Declarations

I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.

Yes

The details of the IRB/oversight body that provided approval or exemption for the research described are given below:

The ethics committee/IRB of the University of Pennsylvania Human Research Protection Program gave ethical approval for this work under study No. 854147.

I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.

Yes

I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).

Yes

I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.

Yes

Data Availability

All data produced in the present study are available upon reasonable request to the authors All data produced in the present work are contained in the manuscript

Copyright 
The copyright holder for this preprint is the author/funder, who has granted medRxiv a license to display the preprint in perpetuity. It is made available under a CC-BY-NC-ND 4.0 International license.
Back to top
PreviousNext
Posted April 25, 2024.
Download PDF
Data/Code
Email

Thank you for your interest in spreading the word about medRxiv.

NOTE: Your email address is requested solely to identify you as the sender of this article.

Enter multiple addresses on separate lines or separate them with commas.
Automating Responses to Patient Portal Messages Using Generative AI
(Your Name) has forwarded a page to you from medRxiv
(Your Name) thought you would like to see this page from the medRxiv website.
CAPTCHA
This question is for testing whether or not you are a human visitor and to prevent automated spam submissions.
Share
Automating Responses to Patient Portal Messages Using Generative AI
Amarpreet Kaur, Alex Budko, Katrina Liu, Eric Eaton, Bryan Steitz, Kevin B. Johnson
medRxiv 2024.04.25.24306183; doi: https://doi.org/10.1101/2024.04.25.24306183
Twitter logo Facebook logo LinkedIn logo Mendeley logo
Citation Tools
Automating Responses to Patient Portal Messages Using Generative AI
Amarpreet Kaur, Alex Budko, Katrina Liu, Eric Eaton, Bryan Steitz, Kevin B. Johnson
medRxiv 2024.04.25.24306183; doi: https://doi.org/10.1101/2024.04.25.24306183

Citation Manager Formats

  • BibTeX
  • Bookends
  • EasyBib
  • EndNote (tagged)
  • EndNote 8 (xml)
  • Medlars
  • Mendeley
  • Papers
  • RefWorks Tagged
  • Ref Manager
  • RIS
  • Zotero
  • Tweet Widget
  • Facebook Like
  • Google Plus One

Subject Area

  • Health Informatics
Subject Areas
All Articles
  • Addiction Medicine (349)
  • Allergy and Immunology (668)
  • Allergy and Immunology (668)
  • Anesthesia (181)
  • Cardiovascular Medicine (2648)
  • Dentistry and Oral Medicine (316)
  • Dermatology (223)
  • Emergency Medicine (399)
  • Endocrinology (including Diabetes Mellitus and Metabolic Disease) (942)
  • Epidemiology (12228)
  • Forensic Medicine (10)
  • Gastroenterology (759)
  • Genetic and Genomic Medicine (4103)
  • Geriatric Medicine (387)
  • Health Economics (680)
  • Health Informatics (2657)
  • Health Policy (1005)
  • Health Systems and Quality Improvement (985)
  • Hematology (363)
  • HIV/AIDS (851)
  • Infectious Diseases (except HIV/AIDS) (13695)
  • Intensive Care and Critical Care Medicine (797)
  • Medical Education (399)
  • Medical Ethics (109)
  • Nephrology (436)
  • Neurology (3882)
  • Nursing (209)
  • Nutrition (577)
  • Obstetrics and Gynecology (739)
  • Occupational and Environmental Health (695)
  • Oncology (2030)
  • Ophthalmology (585)
  • Orthopedics (240)
  • Otolaryngology (306)
  • Pain Medicine (250)
  • Palliative Medicine (75)
  • Pathology (473)
  • Pediatrics (1115)
  • Pharmacology and Therapeutics (466)
  • Primary Care Research (452)
  • Psychiatry and Clinical Psychology (3432)
  • Public and Global Health (6527)
  • Radiology and Imaging (1403)
  • Rehabilitation Medicine and Physical Therapy (814)
  • Respiratory Medicine (871)
  • Rheumatology (409)
  • Sexual and Reproductive Health (410)
  • Sports Medicine (342)
  • Surgery (448)
  • Toxicology (53)
  • Transplantation (185)
  • Urology (165)