Abstract
Aims Large language models (LLMs), exemplified by ChatGPT have recently emerged as potential solutions to challenges of traditional diabetes education. This study aimed to explore the feasibility and utility of ChatGPT application in diabetes education.
Methods We conducted a multi-dataset, multi-reviewer study. In the retrospective dataset evaluation, 85 questions covering seven aspects of diabetes education were collected. Three physicians evaluate the ChatGPT responses for reproducibility, relevance, correctness, helpfulness, and safety, while twelve laypersons evaluated the readability, helpfulness, and trustworthiness of the responses. In the real-world dataset evaluation, three individuals with type 2 diabetes (a newly diagnosed patient, a patient with diabetes for 20 years and on oral anti-diabetic medications, and a patient with diabetes for 40 years and on insulin therapy) posed their questions. The helpfulness and trustworthiness of responses from ChatGPT and physicians were assessed.
Results In the retrospective dataset evaluation, physicians rated ChatGPT responses for relevance (5.98/6.00), correctness (5.69/6.00), helpfulness (5.75/6.00), and safety (5.95/6.00), while the ratings by laypersons for readability, helpfulness, and trustworthiness were 5.21/6.00, 5.02/6.00, and 4.99/6.00, respectively. In the real-world dataset evaluation, ChatGPT responses received lower ratings compared to physicians’ responses (helpfulness: 4.18 vs. 4.91, P <0.001; trustworthiness: 4.80 vs. 5.20, P = 0.042). However, when carefully crafted prompts were utilized, the ratings of ChatGPT responses were comparable to those of physicians.
Conclusions The results show that the application of ChatGPT in addressing typical diabetes education questions is feasible, and carefully crafted prompts are crucial for satisfactory ChatGPT performance in real-world personalized diabetes education.
What’s new?
This is the first study covering evaluations by doctors, laypersons and patients to explore ChatGPT application in diabetes education. This multi-reviewer evaluation approach provided a multidimensional understanding of ChatGPT’s capabilities and laid the foundation for subsequent clinical evaluations.
This study suggested that the application of ChatGPT in addressing typical diabetes education questions is feasible, and carefully crafted prompts are crucial for satisfactory ChatGPT performance in real-world personalized diabetes education.
Results of layperson evaluation revealed that human factors could result in disparities of evaluations. Further concern of trust and ethical issues in AI development are necessary.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study was funded by he grants from the National Key Research and Development Program of China (No. 2022YFC2505204), the Shanghai Municipal Health Commission (No. 2022JC015), and the National Nature Science Foundation (No. 82000822).
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present study are available upon reasonable request to the authors
