Abstract
Background AI-driven symptom checkers (SC) are increasingly adopted in healthcare for their potential to provide users with accessible and immediate preliminary health education. These tools, powered by advanced artificial intelligence algorithms, assist patients in quickly assessing their symptoms. Previous studies using clinical vignette approaches have evaluated SC accuracy, highlighting both strengths and areas for improvement.
Objective This study aims to evaluate the performance of the Ubie Symptom Checker (Ubie SC) using an innovative large language model-assisted (LLM) simulation method.
Methods The study employed a three-phase methodology: gathering 400 publicly available clinical vignettes, medical entity linking these vignettes to the Ubie SC using large language models and physician supervision, and evaluation of accuracy metrics. The analysis focused on 328 vignettes that were within the scope of the Ubie SC with accuracy measured by Top-5 hit rates.
Results Ubie achieved a Top-5 hit accuracy of 63.4% and a Top-10 hit accuracy of 71.6%, indicating its effectiveness in providing relevant information based on symptom input. The system performed particularly well in domains such as the nervous system and respiratory conditions, though variability in accuracy was observed across different ICD groupings, highlighting areas for further refinement. When compared to physicians and comparator SC’s that used the same clinical vignettes set, Ubie compared favorably to the median physician hit accuracy.
Conclusions The Ubie Symptom Checker shows considerable promise as a supportive education tool in healthcare. While the study highlights the system’s strengths, it also identifies areas for improvement suggesting continued refinement and real-world testing are essential to fully realize Ubie’s potential in AI-assisted healthcare.
Competing Interest Statement
NKT serves as a consultant for Ubie Inc., and TN is an employee of Ubie Inc.
Funding Statement
This study was funded by Ubie Inc.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Data Availability
All data produced in the present work are contained in the manuscript
