Abstract
Background Interest surrounding generative large language models (LLMs) has rapidly grown. While ChatGPT (GPT-3.5), a general LLM, has shown near-passing performance on medical student board examinations, the performance of ChatGPT or its successor GPT-4 on specialized exams and the factors affecting accuracy remain unclear.
Objective To assess the performance of ChatGPT and GPT-4 on a 500-question mock neurosurgical written boards examination.
Methods The Self-Assessment Neurosurgery Exams (SANS) American Board of Neurological Surgery (ABNS) Self-Assessment Exam 1 was used to evaluate ChatGPT and GPT-4. Questions were in single best answer, multiple-choice format. Chi-squared, Fisher’s exact, and univariable logistic regression tests were employed to assess performance differences in relation to question characteristics.
Results ChatGPT (GPT-3.5) and GPT-4 achieved scores of 73.4% (95% confidence interval [CI]: 69.3-77.2%) and 83.4% (95% CI: 79.8-86.5%), respectively, relative to the user average of 73.7% (95% CI: 69.6-77.5%). Question bank users and both LLMs exceeded last year’s passing threshold of 69%. While scores between ChatGPT and question bank users were equivalent (P=0.963), GPT-4 outperformed both (both P<0.001). GPT-4 answered every question answered correctly by ChatGPT and 37.6% (50/133) of remaining incorrect questions correctly. Among twelve question categories, GPT-4 significantly outperformed users in each but performed comparably to ChatGPT in three (Functional, Other General, and Spine) and outperformed both users and ChatGPT for Tumor questions. Increased word count (odds ratio [OR]=0.89 of answering a question correctly per +10 words) and higher-order problem-solving (OR=0.40, P=0.009) were associated with lower accuracy for ChatGPT, but not for GPT-4 (both P>0.005). Multimodal input was not available at the time of this study so, on questions with image content, ChatGPT and GPT-4 answered 49.5% and 56.8% of questions correctly based upon contextual context clues alone.
Conclusion LLMs achieved passing scores on a mock 500-question neurosurgical written board examination, with GPT-4 significantly outperforming ChatGPT.
Competing Interest Statement
The authors have declared no competing interest.
Funding Statement
This study did not receive any funding.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Data Availability Statement: Due to the proprietary nature of the dataset used for this study (Self-Assessment Neurosurgery Exams American Board of Neurological Surgery Self-Assessment Exam 1), the authors are unable to post the raw data used for the analysis. However, the authors are able to share any collected data (ex. word count, question classification, ChatGPT responses, etc.) on request to other investigators who have access to this self-assessment exam.
Code Availability Statement: Code used for this study’s analyses is available to download from a public GitHub repository (https://github.com/oliverytang/chatgpt_neurosurgery).
Conflicts of Interest: The authors report no conflict of interest concerning the materials or methods used in this study or the findings specified in this paper. However, we would like to acknowledge and thank the Congress of Neurological Surgeons and American Board of Neurological Surgery for their development and dissemination of the mock exam questions used for this study.
Disclosure of Funding: The authors have no funding relevant to the conduct of this study to disclose.
Details of Previous Presentation: None.
Data Availability
Due to the proprietary nature of the dataset used for this study (Self- Assessment Neurosurgery Exams American Board of Neurological Surgery Self-Assessment Exam 1), the authors are unable to post the raw data used for the analysis. However, the authors are able to share any collected data (ex. word count, question classification, ChatGPT responses, etc.) on request to other investigators who have access to this self-assessment exam. Code used for this study's analyses is available to download from a public GitHub repository.
