RT Journal Article
SR Electronic
T1 ChatGPT 4 Versus ChatGPT 3.5 on The Final FRCR Part A Sample Questions. Assessing Performance and Accuracy of Explanations
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.09.06.23295144
DO 10.1101/2023.09.06.23295144
A1 Ghosn, Youssef
A1 Sardouk, Omar El
A1 Jabbour, Yara
A1 Jrad, Manal
A1 Kamareddine, Mohammed Hussein
A1 Abbas, Nada
A1 Saade, Charbel
A1 Ghanem, Alain Abi
YR 2023
UL http://medrxiv.org/content/early/2023/09/08/2023.09.06.23295144.abstract
AB Objective To evaluate the performance of two versions of ChatGPT, GPT4 and GPT3.5, on the Final FRCR (Part A) also referred to as FRCR Part 2A radiology exam. The primary objective is to assess whether these large language models (LLMs) can effectively answer radiology test questions while providing accurate explanations for the answers.Methods The evaluation involves a total of 281 multiple choice questions, combining the 41 FRCR sample questions found on The Royal Collage of Radiologists website and 240 questions from a supplementary test bank. Both GPT4 and GPT3.5 were given the 281 questions with the answer choices, and their responses were assessed for correctness and accuracy of the explanations provided. The 41 FRCR sample questions difficulty was ranked into “low order” and “high order” questions. A significance level of p<0.05 was used.Results GPT4 demonstrated significant improvement over GPT3.5 in answering the 281 questions, achieving 76.5% correct answers compared to 52.7%, respectively (p<0.001). GPT4 demonstrated significant improvement over GPT3.5 in providing accurate explanations for the 41 FRCR sample questions, with an accuracy of 65.9% and 31.7% respectively (p=0.002). The difficulty of the question did not significantly affect the models’ performances.Conclusion The findings of this study demonstrate a significant improvement in the performance of GPT4 compared to GPT3.5 on FRCR style examination. However, the accuracy of the provided explanations might limit the models’ reliability as learning tools.Advances in Knowledge The study indirectly explores the potential of LLMs to contribute to the diagnostic accuracy and efficiency of medical imaging while raising questions about the current LLMs limitations in providing reliable explanations for radiology related questions hindering its uses for learning and in clinical practice.HighlightsChatGPT4 passed an FRCR part 2A style exam while ChatGPT3.5 did not.ChatGPT4 showed significantly higher correctness of answers and accuracy of explanations.No significant difference in performance was observed between “high order” and “lower order” questions.Explanation accuracy was lower than correct answers rate limiting the Models’ reliability aslearning tools.Competing Interest StatementThe authors have declared no competing interest.Funding StatementThis study did not receive any fundingAuthor DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesI confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesAll data produced in the present study are available upon reasonable request to the authorsFRCRFellowship of Royal College of RadiologistsRCRRoyal College of RadiologistsLMLsLarge Language ModelsChatGPTChat Generative Pre-trained TransformerAIArtificial IntelligenceLSATLaw School Admission TestUSMLEUnited States Medical License ExaminationPGYPost Graduate Year