RT Journal Article
SR Electronic
T1 Visualising candidate behaviour in computer-based testing: Using <em>ClickMaps</em> for exploring <em>ClickStreams</em> in undergraduate and postgraduate medical examinations
JF medRxiv
FD Cold Spring Harbor Laboratory Press
SP 2023.06.13.23291148
DO 10.1101/2023.06.13.23291148
A1 McManus, I. C.
A1 Chis, Liliana
A1 Ferro, Albert
A1 Oram, S. Helen
A1 Galloway, James
A1 O’Neill, Vikki
A1 Myers, Gil
A1 Sturrock, Alison
YR 2023
UL http://medrxiv.org/content/early/2023/06/20/2023.06.13.23291148.abstract
AB Background The rapid introduction of computer-based testing (CBT) in UK (United Kingdom) undergraduate and postgraduate medical education, mainly as a result of the COVID-19 pandemic, has generated large amounts of examination data, which we call the ClickStream. As candidates navigate through exams, read questions, view images, choose answers and sometimes change answers and return, re-read, and make further changes, the multiple actions are recorded as a series of time-stamped clicks or keystrokes. Analysing that mass of data is far from simple, and here we describe the creation of ClickMaps, which allow examiners, educationalists and candidates to visualise behaviour in examinations.Methods As an example of ClickMaps, we describe data from a single examination lasting three hours, with 100 ‘best-of-five’ questions, which was one of two papers sat in 2021 by 508 candidates as a part of the MRCP(UK) Part 2 exam. Two ClickMaps were generated for each candidate. The Full ClickMap allows the complete three-hours of the examination to be visualised, while the Early ClickMap, shows in more detail how candidates responded during the first six minutes of presentation of each of the 100 questions in the exam.Results Since the primary purpose of this paper is expository, detailed descriptions and examples of ClickMaps from eleven candidates were chosen to illustrate different patterns of responding, both common and rare, and to show how straightforward are ClickMaps to read and interpret.Conclusions The richness of the data in ClickStreams allows a wide range of practical and theoretical questions to be asked about how candidates behave in CBTs, which are considered in detail. ClickMaps may also provide a useful method for providing detailed feedback to candidates who have taken CBTs, not only of their own behaviour but also for comparison with different strategies used by other candidates, and the possible benefits and problems of different approaches. In research terms, educationalists urgently need to understand how differences in ClickMaps relate to differences in student characteristics and overall educational performance.Competing Interest StatementAF, ICM and LC are employed, are funded by, or receive consultancy fees from MRCP(UK), AF as Associate Director for Written Examinations, ICM as Educational Advisor, and LC as Head of the Research Unit. All may be contacted via MRCP(UK) Central Office, Royal College of Physicians, 11 St Andrews Place, London NW1 4LE.Funding StatementNo external funding was provided to support this research.Author DeclarationsI confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.YesThe details of the IRB/oversight body that provided approval or exemption for the research described are given below:This paper was discussed at the meeting of the MRCP(UK)'s Management and Policy Board meeting of 23rd February 2023. The Board approved the plan to publish the paper and agreed that ethical approval was not required as it was based on an audit of data rather than research. The Board noted that the decision tool of the Health Research Authority (HRA) confirmed that the study was audit and not research, and therefore did not require ethical approval (https://www.hra-decisiontools.org.uk/research/question1.html).I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.YesI understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).YesI have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.YesData from MRCP(UK) examinations are confidential and not available.