Abstract
Objectives Global, Covid-driven restrictions around face-to-face interviews for healthcare student selection have forced admissions staff to rapidly adopt adapted online systems before supporting evidence is available. We have developed, what we believe is, the first fully automated interview grounded in Multiple Mini-Interview (MMI) methodology. This study aimed to explore test re-test reliability, acceptability and usability of the system.
Design, setting and participants mixed-methods feasibility study in Physician Associate (PA) programmes from two UK and one US university during 2019 - 2020.
Primary, secondary outcomes Feasibility measures (test retest reliability acceptability and usability) were assessed using intra-class correlation (ICC), descriptive statistics, thematic and content analysis.
Methods Volunteers took (T1), then repeated (T2), the automated MMI, with a seven-day interval (+/− 2) then completed an evaluation questionnaire. Admissions staff participated in focus group discussions.
Results Sixty-two students and seven admission staff participated; 34 students and four staff from UK and 28 students and three staff from US universities.
Good-excellent test-retest reliability was observed with T1 and T2 ICC between 0.62-0.81 (p<0.001) when assessed by individual total scores (range 80.6-119), station total scores 0.6-0.91, p<0.005, individual site (all ICC≥ 0.76 p<0.001) and mean test retest across sites 0.82 p<0.001 (95% CI 0.7-0.9).
Admissions staff reported potential to reduce resource costs and bias through a more objective screening tool for pre-selection or to replace some MMI stations in a ‘hybrid model’. Maintaining human interaction through ‘touch points’ was considered essential.
Users positively evaluated the system, stating it was intuitive with an accessible interface. Concepts chosen for dynamic probing needed to be appropriately tailored.
Conclusion These preliminary findings suggest that the system is reliable, generating consistent scores for candidates and is acceptable to end-users provided human touchpoints are maintained. Thus, there is evidence for the potential of such an automated system to augment healthcare student selection processes.
Competing Interest Statement
Authors AC and LG are co-founders and Ach is an employee of Sammi-Select a spinout company from the University of Surrey, UK set up after these data were collected but before this paper was drafted in its final form.
Funding Statement
this work was supported by the United Kingdom Engineering and Physical Sciences Research Council, Impact Acceleration Fund and Innovate UK.
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
All participants gave informed consent. Ethical approval was given from all participating sites including: The primary site, The University of Surrey Research Ethics Committee (UEC/2017/111/FHMS), St Georges University of London Joint Research and Enterprise Services (title used as reference, no number allocated), University of Utah Research Integrity and Compliance Committee IRB, reference: 00125158
All necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines and uploaded the relevant EQUATOR Network research reporting checklist(s) and other pertinent material as supplementary files, if applicable.
Yes
Footnotes
This version revised with title changed to MMI in full: Multiple Mini Interview with MMI in brackets
Data Availability
The datasets used and/or analysed during the current study are available from the corresponding author on reasonable request. Detailed technical information is withheld due to commercial sensitivity.
