Abstract
Objectives Ensuring equity, inclusivity, and diversity in health professions selection is an ethical and practical imperative. We have built the first known online asynchronous Multiple Mini-Interview (MMI).
We aimed to explore psychometric properties for all users with sub-group analysis by key characteristics, acceptability, and usability.
Design, setting, participants Cross-discipline multi-method evaluation with applicants to Nursing, Midwifery and Paramedic Science under-graduate programmes from one UK university (2021/2022).
Primary, secondary outcome measures Psychometric properties (internal consistency, construct validity, dimensionality) were assessed using Cronbach’s alpha (α), parallel analysis (PA), Schmid-Leiman transformation and ordinal confirmatory factor analysis (CFA). Usability and acceptability were evaluated using descriptive statistics and conventional content analysis.
Methods The system was configured in a seven question four-minute MMI. Applicants’ video-recorded their answers which were later assessed by interviewers and scores summed. Applicants and interviewers completed online evaluation questionnaires.
Results Performance data from 712 applicants determined good-excellent reliability for the asynchronous MMI assessment (mean α 0.72) with similar results across sub-groups (gender, age, disability/support needs, UK/non-UK). Parallel analysis and factor analysis results suggested that there were seven factors relating to the MMI questions with an underlying general factor that explained the variance in observed candidate responses. A confirmatory factor analysis testing a seven-factor hierarchical model showed an excellent fit to the data (Confirmatory Fit Index =0.99), Tucker Lewis Index =0.99, RMSE=0.034).
Applicants (n=210) viewed the flexibility, relaxed environment, and cost savings advantageous. Interviewers (n=65) reported the system intuitive, flexible with >70% time saved compared to face-to-face interviews. Reduced personal communication was cited as the principle disadvantage.
Conclusions Our findings suggest that the asynchronous MMI is reliable, time-efficient, fair, and acceptable. In the absence of any known precedent, these internationally applicable, cross discipline insights inform the future configuration of online interviews where building-in principles for fairness are relatively straight forward to implement.
Study strengths and limitations
The theoretical approach aligned with an iterative process necessary to design a new technology to reduce bias.
The large sample enabled us to assess psychometric properties with sub-group analysis for the first time in this context.
The study provides perspectives from one large site; a necessary step to inform a planned international multi-site evaluation.
The multi-method design provided insights necessary to embed fairness into online selection approaches in the absence of best practice guidance.
Competing Interest Statement
Authors AC and LG are co-founders and Ach is an employee of Sammi-Select Lt, funded by UKRI as a spinout company from the University of Surrey to build the platform.
Funding Statement
This study was funded in part by Innovate UK
Author Declarations
I confirm all relevant ethical guidelines have been followed, and any necessary IRB and/or ethics committee approvals have been obtained.
Yes
The details of the IRB/oversight body that provided approval or exemption for the research described are given below:
The University of Surrey Research Ethics Committee gave ethical approval for this work (UEC/2017/111/FHMS). All participants sharing personal data gave informed consent.
I confirm that all necessary patient/participant consent has been obtained and the appropriate institutional forms have been archived, and that any patient/participant/sample identifiers included were not known to anyone (e.g., hospital staff, patients or participants themselves) outside the research group so cannot be used to identify individuals.
Yes
I understand that all clinical trials and any other prospective interventional studies must be registered with an ICMJE-approved registry, such as ClinicalTrials.gov. I confirm that any such study reported in the manuscript has been registered and the trial registration ID is provided (note: if posting a prospective study registered retrospectively, please provide a statement in the trial ID field explaining why the study was not registered in advance).
Yes
I have followed all appropriate research reporting guidelines, such as any relevant EQUATOR Network research reporting checklist(s) and other pertinent material, if applicable.
Yes
Footnotes
Competing interest statement Authors AC and LG are co-founders and Ach is an employee of Sammi-Select Lt, funded by UKRI as a spinout company from the University of Surrey to build the platform.
Ethical approval The study received a favourable ethical opinion from the Universities Research Ethics Committee (UEC/2017/111/FHMS). All participants sharing personal data gave informed consent.
Ethics This study received a favourable ethical opinion (FEO) from the University Research Ethics committee (UEC/2017/111/FHMS). Participants gave informed consent where personal data was included.
Data Availability
All data produced in the present study are available upon reasonable request to the authors. Assessment criteria are withheld due to university interview test security.
