A Match Made in Cyberspace: Applicant Perspectives on a Virtual Residency Application Process

Introduction

The COVID-19 pandemic caused a paradigm shift in the residency application process, with precautions requiring the suspension of in-person away rotations and pivoting to virtual interviews during the 2020-2021 cycle.^1–3

Existing studies on the 2020-2021 virtual application process surveyed applicants before interviews occurred, asked residents in the context of fellowship applications, or assessed how faculty viewed virtual interviews.^4–10 While studies indicate that fellowship program directors prefer in-person interviews, residents applying for fellowship expressed mixed opinions about whether virtual interviews should continue.^5–10 Fellowship applicants noted the financial and time benefits of virtual interviews but expressed concerns with evaluating fit.^5–8 A comprehensive evaluation of a virtual Match process requires input from all stakeholders, including applicants to residency positions. The data remain unclear regarding whether this population wants virtual interviews to continue.^4,5,8

One noted benefit of virtual interviews is the greatly reduced monetary commitment. Before the pandemic, there were calls for change regarding the financial costs of residency applications, with several studies finding that video interviews could be a cost-saving alternative to traditional in-person interviews.^11–15 The 2020-2021 Match confirmed this, as the virtual application process was estimated to save candidates nearly $6,000.¹⁶ The switch to virtual interviews also sparked a surge in residency programs’ social media presence.^17–20 However, the utility of social media and other remote resources such as websites and virtual programming for applicants remains unclear. Another area of study is evaluating policies that address the increasing number of applications and interviews per applicant.²¹ Finally, virtual interviews may exacerbate inequities for those without access to electronic equipment and high-speed Internet, a phenomenon termed digital redlining.^22–24

Our study aims to evaluate applicants’ perspectives on the 2020-2021 virtual residency application cycle. We identified three key outcomes: [1] general thoughts including overall satisfaction and concerns with the virtual environment, [2] utility of resources provided by programs for evaluating fit, and [3] perspectives on changes to make during future cycles.

Methods

This study aimed to characterize applicants’ perspectives post-Match on the impact of a completely virtual setting on the 2020-2021 residency application process. A survey was distributed through a national medical student organization email list and grassroots social media efforts.

Given that the 2020-2021 cycle was the first completely virtual residency application process, no prior surveys were available to assess student opinions on this topic. Therefore, a 30-question survey instrument was developed by consulting members of a national medical student committee focused on medical education, 2020-2021 residency applicants, and survey methodology experts. Surveys by the National Resident Matching Program and specific specialties were also referenced.

Five M.D. and/or Ph.D. methodology experts evaluated the appropriateness of our instrument in addressing the study’s objectives as part of pre-testing. A pilot study was then conducted with 20 fourth-year medical students to ensure question clarity prior to wide distribution.

We identified three key outcomes to assess: [1] general thoughts, [2] program resources, and [3] perspectives on future cycles. First, it was important to characterize applicants’ overall satisfaction with the cycle and concerns unique to the virtual setting. Second, given how applicants were not able to visit programs in-person, we aimed to identify what resources provided by residencies were most useful for determining compatibility with a program. Third, we assessed applicant opinions on policies to implement during future cycles in response to the virtual setting.

Most questions utilized a 5-point Likert scale by which students either rated their agreement with statements about the application cycle or the helpfulness of listed resources. Five free response questions allowed applicants to elaborate on their application experience. The survey instrument can be accessed in the online supplemental content.

Response percentages, 95% confidence intervals (CIs), and statistical significance were calculated using a customized Python computing package.²⁵ For analysis of survey items using Likert scales, categories of “Agree” and “Strongly Agree” were grouped together as “Agree”, “Disagree” and “Strongly Disagree” were grouped together as “Disagree”, “Helpful” and “Extremely Helpful” were grouped together as “Helpful”, and “Unhelpful” and “Extremely Unhelpful” were grouped together as “Unhelpful”.²⁶ Statistical significance was determined by testing for overlap among the 95% CIs for “Agree”, “Neutral”, and “Disagree” for questions asking about agreement with a statement and “Helpful”, “Neutral”, and “Unhelpful” for questions asking about the helpfulness of a resource. For analysis of free response questions, three independent reviewers identified common themes that provided nuance to the Likert scale results. Discrepancies were resolved by discussion among the analysts.

The Vanderbilt Human Research Protections Program granted this study Institutional Review Board exemption.

Results

The survey was emailed to members of a national medical student organization. 11,243 students viewed the correspondence and, of those, we estimate that 25% were eligible to participate since students apply to residency during their last year of medical school. Therefore, we conclude that 2,811 email recipients met study criteria. Given that grassroots social media efforts were also utilized, students contacted via social media likely overlapped with students contacted via the national organization. It is therefore not possible to determine the exact numbers of students reached with these combined methods. With a total of 158 responses, we conservatively estimate our response rate to be approximately 5%. This survey captured individuals from 24 states who applied into 31 specialties, including 8 combined specialty programs (Table 1).

Discussion

Most applicants were satisfied with the 2020-2021 Match and wanted residencies to continue providing virtual interviews. However, many expressed equity concerns including greater opportunities for interview hoarding and disparities in resources provided by medical schools. For determining fit, Instagram and Twitter were found to be particularly helpful social media platforms, and information on official program websites was consistently found to be helpful. The majority of applicants supported a preference signaling mechanism and interview limit, but only a minority supported an application limit.

Pre-pandemic studies disagreed on whether applicants preferred virtual interviews to the traditional in-person format.^27,28 Those supporting virtual interviews primarily did so due to financial considerations.²⁸ Our survey of applicants who participated in a completely virtual residency application process demonstrated that the majority were satisfied with their Match experience and want virtual interviews to continue. However, free response comments indicated equity concerns if virtual interviews are offered as an alternative to in-person interviews due to the travel costs associated with the in-person format. Although our study was concordant with past studies finding that most applicants felt they had adequate resources to complete virtual interviews, a majority of our participants thought that their schools did not provide sufficient electronic resources.² As virtual interviews continue, this could be an opportunity for medical schools to combat digital redlining by providing students with the technology required to succeed in the virtual setting.^29–31 A recognized disadvantage of virtual interviews is the difficulty applicants have with evaluating compatibility with a program.²⁷ This is evidenced by how most of our study participants agreed that a completely virtual application process made ranking programs more challenging. To address the challenge of evaluating fit virtually, applicants have increasingly relied on the social media presence of programs.^32,33 Few studies have evaluated the utility of social media platforms for applicants in this context. Interestingly, one pre-pandemic study found that students did not believe Twitter or Instagram presence to be important when evaluating residency programs.²⁷ Our results suggest that not only did study participants find Instagram and Twitter to be helpful when evaluating program compatibility, they also found websites on the whole and most programming organized by residencies to be helpful. However, free response comments expressed caution with this increased reliance on social media due to the potential for developing bias towards students more active on social media.

Our study found that the abrupt adoption of virtual interviews raised other concerns. One was students’ increased ability to hoard interviews in the virtual setting. Possible means to address this include application and interview limits, with our study only demonstrating majority support for the latter. Interestingly, a few specialties—emergency medicine, ophthalmology, and psychiatry— implemented interview caps for the 2021-2022 cycle.^34,35 Additionally, most of our study participants supported preference signaling to address the congestion associated with the increasing number of applications residency programs receive.²¹ Otolaryngology implemented a preference signaling system during the 2020-2021 Match, and the majority of the participants were satisfied with their experience.³⁶ During the 2021-2022 cycle, more specialties—internal medicine, dermatology, and general surgery—are utilizing a preference signaling mechanism within their supplemental applications.³⁷ With more widespread adoption of both interview limits and preference signaling, it remains to be seen how these new policies will be received by Match stakeholders. Considering these findings and results from other studies, we propose four recommendations (Figure 1).^{2,27,28,30,31,34–41}

Our study has several limitations. Given that survey participation was voluntary, there was potential for response bias towards applicants with particularly strong feelings about their Match experience. Despite extensive grassroots efforts, we collected responses from only a small fraction of all Match participants, limiting analysis based on applicant identity, geography, and specialty. Additionally, since all survey items were optional, there were variable response rates across questions. Due to the lack of standardized survey instruments on this topic, we developed our own survey questions and are only able to demonstrate internal validity through pilot testing. Further research on this topic with larger sample sizes or qualitative methodology is needed to solidify students’ perspectives on this topic

Future directions for our study include assessing applicant satisfaction with interview caps and preference signaling in the virtual setting. With the possibility of in-person interviews continuing, applicant satisfaction with a hybrid interview process should also be evaluated.

Conclusions

Our study demonstrates that, across specialty and geography, Match participants support virtual interviews. Residencies can help applicants better evaluate program compatibility by investing in social media, official websites, and virtual programming, while medical schools can combat potential disparities by providing students with the resources needed for virtual interviews. Residency applicants also support interview caps and preference signaling to address the high number of applications residency programs receive and the greater potential to hoard interviews in the virtual setting.