National Pediatric Experience With Virtual Interviews: Lessons Learned and Future Recommendations

The COVID-19 pandemic had several adverse impacts on medical students and residents pursuing pediatric residency or fellowship positions, respectively. First, many applicants had an altered exposure to both general and subspecialty pediatric disciplines. Medical students experienced cancellation or significant disruption in their clerkships, including subinternships, which affected both their clinical experiences and ability to obtain letters of recommendation. Applicants were also impacted by the cancellation of away rotations. Programs had less data than in previous years to assess applicants because research experiences and volunteer activities were interrupted, and licensure examinations, including both US Medical Licensing Examination and Comprehensive Osteopathic Medical Licensing Examination, were postponed. The COVID-19 pandemic disproportionately affected Black, Indigenous, and people of color applicants; those from lower socioeconomic status backgrounds; and international medical graduates (IMGs).^4–8  In response to these concerns, APPD, COMSEP, CoPS, and AMSPDC proposed several recommendations and developed several initiatives designed to assist applicants and programs.

Our action teams took several steps to help applicants navigate the virtual interview season. Through the wide dissemination of our recommendations and our webinars, we reassured residency and fellowship applicants that programs recognized that there might be gaps in their clinical, research, advocacy, and other extracurricular experiences because of the impacts of COVID-19. Our initial communication with programs and applicants referred applicants to the Association of American Medical Colleges (AAMC) “Apply Smart” Web site,⁹  which provides evidence-based guidance for applying to an optimal number of residency programs. We also provided reminders of this guidance during the webinars.

APPD and COMSEP partnered with a new medical student–led group, FuturePedsRes, and hosted a series of webinars designed to improve student awareness of programs across the country and help mitigate applicant anxiety. In total, we hosted nine 90-minute webinars (2 general webinars regarding the application and interview process, 5 regional webinars, 1 webinar for osteopathic students, and 1 webinar for IMGs). The regional webinars consisted of (1) a brief overview of the region, (2) small groups composed of 5 to 8 programs that made brief presentations on their programs, (3) live question-and-answer sessions with programs, and (4) opportunities for students to meet exclusively with residents. The didactic content of these webinars lasted 60 minutes, and the time with the students and residents was 30 minutes. Programs developed a 1-slide infographic using a uniform template to standardize the information available to the applicant attendees. We recorded the small-group presentations and placed them on YouTube (https://www.youtube.com/c/futurepedsres) for access by interested applicants. These webinars were well received, with more than half of all pediatric programs participating (N = 138 of 211) and a consistently large number of applicants in attendance for each session (range 90–936, mean 431). FuturePedsRes also encouraged pediatric applicants to “take the pledge” through the social media #ApplySmart Campaign to limit their applications to 15 programs, as recommended by the AAMC (barring other mitigating circumstances), with the goal of decreasing overall numbers of applications so programs could complete holistic applicant reviews. For fellowship applicants, APPD and CoPS hosted a virtual café to provide information about the fellowship season and answer applicant questions.

APPD, COMSEP, and CoPS provided guidance and support to programs in several different ways. First, we encouraged programs to develop strategies to showcase their programs virtually, by updating their Web sites and developing digital brochures, videos, or other resources to highlight important aspects of their programs. APPD, COMSEP, CoPS, and AMSPDC developed a series of virtual cafés to provide an opportunity for programs to share best practices in virtual recruitment.¹⁰  A group of >20 program coordinators formed the APPD Recruitment Resource Toolkit Development Team and compiled an extensive collection of tools to support virtual recruitment and the professional development of program learners and leaders.¹¹ 

Second, we provided programs with guidance about conducting virtual interviews. Given the national recommendations for virtual interviews, we discouraged programs from scheduling any in-person interviews, even for internal or local applicants. We recommended that programs offer the daytime components of the interview day (individual interviews, conferences, tours, etc) on a single day to make scheduling easier for applicants.

Despite the challenges of interviewing in a new format and during a pandemic, APPD and COMSEP also issued guidance designed to optimize equity for all parties. First, although holistic review of residency and fellowship applications has been discussed for several years, our heightened focus on equity helped us to strongly encourage programs to move toward a holistic review of applications. The APPD described and highlighted models for performing holistic reviews during webinars for program leaders. We reminded programs that students’ access to educational, research, and extracurricular opportunities are inequitable at baseline, and many students experienced nontraditional clerkships during the pandemic, including virtual-learning experiences and online educational programming. In addition, we asked programs to reexamine their traditional criteria for offering interviews and to modify or waive requirements for pediatric-specific letters of recommendation, a pediatric subinternship rotation, and completion of US Medical Licensing Examination Step 2CK or Comprehensive Osteopathic Medical Licensing Examination Level 2-CE by the time of initial review. Although we developed these recommendations in response to COVID-19 restrictions, applicants also benefitted from minimizing the use of test scores and other elements of an applicant’s portfolio that are known to introduce biases into the interview process.

Second, for residency interviews, we recommended that the AAMC postpone the Electronic Residency Application Service release date by 3 weeks to allow adequate time for applicants and medical schools to prepare their applications and asked programs to notify all applicants if they would be offered an interview, waitlisted, or not offered an interview by December 15th. We recommended similar adjustments for the fellowship interview season. Again, we strongly discouraged second-look visits, even for internal or local applicants to maintain a safe and equitable experience for all applicants.

We also made recommendations to limit postinterview communication. Beyond adhering to existing National Resident Matching Program rules, we encouraged programs and applicants to only communicate in the event of specific questions or to link applicants with mentors and/or research colleagues. Recognizing that programs were under significant financial constraints because of COVID-19 and that some programs had reduced administrative support, we asked that programs not provide applicants with gift cards for food and that they not provide any program-related gifts to applicants.

Finally, we structured the regional webinars for residency applicants described previously to provide equitable opportunities for programs. We invited all programs to participate; elected regional APPD leaders gave an overview of the opportunities in the region; and programs presented information about their program using a templated infographic to reduce variability. To remain equitable to the programs as well as applicant participants across the series, we randomly assigned students to small groups for the residency program presentations, with access to recordings of each group afterward.

We deployed 2 sets of surveys to assess the impact of the virtual interview season on stakeholders, including program directors, program coordinators, and residency applicants.

The first survey, developed by the APPD Recruitment Resource Toolkit Development Team, solicited feedback from residency programs. This survey included questions to assess the usefulness and efficacy of the virtual toolkit, as well as programs’ overall satisfaction with the tools provided and/or used. Additionally, the survey compared the 2019–2020 traditional interview season to the 2020–2021 virtual season regarding areas such as time commitment, workload, and technology. The survey consisted of both forced-choice and open-ended questions and was institutional review board–approved and reviewed by the APPD Research and Scholarship Learning Community. We distributed the survey by e-mail to residency program directors from January 2021 to May 2021, with 2 additional reminders for nonrespondents.

We received responses from 97 of 200 residency program directors (49%). Residency programs reported completing a greater number of interviews in 2020–2021 (Table 1). Although programs reported an initial increase in workload in preparing for the virtual season, they also endorsed that the inaugural virtual interview season went better than expected. In addition, they felt that the virtual process saved time and anticipated continued time savings if the interview season remained virtual in future years. Although a hybrid model for interviews was not offered in this survey, residency directors (58%) had a preference for virtual interviews only if an applicant had extenuating circumstances, and only 2% expressed a preference for in-person interviews for all applicants. Advantages to the virtual interview seasons reported by respondents included greater equity for applicants and cost savings for both programs and applicants. Challenges included technological issues and less opportunity for in-person interactions.

The second survey was sent to residency applicants who participated in the #PedsMatch21 Webinar Series. This survey was used to solicit feedback on the webinars and to collect data related to the virtual interview process, including perceptions on the current season and their preferences for future interviewing models. This survey was distributed by e-mail in March 2021, with 2 reminders for nonrespondents, and closed before Match Day (March 19, 2021). The survey included both forced-choice and open-ended questions.

We received responses from 265 residency applicants (33%) of the 805 who attended the webinars. Recognizing that groups of applicants may have varying perspectives, we categorized respondents into 3 groups: MD, DO, and IMGs (Table 2). In the survey, we found that DO and IMGs applied to more programs than MD students and that both MD and DO applicants interviewed with and ranked a similar number of programs. All 3 groups canceled a low number of interviews. Each of the groups reported that they were only slightly more likely to rank programs in cities that they had lived in or previously visited (MD 2.5 [SD = 1.4], DO 2.5 [1.5], and IMGs 2.4 [1.4], in which 1 = strongly agree and 3 = neutral), which was unexpected given that previous studies identified geography as an important predictor regarding how applicants rank programs.^12,13  Using a slider scale from 0 (no preference) to 100 (best situation imaginable), all applicants rated a hybrid model (defined as virtual faculty interviews with on-site visits for tours and meetings with residents and/or program leaders) significantly higher than the in-person or virtual models (hybrid 78.5 [SD = 28.4]) versus in person (56.0 [26.1]) versus virtual (50.6 [27.2]) (P < .0001 for both comparisons).

When asked about advantages of the virtual interview season, a majority of respondents reported significant savings in interview costs and time, less fatigue because of lack of travel, and less difficulty scheduling interviews. Several respondents stated some variation of a “more even playing field for those who do not have as much financial ability to travel,” as an additional advantage. In terms of disadvantages to the virtual season, many respondents reported the inability “to more thoroughly get to know a program, its culture and people, its facilities, and its city.” Others reported fatigue from being on virtual interviews for long periods of time, and some applicants were concerned that programs were able to hide the less desirable aspects of their programs. Some respondents also expressed concern about “interview hoarding” by other applicants.

The Residency Recruitment and Fellowship Recruitment action teams met regularly during the interview season to review the experiences of stakeholders and after the residency and fellowship matches to consider 3 options for conducting interviews during the 2021–2022 season. In April 2021, the Coalition for Physician Accountability issued preliminary recommendations from their Undergraduate Medical Education to Graduate Medical Education Review Committee for the undergraduate medical education to graduate medical education transition.¹⁴  The recommendations Coalition for Physician Accountability made about the interview process were consistent with the consensus of our action teams.

The 3 options for interviews that the action teams discussed included conducting interviews all in person, all virtually, and a hybrid approach (consisting of virtual interviews and a separate, often abbreviated in-person component) (Table 3). Below we discuss the pros and cons of each model:

This model is the traditional model for residency and fellowship interviews, in which interviews occur in person and applicants are able to tour the clinical facilities, meet with program residents and faculty, and potentially explore the city in which they would be living. Benefits of this model include better ability of programs to assess applicant communication skills; more organic and informal conversation between applicants and residents and/or faculty and/or coordinators at programs; and the opportunity for applicants to evaluate the facilities and resources of the programs. This option also allows applicants to see the city in which they might live and work. In addition, UIM applicants have highlighted that in-person interviews provide an opportunity to consider the institution’s culture and inclusion.

There are several costs associated with this option. Although the financial cost to applicants varies on the basis of specialty choice and range of geography desired, it has been estimated that the median cost of a traditional residency interview season is ∼$4000, with a range from $1000 to >$11000.¹⁵  The median cost per program for residency recruitment for internal medicine programs was $148 000 in 2009¹⁶ ; using different methodology, authors of a survey of family medicine programs estimated a cost in excess of $25 000 per program in 2019.¹⁷  In addition to the financial costs, applicants usually spend several weeks traveling to and interviewing at programs, which results in decreased time on clinical rotations. In addition to financial costs, there are broader environmental impacts of interview travel. Recently, in a study from the University of Michigan, researchers estimated a carbon footprint from travel of 3.07 metric tons of carbon dioxide per student, who attended an average of 14.4 interviews per person.¹⁸ 

Although vaccination rates are rising, especially among health care personnel, the medical risk of travel, the differing ability of health systems to host large numbers of applicants, and the variable restrictions on travel by home institutions and around the country are further considerations for the 2021–2022 interview season.

The advantages of this model, based on the 2020–2021 interview season, were the significantly reduced financial costs for applicants and programs and less time away from the education program, an observation that was particularly notable for fellowship applicants. An all-virtual model for interviews may also support the shift toward a more holistic review of applicants because it may foster an increased focus on the application materials, with attempts to select candidates on the basis of a broad range of characteristics that are important to programs. Virtual interviews may also mitigate some of the inequity inherently present in the current interview process. For the 2020–2021 interview season, having all programs and applicants interview using this model equalized the playing field for all involved. The significantly lower financial costs of virtual interviews also resulted in a more equitable opportunity for applicants who could not afford to interview at as many places in person. The virtual-only interview process may have encouraged applicants to apply to more geographically diverse programs. The virtual model was also used successfully for medical school interviews, with students citing many of the same advantages as residents.¹⁹ 

In contrast, a virtual-only interview process makes it challenging for programs and applicants to really get a feel for each other, an impact that is hard to define and difficult to measure. Additionally, the lower cost of interviewing could lead to applicants interviewing at more programs, adding to the administrative work of programs to review more applications. This was counteracted for residency applications in 2020 with the extensive FuturePedsRes, APPD, and COMSEP campaign to only apply to 15 programs for the residency match. Electronic Residency Application Service reports that overall, pediatric applicants applied to a slightly decreased median number of programs in 2020–2021 compared with 2019–2020 (30 vs 31), whereas other large specialties saw increases in the mean number of applications per person (family medicine +1.46, internal medicine +7.55, and general surgery +5.74). Given that IMG and DO applicants generally apply to more programs, and more IMG applicants were in the pools for general surgery and internal medicine, we also looked at the change in the mean numbers of applications for MD applicants from 2019–2020 to 2020–2021 seasons for these specialties and found that only pediatrics had fewer applications per applicant: pediatrics (-2.06), family medicine (+1.86), internal medicine (+1.89), and general surgery (+3.63).²⁰ 

Given the challenges of both the in-person and virtual models, a hybrid model might maximize the benefits of each option while providing flexibility for applicants and programs. The hybrid model has been explored by O’Malley et al before the COVID-19 pandemic.²¹  In their model, the University of Arizona Internal Medicine Residency program developed a “SPLIT” interview process in 2017 to address costs, faculty and resident burden, and duplication of program director efforts. Their outcomes have included increasing the number of interviews by 50%, improved flexibility, and decreased total visit events by 70%, and allowed 100% of applicants to be interviewed by a program director and associate program director.²¹ 

There are 2 major obstacles to the widespread use of this model, particularly for the main residency match: scheduling logistics and potential for inequity. Planning a hybrid schedule is complex because this requires programs to allow for a series of virtual interview days in addition to a set of in-person sessions. Applicants would have to manage a more complicated scheduling process to accommodate all of these events. Secondly, regarding equity, even if programs stated that an in-person visit was not required for an applicant to be ranked, applicants would likely feel an obligation to travel because of the inherent competitiveness of the application process, and some applicants may be better positioned than others to avail themselves of this option. There is also the potential for programs’ bias toward those applicants who attend in-person events. The COVID-19 pandemic has shed light on these inequities in the art of recruitment, and although we must balance the potential impact of signals related to postinterview communication, interview interactions, and so on, we must also be mindful that the nature of such signals are subjective measures that can add bias to the interview process. An option to limit this bias would be to separate the in-person visit from the ranking process, but the logistic challenges of scheduling these events would still exist. One consideration is to partner with the National Resident Matching Program to move the program rank date a month earlier than applicants, thereby allowing applicants the option to complete an in-person visit without worry that it would influence the program’s rank list. Another option is to have a centralized graduate medical education office run the in-person visit component to minimize any bias from program and/or applicant interaction. Finally, in-person visits increase the cost to both applicants and programs, challenging the goals of equity that our community strives to uphold.

After gathering data from stakeholders, including applicants, program leadership, and chairs, the recruitment action teams have recommended proceeding with virtual-only interviews in 2021–2022, with the plan to reevaluate for future years thereafter.²²  There is still significant uncertainty about the impacts of the COVID-19 pandemic on health systems and cities across the country. Given that programs and applicants felt that virtual residency recruitment was overall effective this past year, the action teams believe that the benefit of reduced financial costs and increased applicant equity in a virtual system outweigh the downsides of not having program leaders and applicants meet in person. If the health risks continue to decline, it would certainly be possible for applicants to visit the cities in which they might potentially match; we still recommend that programs not host applicants in their health systems (because of burdens on the system) or have program personnel meet with the applicants (because of the risk for inequity among applicants and programs during such a visit).

An important finding from this work was the value of advocacy from the student-led, grassroots group FuturePedsRes. Navigating the myriad challenges related to moving to a virtual interviewing season was supported by the innovative FuturePedsRes organization and their use of social media to develop a more effective communication strategy for pediatric applicants. They were also able to support our efforts to recommend adoption of the Apply Smart Campaign, and their advocacy efforts may play a role in assisting our pediatric organizations in many domains of the medical education process in the future.

There are many questions to be answered to make recommendations for the future, beyond the current pandemic.²³  First, what is the impact of not having applicants and programs meet in person? Will more residents and fellows change programs or is there enough information in virtual interviews to ensure a good match? Can programs improve their own virtual interviewing skills to glean the information necessary to mitigate possible mismatches? Can programs create virtual events to share their program culture in a way that allows applicants to get a feel for their future program? What is the impact of virtual-only interviews on the geographic distribution of trainees?

Second, are there ways to improve equity during the interview process? What are the best practices for decreasing bias? Can the interview season evolve to allow for reduced financial expense for applicants and programs? Can we find a way to benefit applicants while reducing the number of interviews that each program has to conduct to reliably fill their program? Can some of the signaling methodologies (in which applicants are given a limited number of signals that they can send to certain programs to indicate a high level of interest in the program), as used in otolaryngology residency programs,²⁴  be a solution? Can programs enter their match list before applicants, allowing time for applicants to experience in-person visits to a select number of programs before entering their own match lists?

Finally, do residency and fellowship programs need to function in the same way? What special accommodations might be made for fellowship applicants, for whom a niche program or the connection with specific mentors might be more important than for applicants of a residency program? Is a modified process needed for smaller programs or those in rural areas to attract applicants? How do applicants who are couples matching in different specialties manage the differences in interview processes? And can modifications reduce the impact of the interview season on the environment?

The COVID-19 pandemic has created an opportunity to reexamine the entire recruitment process for residency and fellowship training. It has also highlighted inequities in the traditional system and forces educators to refine, or even redefine, the process so it is effective and equitable for both for applicants and programs.

Melissa Bales, C-TAGME, Indiana University School of Medicine; Katherine M. Bartoletta, MD, MS, University of Washington, Seattle Children’s Hospital; Debra Boyer, MD, MHPE, Boston Children’s Hospital; April O. Buchanan, MD, University of South Carolina School of Medicine Greenville/Prisma Health; Douglas Carlson, MD, Southern Illinois University; Jennifer G. Duncan, MD, Washington University School of Medicine; Molly Rose Elkins-Ryan, Rush University; Rebekah Fenton, MD, Ann & Robert H. Lurie Children’s Hospital of Chicago; Hayley Gans, MD, Stanford School of Medicine; Nami Jhaveri MD, MPH, Kaiser Permanente Northern California; Carrie M. Johnson, MBA, Stanford School of Medicine; Abigail Keogh, MD, Brown University; Jennifer C. Kesselheim, MD, Med, Dana-Farber/Boston Children's Cancer and Blood Disorders Center, Harvard Medical School; Rachel E. Korus MD, Boston Children’s Hospital and Harvard Medical School; Alexandra Mientus, MD, University of Louisville; Amanda D. Osta, MD, Loyola University Medical Center; Kris Rooney, MD, Lehigh Valley Reilly Children’s Hospital; Sandra M. Sanguino, MD, MPH, Northwestern University Feinberg School of Medicine; Danielle Shin, MD, PhD, Stanford School of Medicine; Adrienne Smallwood, MD, Baylor School of Medicine; Meghan Stawitcke, BA, Stanford School of Medicine; Alicia Williams, MD, Ann & Robert H. Lurie Children’s Hospital of Chicago; Xavier Williams, MD, University of North Carolina

AAMC

Association of American Medical Colleges

AMSPDC

Association of Medical School Pediatric Department Chairs

APPD

Association of Pediatric Program Directors

COMSEP

Council on Medical Student Education in Pediatrics

CoPS

Council of Pediatric Subspecialties

COVID-19

coronavirus disease 2019

DO

doctor of osteopathy

IMG

international medical graduate

MD

medical doctor