BACKGROUND AND OBJECTIVES

Because of the coronavirus disease 2019 pandemic and recommendations from a range of leaders and organizations, the pediatrics subspecialty 2020 recruitment season was entirely virtual. Minimal data exist on the effect of this change to guide future strategies. The aim of this study was to understand the effects of virtual recruitment on pediatric subspecialty programs as perceived by program leaders.

METHODS

This concurrent, triangulation, mixed-methods study used a survey that was developed through an iterative (3 cycles), consensus-building, modified Delphi process and sent to all pediatric subspecialty program directors (PSPDs) between April and May 2021. Descriptive statistics and thematic analysis were used, and a conceptual framework was developed.

RESULTS

Forty-two percent (352 of 840) of PSPDs responded from 16 of the 17 pediatric (94%) subspecialties; 60% felt the virtual interview process was beneficial to their training program. A majority of respondents (72%) reported cost savings were a benefit; additional benefits included greater efficiency of time, more applicants per day, greater faculty involvement, and perceived less time away from residency for applicants. PSPDs reported a more diverse applicant pool. Without an in-person component, PSPDs worried about programs and applicants missing informative, in-person interactions and applicants missing hospital tours and visiting the city. A model based upon theory of change was developed to aid program considerations for future application cycles.

CONCLUSIONS

PSPDs identified several benefits to virtual recruitment, including ease of accommodating increased applicants with a diverse applicant pool and enhanced faculty involvement. Identified limitations included reduced interaction between the applicant and the larger institution/city.

What’s Known on the Subject:

The 2020 fall match season was the first to exclusively use virtual recruitment and interviewing methods across pediatric subspeciality programs. Little information exists regarding the cumulative pediatric program experience with this transition from an in-person to virtual process.

What This Study Adds:

This is the first study to evaluate the multifaceted impact of the virtual recruitment/interview process on pediatric subspecialty programs, an undertaking vital to shaping future recruitment cycles to ensure an efficient and equitable recruitment process.

The coronavirus disease 2019 (COVID-19) pandemic substantially disrupted many components of medical education, including clinical and didactic experiences. Accrediting bodies recommended the severe limitation of “away” rotations.1  Many residents believe that the pandemic adversely impacted their clinical training as patient volumes decreased in many specialties, including procedural subspecialties.2  In addition, some residents were platooned or asked to provide patient care in areas outside of their specialty training. COVID-19 also disproportionately affected certain populations, including applicants identified as from a Black, indigenous, or person of color population, lower socioeconomic status, and international medical graduates.3,4  Additionally, residents selecting to pursue pediatric subspecialty (PS) training also encountered a unique recruitment and interviewing experience because of COVID-19 mitigation efforts, with all subspecialties adopting a virtual recruitment/interview process for the fall 2020 season.

National pediatric educational organizations, convened by the Association of Pediatric Program Directors (APPD), formed a Fellowship Recruitment Action Team (FRAT) to develop consensus recommendations.5  The FRAT included representatives of the APPD, the Council of Pediatric Subspecialties, the American Academy of Pediatrics Section on Pediatric Trainees, the Association of Medical School Pediatric Department Chairs, and PS trainee representatives. The FRAT recommended all PS programs transition to a virtual interview process for the 2020–2021 academic year, a recommendation that was later endorsed for other (sub)specialty recruitments by the Council for Physician Accountability.6  Despite this change, the 2020 National Resident Matching Program application rates for PS were similar or increased for 16 of 17 PS’s.7  The FRAT again recommended a virtual interview process for the 2021–2022 academic year (as had the Council for Physician Accountability) and recommendations for the 2022–2023 cycle are still being deliberated; therefore, it is important to obtain additional information regarding the challenges and successes related to this virtual format for the graduate medical education (GME) interviewing process.

Some residency and fellowship programs had previous experiences with virtual and hybrid (combination of virtual and in-person) interviewing models before this entirely virtual recruitment season. The hybrid model has been explored before the COVID-19 pandemic by O’Malley and colleagues. In 2017, the University of Arizona internal medicine residency program implemented a hybrid interview process wherein they held required virtual interviews, followed by optional, in-person applicant visit days. Overall, they reported measurable positive outcomes, including a 50% increase in the number of interviews conducted per season, all interviewees having interviews with the program director (PD) and associate PD, and improved flexibility within the interview process. Notably, their intent was to continue to offer this hybrid model to allow applicants the opportunity to visit their campus in-person; this was modified to a wholly virtual process (ie, no in-person option) in 2020 because of COVID-19–mitigation efforts.8 

In response to the COVID-19 pandemic, several recommendations have been published regarding virtual interviewing in medical training programs.5,9,10  Advantages of virtual interviewing highlighted in these publications include improvements in efficiency, cost, and time savings. Alternatively, virtual interviews may find applicants struggling to get a “feel” of a program and its people, important components of the culture within a program. Other proposed challenges of virtual interviewing include technical difficulties, less opportunity to fully appreciate an applicant’s behavior and/or personality, and an increased administrative burden.

This is the first study to evaluate the impact of the virtual recruitment/interview process on pediatric subspecialty programs, an undertaking vital to shaping future recruitment cycles. Given the differences in applicant cohort preferences when selecting fellowships compared with residencies, our study provides valuable data that expands upon existing data regarding the virtual interview experience for residents.

A working group of 12 pediatric intensivists from the Education in Pediatric Intensive Care (E.P.I.C) Investigators’ Collaborative convened to conduct a survey among Pediatric Critical Care Medicine PDs. The initial survey was developed through an iterative, consensus-building, modified Delphi process (iteration 1).11  A collaboration developed through professional networking between E.P.I.C. and members of the FRAT modified the initial survey to be more broadly applicable across all PDs, aimed at pediatric subspecialty PDs’ (PSPDs’) perspectives and experiences (iteration 2). This version was piloted with GME stakeholders and content experts from a variety of disciplines, including a pediatric residency PD, anesthesia residency and fellowship program leadership, and former PSPDs and GME executive leadership.

Cognitive interviewing was done with select members of the pilot group to enhance construct validity of included questions.12  Willing members of the pilot group discussed their interpretation of each question and identified areas that were recommended for modification to aid understanding of each question. Feedback was incorporated into the final survey version (iteration 3), with item generation and face validity being consistent with survey design methods that have been previously described.13  Terms that could have definitional variation had group-consensus definitions determined and respondents could hover over the terms for clarity of definition during survey completion (Supplemental Information). Respondents were able to offer additional written feedback about their virtual recruitment experience.

Purposive/deliberate sampling of PSPDs was done through combining distribution lists from E.P.I.C., APPD, and the Council of Pediatric Subspecialties membership databases.14  Additionally, members of each group were encouraged to share the survey link with PSPDs who were not members of 1 of the sponsoring organizations or who would not have otherwise received it. Study data were collected and managed using Research Electronic Data Capture.15  Responses were collected between April and May 2021, with 1 reminder e-mail to participate. No incentive for participation was provided. Using a concurrent triangulation design, descriptive statistics were used for quantitative analysis; free-text responses were coded and analyzed for themes. Using thematic analysis, an inductive approach was used by authors to inform the development of a conceptual framework on the basis of the theory of change.16  The study was exempted by Vanderbilt University’s institutional review board.

Response rate was calculated on the basis of the total number of accredited PS programs submitting rank lists for the 2020–2021 recruitment cycle (n = 840). A total of 352 PSPDs (42%) who participated in recruitment responded, and 16 additional PSPDs indicated that they did not recruit during the 2020–2021 recruitment cycle. Respondents represented 16 of the 17 pediatric subspecialities. Forty-six respondents did not select a subspeciality and were thus categorized as “unspecified,” which represented 9% (46 of 534) of the total remaining respondent pool and 13% (46 of 352) of all survey respondents (Table 1). Of the programs surveyed, 9.7% reported using virtual recruitment methods in previous recruitment cycles for both screening and full interviews. Eighty-percent of PSPDs (283 of 352) indicated that increased time and effort were required in transitioning to a virtual recruitment format, with design of the virtual interview day (268 of 352; 76%), program Web site design/redesign (200 of 352; 57%), and videography (169 of 352; 48%) reported as the most time-consuming aspects of transition.

TABLE 1

Number of Programs Responding by Subspecialty Compared With Total Programs Submitting Rank List for AY 2020 to 2021

SubspecialtyResponse Rate, % (n of N)
Academic generalist 33 (4 of 12) 
Adolescent medicine 25 (6 of 24) 
Cardiology 32 (19 of 60) 
Child abuse 52 (11 of 21) 
Critical care medicine 51 (35 of 69) 
Developmental and behavioral medicine 47 (15 of 32) 
Emergency medicine 18 (15 of 82) 
Endocrinology 24 (16 of 66) 
Gastroenterology 41 (26 of 64) 
Hematology/oncology 38 (27 of 72) 
Hospital medicine 43 (23 of 54) 
Infectious diseases 29 (16 of 55) 
Neonatology 65 (62 of 96) 
Nephrology 30 (13 of 44) 
Pulmonology 27 (13 of 48) 
Rheumatology 18 (5 of 28) 
Unspecifieda 9 (46 of 521)a 
SubspecialtyResponse Rate, % (n of N)
Academic generalist 33 (4 of 12) 
Adolescent medicine 25 (6 of 24) 
Cardiology 32 (19 of 60) 
Child abuse 52 (11 of 21) 
Critical care medicine 51 (35 of 69) 
Developmental and behavioral medicine 47 (15 of 32) 
Emergency medicine 18 (15 of 82) 
Endocrinology 24 (16 of 66) 
Gastroenterology 41 (26 of 64) 
Hematology/oncology 38 (27 of 72) 
Hospital medicine 43 (23 of 54) 
Infectious diseases 29 (16 of 55) 
Neonatology 65 (62 of 96) 
Nephrology 30 (13 of 44) 
Pulmonology 27 (13 of 48) 
Rheumatology 18 (5 of 28) 
Unspecifieda 9 (46 of 521)a 

AY, academic year.

a

Forty-six programs chose not to specify which subspecialty they represented (percentage calculated unspecified respondents/total remaining programs).

The transition to a virtual format resulted in 43% (152 of 352) of programs interviewing more applicants than previous years, with 21% reporting a 10% to 25% increase, 12% reporting a 26% to 49% increase, and 11% reporting a >50% increase; 53% of programs remained within 10% of number of applicants interviewed in previous years. Although the majority (207 of 352; 59%) of programs reported ranking within 10% of the number of applicants in previous years, 36% (145 of 352) reported ranking more applicants than previous years, whereas 6% reported ranking less applicants than previous years.

Overall, 60% of programs strongly agreed (11%) or agreed (49%) that the virtual interview process was beneficial to their training program; 72% reported a decrease in overall budget spent on virtual recruitment. Furthermore, 62% of programs would include some form of virtual interviewing if in-person interviews were allowed in upcoming recruitment cycles, whereas 33% were unsure. From free-text feedback, 3 broad themes emerged, including: time, travel, and the virtual format.

Efficiency was commonly cited as an advantage to the virtual format, with the ability to interview more applicants in each interview day, compression of the interview day itself, and program overviews being relayed consistently and succinctly. As a result of the compressed interview day, PSPDs reported less time dedicated to the interview day. Programs reported increased faculty availability for the interview day, leading to increased ease in arranging the interview day schedule, in addition to accommodating more applicants on a given interview day. Applicants spent less time away from their residency training programs, which PSPDs felt would potentially allow them to interview at more programs.

Although there are advantages in less time spent during the actual interview day, this was noted to lead to fewer interactions with faculty and current fellows, resulting in applicants meeting fewer members of the fellowship and clinical teams. Several PSPDs cited difficulty in getting to know their applicant’s personality because of decreased informal interactions such as loss of conversation while walking together to the next meeting during in-person interviews.

The transition to the virtual format obviated the need for travel, resulting in cost savings for both programs and applicants. Equity was cited by PSPDs as an additional benefit because applicants did not have to weigh the time and cost of travel to the various programs they were considering, allowing those with financial or other travel constraints to apply more broadly. Importantly, PSPDs reported their perceptions of an increased geographically and financially diverse applicant pool and improved program visibility. Additionally, applicants may have virtually visited programs that they may not have otherwise considered. Alternatively, applicants who are less serious about programs may have interviewed without seriously considering a program, with some programs citing an increase in cancellation rates. Reduction in the carbon footprint was also referenced as a benefit of the virtual format.

Although there were noted benefits to the elimination of travel from the recruitment process, PSPDs also cited several potential downsides, including the inability of applicants to visit the city in which programs are located. PSPDs expressed that this may disproportionally impact smaller programs or programs from less well-known cities where they felt, historically, exposure to the medical campus was a key component to recruiting applicants. Furthermore, PSPDs felt it may be very difficult for an applicant to seriously consider (ie, highly rank) a program in a city in which they have never visited and were unable to bring significant others, or those impacted by their potential move, to a new location to assess the job market and living situation.

Because applicants were unable to travel, they were also unable to experience in-person tours of the learning environment, resulting in a loss of observation of casual interactions that occur during an in-person interview day (ie, interactions between fellows, faculty, and staff), as well as the opportunity to participate in rounds. These expressions highlight the lack of the “intangible” experience that applicants gain by being present during an interview day. Because applicants are not physically present, they may not get to experience a program’s “culture.”

The virtual format conferred an advantage for faculty, including ease of remote faculty interviewing, allowing for increased faculty availability for interviews, including faculty outside of the subspecialty. Applicants were also able to meet virtually with faculty on days outside of the specified interview day. Convenience and more time spent interviewing applicants, “more time talking, less time walking,” were additional benefits reported by PSPDs. Directors were also able to incorporate standardized questions more readily into the interview day, allowing for cross-comparison of applicants.

The virtual format also forced programs to create more online content and update Web sites, allowing for programs to showcase their program in visual media and engaging applicants on social media. Because of the emphasis on media, programs were able to provide a more consistent message about their program. PSPDs described devoting a great amount of effort/time in converting to a virtual format, mostly before interviews began. Many PSPDs reported lack of resources and support in this conversion, specifically in support of videography.

PSPDs learned that many online platforms were available and used by different programs for the actual interview day, which may have impacted the applicant experience. Technical issues were also noted as a concern with the transition to a virtual format. One of the most frequent concerns of PSPDs with a transition to the virtual format was a less-dynamic interview experience for applicants. In addition to “online fatigue,” another concern with an entirely virtual format was the difficulty in showcasing the interspecialty relationships. Creating meaningful social interactions between programs and applicants was cited as challenging, with social happy hours and meet-and-greets being poorly attended, resulting in fellows being unable to provide much feedback on applicants. Nonverbal cues and communication skills were reported to be more difficult to assess in a virtual format.

Conceptual Framework

Balancing the strengths and challenges identified in this study by PSPDs, a conceptual framework to guide future virtual cycles was created on the basis of a theory of change. This framework used an efficient and equitable recruitment/interview process for meeting workforce needs as the long-term outcome. Our framework aims to help stakeholders look beyond whether the process of virtual recruitment worked or not and identify the conditions under which such a system can be successful for meeting the long-term goal. Although the context for this specific change was a global pandemic that we had little ability to impact, program leaders will be left with the enduring question of whether this change, or elements of it, persist beyond the specific context of COVID-19.

The 3 moderators of change identified allow all programs to look at how each is applicable within their own program.

This study highlights the unique perspectives of PSPD experiences with virtual recruitment across most pediatric subspecialties. Because residents are choosing pediatric subspecialties at a faster rate than the small increase in those medical students who are entering pediatrics, identifying ways that programs can accommodate the growing applicant pool is important.17,18  This need has been accelerated by the pandemic and social issues, requiring revision of our “old” methods. The current study demonstrates the benefits of virtual interviewing that would allow for programs to accommodate a growing applicant pool, including: the relative ease of scheduling multiple interviewees simultaneously; the lower cost of virtual interviews, which would allow for more program bandwidth; and the ease of faculty participation, which promotes more flexibility with interviewing. For subspecialty training, faculty mentorship is extremely important, and allowing applicants to more broadly connect with faculty who could serve as potential clinical, scholarly, or professional mentors is vital, a benefit of enhanced faculty access through virtual interviewing.

Given the growing subspecialty candidate population, programs will be better positioned if they can attract applicants from diverse geographical and personal backgrounds. In the current study, program leadership felt that there was a more diverse applicant pool compared with previous years. In addition, program visibility is crucial to attracting candidates and, in this study, programs reported having to develop more accessible program information for potential applicants, which in the end, increased program exposure. Although there was a negative effect on time in preparing online information, programs felt the materials were extremely helpful and could be viewed asynchronously, allowing for a more efficient and personal interview experience. However, although programs felt they struggled with conveying the unique culture of their institutions, the majority of PSPDs also stated that program selection committees were able to evaluate applicants in a similar manner compared with previous in-person interviews. Determining if both programs and applicants were able to accurately assess compatibility will be important as a next step to evaluate as trainees enter their matched programs.

The impact of virtual interviewing is likely different dependent on (sub)specialty and size and location of programs. However, as programs become more adept at developing meaningful program and location information and interview experiences, the limitations may be dwarfed by the benefits of a larger online presence to attract candidates that otherwise would not have known or considered their program. In addition, if virtual interviewing continues at all trainee levels, (sub)specialty applicants will also have more experiences and ability to judge suitability of a training program and location virtually.

The personal impact of the COVID-19 pandemic is also an important consideration when analyzing the national trends among the cohort of medical trainees who self-select into the fellowship application process. New priorities may emerge when determining potential program strengths amid personal and/or family members’ health concerns, such as access to established health care and/or geographic proximity to support systems. Additionally, the broader societal impacts of the pandemic on sectors outside of health care, such as effects on the labor market, housing, financial market, and education,19  may affect decision-making as they relate to relocating significant others and other family members, as well as the monetary stressors of moving. Interestingly, despite these potential challenges of the pandemic, the 2021 appointment year (ie, 2020 match) was the largest in the history of the National Resident Matching Program specialties matching service.7  Longitudinal data trends will reveal if the 2020 match was a unique year or if it signals an ongoing trend of more residents entering subspeciality training. In-depth analysis surrounding applicant motivations for their decision to enter subspeciality training would also provide more insight into the record-breaking applicant numbers seen across many subspecialities this past year.

Other studies have also highlighted the cost and time efficiency of virtual recruitment from both the PD and applicant perspectives, and have demonstrated the strengths and weaknesses of the approach, even in consideration of equity.20,21  However, this is the first study to report on the unique impacts on pediatric subspecialty recruiting, including the vital role of faculty involvement and the growing competitive environment in some pediatric subspecialties, making the cost and time benefits essential. Importantly, the current study described the increased diversity of the applicant pools, not just the increase in applications as previously reported. The findings of the current study describing the challenge of programs conveying the culture of their programs was echoed in other studies,2022  and could provide important opportunities for improvements within the recruitment process by further enhancing the online offerings and less-informational interviews concentrating solely on the community and personal aspects of a program. The enhanced online visibility was seen as a program benefit in the current study, especially in the competitive environment of pediatric subspecialty recruitment.

There are some limitations to our study. First, although the majority of US PSPDs were invited to participate, those specialties with a larger number of available positions, such as pediatric critical care and neonatology, comprise a large percentage of our survey response rate (13% and 16%, respectively), potentially introducing sampling bias. However, as noted, these disciplines also comprise a high percentage of available fellowship positions across pediatrics (pediatric critical care and neonatology combined for 27% of all available pediatric subspeciality positions for the 2020 match year). Second, this study focused on the program director’s perspective of the virtual recruitment and interviewing process and did not solicit applicant perspectives. Given the number of applicants this would have included (1681 applicants)7  and complexities in obtaining reliable applicant contact information, obtaining this data with an appropriate response rate was considered a limitation to timely dissemination of data as it relates to informing current and upcoming application cycles. Future studies directed toward (sub)specialty-specific applicant cohorts may provide useful data surrounding the applicant experience, including analysis of perceptions after matriculation resulting from this virtual experience. Finally, this was a convenience sample of pediatric subspeciality training programs, thus programs such as internal medicine and surgical subspeciality programs were not included and responses focus on subspecialty training, potentially limiting generalizability. Given the virtual recruitment and interviewing process was recommended across all training programs in 2020, we feel that our results are generalizable across all fellowship training programs, given the need for most programs to adapt to this new approach to interacting with their applicants, but future work exploring these findings in other specialties will be important.

This study highlights important themes surrounding the experience of the transition to a wholly virtual recruitment and interviewing process. PSPDs report many benefits of the virtual methods, including ease of accommodating increased numbers of applicants within reported perceptions of a more diverse pool, along with enhanced faculty involvement. Perceived applicant benefits focus on efficiency and accessibility to viewing a broader range of programs. Future work focused on applicant perspectives and postmatch outcomes analysis, including applicant integration into programs, will be helpful in informing best practices within the virtual recruitment and interviewing process.

We thank the Council of Pediatric Subspecialties, the APPD, the Association of Medical School Pediatric Department Chairs Fellowship Recruitment Action Team, the E.P.I.C. Investigator Network, and the participants who made this work possible.

Drs Petersen and King conceptualized and designed the study, designed the data collection instruments, collected data, conducted the initial analyses, and drafted, reviewed, and revised the manuscript; Dr Boyer contributed to the study conception, design and interpretation of data, and drafted and critically reviewed the manuscript for important intellectual content; Drs Fussell, Gans, and Waggoner-Fountain contributed to the data collection instrument design and data collection, the interpretation of data, and writing/review of the manuscript; Drs Castro, Green, Hamilton, Marcdante, Mink, Turner, Watson, and Zurca contributed to the study and data collection instrument design, and reviewed the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: No external funding.

CONFLICT OF INTEREST DISCLAIMER: The authors have indicated they have no conflicts of interest relevant to this article to disclose.

APPD

Association of Pediatric Program Directors

COVID-19

coronavirus disease 2019

E.P.I.C.

Education in Pediatric Intensive Care

FRAT

Fellowship Recruitment Action Team

GME

graduate medical education

PD

program director

PS

pediatric subspecialty

PSPD

pediatric subspecialty program directors

1
AAMC The Coalition for Physician Accountability’s Work Group on Medical Students in the Class of 2021 Moving Across Institutions for Post Graduate Training
.
Final report and recommendations for medical education institutions of LCME-accredited, U.S. osteopathic, and non-U.S. medical school applicants
.
2
Rana
T
,
Hackett
C
,
Quezada
T
, et al
.
Medicine and surgery residents’ perspectives on the impact of COVID-19 on graduate medical education
.
Med Educ Online
.
2020
;
25
(
1
):
1818439
3
Schwartz
AM
,
Wilson
JM
,
Boden
SD
,
Moore
TJ
Jr
,
Bradbury
TL
Jr
,
Fletcher
ND
.
Managing resident workforce and education during the COVID-19 pandemic: evolving strategies and lessons learned
.
JBJS Open Access
.
2020
;
5
(
2
):
e0045
4
Nguyen
LH
,
Drew
DA
,
Graham
MS
, et al.
Coronavirus Pandemic Epidemiology Consortium
.
Risk of COVID-19 among front-line health-care workers and the general community: a prospective cohort study
.
Lancet Public Health
.
2020
;
5
(
9
):
e475
e483
5
Frohna
JG
,
Waggoner-Fountain
LA
, %
Edwards
J
, et al.
Pediatrics Recruitment Study Team
.
National pediatric experience with virtual interviews: lessons learned and future recommendations
.
Pediatrics
.
2021
;
148
(
4
):
e2021052904
6
ACGME Coalition for Physician Accountability
.
Recommendations for away rotations and interviews for graduate medical education fellowship applicants during the 2020–2021 academic year
.
7
National Resident Matching Program
.
Results and data: specialties matching service 2021 appointment year
.
8
O’Malley
CW
,
Bergin
C
,
Shinar
B
.
Virtual residency interviewing strategies. SPLIT recruitment: a novel approach to residency interviews during COVID-19 and beyond
.
Available at: https://youtu.be/yDMaYY9_r5g. Accessed October 23, 2021
9
Bernstein
SA
,
Gu
A
,
Chretien
KC
,
Gold
JA
.
Graduate medical education virtual interviews and recruitment in the era of COVID-19
.
J Grad Med Educ
.
2020
;
12
(
5
):
557
560
10
Joshi
A
,
Bloom
DA
,
Spencer
A
,
Gaetke-Udager
K
,
Cohan
RH
.
Video interviewing: a review and recommendations for implementation in the era of COVID-19 and beyond
.
Acad Radiol
.
2020
;
27
(
9
):
1316
1322
11
Passmore
C
,
Dobbie
AE
,
Parchman
M
,
Tysinger
J
.
Guidelines for constructing a survey
.
Fam Med
.
2002
;
34
(
4
):
281
286
12
Collins
D
.
Pretesting survey instruments: an overview of cognitive methods
.
Qual Life Res
.
2003
;
12
(
3
):
229
238
13
Burns
KE
,
Duffett
M
,
Kho
ME
, et al.
ACCADEMY Group
.
A guide for the design and conduct of self-administered surveys of clinicians
.
CMAJ
.
2008
;
179
(
3
):
245
252
14
Battaglia
MP
.
Purposive sample
. In:
Lavrakas
PJ
, ed.
Encyclopedia of Survey Research Methods
.
Thousand Oaks, CA
:
SAGE Publishing
;
2008
:
645
647
15
Harris
PA
,
Taylor
R
,
Thielke
R
,
Payne
J
,
Gonzalez
N
,
Conde
JG
.
Research electronic data capture (REDCap)–a metadata-driven methodology and workflow process for providing translational research informatics support
.
J Biomed Inform
.
2009
;
42
(
2
):
377
381
16
Reinholz
DL
,
Andrews
TC
.
Change theory and theory of change: what’s the difference anyway?
IJ STEM Ed
.
2020
;
7
(
2
)
17
Association of American Medical Colleges
.
Physician specialty data report
.
18
The American Board of Pediatrics
.
Subspecialty fellowship statistics
.
19
Schnitzler
L
,
Janssen
LMM
,
Evers
SMAA
, et al
.
The broader societal impacts of COVID-19 and the growing importance of capturing these in health economic analyses
.
Int J Technol Assess Health Care
.
2021
;
37
(
37
):
e43
20
Rockney
D
,
Benson
CA
,
Blackburn
BG
, et al
.
Virtual recruitment is here to stay: a survey of ID fellowship program directors and matched applicants regarding their 2020 virtual recruitment experiences
.
Open Forum Infect Dis
.
2021
;
8
(
8
):
ofab383
21
Domingo
A
,
Rdesinski
RE
,
Stenson
A
, et al
.
Virtual residency interviews: applicant perceptions regarding virtual interview effectiveness, advantages, and barriers
.
J Grad Med Educ
.
2022
;
14
(
2
):
224
228
22
Ding
JJ
,
Has
P
,
Hampton
BS
,
Burrell
D
.
Obstetrics and gynecology resident perception of virtual fellowship interviews
.
BMC Med Educ
.
2022
;
22
(
1
):
58

Supplementary data