Many Pediatric Subspecialty Fellows Are Not Ready to Graduate From Fellowship in 2 Years

Pediatric subspecialty fellowship affords physicians the opportunity to develop clinical expertise in a specific area while also developing scholarly proficiency. Subspecialty certification occurs when the trainee successfully completes all aspects of accredited training and passes their subspecialty-specific certifying examination. Certification signifies that an individual has demonstrated competence to practice safe, evidence-based clinical care in that subspecialty and that they possess the knowledge and skills to inform their own practice and to improve the future of pediatric subspecialty care. Except for pediatric hospital medicine (PHM), pediatric subspecialty fellowships with American Board of Pediatrics (ABP) certification are 3 years in duration and include extensive clinical training along with a core scholarly curriculum. The ABP also requires the generation of a scholarly work product as one criterion for Board certification.^1,2 Although many fellowship graduates do not pursue a research career, competence in the critical evaluation of medical literature to inform evidence-based clinical care is essential.² 

In light of a threatened pediatric subspecialty workforce, there has been a call for creative solutions to increase the number of physicians entering pediatric subspecialties.^3,4 One such proposal includes the reduction of subspecialty training from 3 to 2 years, especially for individuals not planning to pursue a physician-scientist career path.³ Support for this notion comes from the 2-year PHM fellowship experience as well as from internal medicine and family medicine, whose programs and certification requirements allow for 2 years of fellowship training in some areas. The ABP has stated that they would consider proposals for altered duration of pediatric subspecialty training if the subspecialty supports the notion and if a competence-based framework with a measurement component is used to ensure the clinical competence of trainees, such as Entrustable Professional Activities (EPAs) and their associated supervision level ratings.^2,5,6 

EPAs, developed by the pediatric community with guidance by the ABP, are the clinical, professional, and scholarly activities deemed necessary for a physician to provide safe, effective, and evidence-based clinical care while also advancing the field of pediatric care through research, education, advocacy, and/or informed policy. The subcompetency milestones, developed by the Accreditation Council for Graduate Medical Education (ACGME), describe specific observable clinical behaviors and attributes associated with a physician’s development from novice to expert clinician.⁷ Each subspecialty has a unique set of EPAs as well as shared EPAs common to all pediatric subspecialties, which collectively define the practice of a subspecialty.⁸ The EPA level of supervision rating scales assess the amount of supervision a trainee requires along a continuum of entrustment from observation only to unsupervised practice.⁹ As would be expected, there is a strong direct relationship between milestone level and EPA supervision rating.¹⁰ Minimum EPA supervision level thresholds expected for fellowship graduation have previously been reported by fellowship program directors (FPDs) (Table 1).¹¹ Importantly, these thresholds suggest that some graduated fellows might need ongoing supervision of varying degrees after the completion of training.¹² 

The Association of Pediatric Program Directors’ Subspecialty Pediatrics Investigator Network (APPD SPIN), a medical education research network including representatives from all 15 pediatric subspecialties with primary ABP certification,¹³ conducted a study to determine trainee EPA supervision level ratings after 1, 2, and 3 years of fellowship. Using the thresholds for fellowship graduation previously identified by FPDs,¹¹ we sought to determine the percentage of fellows meeting these thresholds after 2 and 3 years of training.

The APPD SPIN Steering Committee includes 1–2 representatives from each pediatric subspecialty. Representatives were tasked with recruiting fellowship programs from all subspecialties except for PHM, which was not yet accredited, to a longitudinal assessment of trainees from fall 2018 through spring 2022. Recruitment methods varied by subspecialty and included emails, presentations in medical educator forums and society meetings, and messages posted on fellowship discussion platforms. Institutional review board (IRB) approval was obtained from each participating site with the University of Utah serving as the lead IRB (#00114958).

Clinical Competency Committees (CCCs) were asked to provide a supervision level rating for each fellow for the Pediatric Common Subspecialty and subspecialty-specific EPAs⁸ twice per year at the time of the ACGME milestone reporting. Concurrently, we asked FPDs to rate fellows’ required level of supervision for the scholarship EPA.^6,8 We provided no instruction regarding how CCCs or FPDs should assign the ratings or any faculty development concerning EPAs or their associated level of supervision scales. Validity evidence for the Pediatric Common Subspecialty EPA supervision level rating scales has previously been published.⁹ Designed to be intuitive, these ordinal 5-level scales indicate whether a trainee needs direct, indirect, or no supervision with case complexity as a variable in determining the level of supervision needed for some EPAs. Rating scales created for the subspecialty-specific EPAs use a similar format; validity evidence for these scales has been presented.¹⁴ 

Within 2 months of each CCC meeting, FPDs asked fellows to self-assess their level of supervision for each EPA using the same data collection instruments used by the CCC and the FPD. For this self-assessment, we asked FPDs to forward a specific survey link to each fellow in their program. FPDs were unaware which fellows did or did not provide ratings, and we did not provide any fellow-specific data to the program.

The anonymity of trainees was ensured by creating a unique participant identifier number using an algorithm developed by the APPD Longitudinal Educational Assessment Research Network.¹⁵ Once this identifier was created, we provided links to the online data collection instruments. In the survey instrument, we presented the title of the EPA, the specific functions of the EPA, and the associated level of supervision scale. Details about the online data collection tools have been previously described.⁹ 

For this analysis, we included data from the spring cycles because this would coincide with the supervision level at the time of fellowship completion. We included only those Pediatric Common Subspecialty EPAs in which the activities involved direct patient care: Handover, Consultation, and Lead Team (ie, “Common Clinical”; Table 1). We collected the trainee’s year of fellowship and institution. For dual-trained fellows, we used the subspecialty and year of training they were in based on milestones reported to the ACGME during the period of evaluation. We also asked the CCC to self-report their understanding of EPAs (unfamiliar, basic, in-depth, or expert).

Minimum EPA supervision level ratings deemed necessary for fellowship graduation were previously identified for the Common Clinical and subspecialty-specific EPAs through a survey of FPDs conducted by APPD SPIN (Table 1).¹¹ We calculated the proportion of fellows achieving these graduation thresholds at the end of their second and third years of training for the 3 Common Clinical EPAs and their subspecialty-specific EPAs. We also determined the proportion of fellows achieving graduation readiness if they were allowed to not achieve 1 or 2 supervision level thresholds for these EPAs. We performed a similar analysis for the fellow self-assessment ratings and compared these proportions with the CCC data using a Cox proportion hazard model accounting for subspecialty. We evaluated the proportion of fellows meeting the minimum scholarship EPA supervision level rating necessary for graduation. To examine the effect of CCC understanding of EPAs on reported fellow graduation readiness, we used a Cox proportion hazard model accounting for subspecialty and program. For this analysis, we combined the data into 2 groups, those CCCs who self-reported their understanding of EPAs as in-depth or good vs those who indicated basic or unfamiliar EPA understanding. Data analysis used R (version 4.2; R Core Team).

Table 2 shows the total number of second- and third-year fellows for whom CCCs reported EPA level of supervision ratings as well as the number of fellows who provided self-reported supervision ratings. CCC EPA assessments of 3009 fellows were provided, of whom 75 were dual-trained. The table also illustrates the percentage of fellows represented in this sample in relation to the total number of fellows enrolled in fellowships nationally. The number of fellows for whom FPDs provided supervision level ratings for the scholarship EPA was similar to those evaluated by CCCs (data not shown).

Figure 1A depicts the proportion of fellows meeting the minimum EPA supervision level thresholds expected for graduation for all Common Clinical and subspecialty-specific EPAs after 2 and 3 years of training as reported by CCCs. The percentage of fellows meeting these thresholds after 2 years of training ranged from a high of 39% in adolescent medicine to a low of 15% in cardiology. After 3 years of fellowship, more than 90% of fellows in most subspecialties met the supervision level thresholds expected for graduation, with cardiology (89%) and hematology-oncology (86%) being the exceptions.

When we analyzed the data examining the proportion of fellows achieving the minimum level of supervision thresholds for graduation for all but 1 Common Clinical or subspecialty-specific EPA, (Figure 1B), the proportion of fellows deemed ready to graduate after 2 years increased across subspecialties, although it was still below 50% (range: low of 25% in cardiology to high of 49% in pulmonology). After 3 years, 95%–100% of trainees met graduation thresholds across subspecialties. When we evaluated graduation readiness allowing for 2 Common Clinical or subspecialty-specific EPA thresholds to not be achieved, the proportion of fellows ready to graduate at 2 years increased further with the highest percentages in pulmonology (58%), rheumatology (51%), and infectious diseases (51%) and the lowest in child abuse, emergency medicine, endocrinology, and neonatology (all 38%; Figure 1C). In this scenario, after 3 years at least 97% of fellows were deemed ready to graduate across subspecialties.

Figure 2 shows the proportion of fellows meeting the minimum Common Clinical and subspecialty-specific EPA supervision level thresholds expected for graduation based on fellow self-assessment as compared with CCC ratings. After 2 and 3 years of fellowship, the proportions meeting graduation thresholds were similar between CCC and fellow assessments.

The impact of CCC understanding of EPAs on their assessment of fellow readiness to graduate is shown in Figure 3, comparing those CCCs who reported a high understanding (in-depth or good) with those indicating a low understanding (basic or unfamiliar). Across all subspecialties, CCC understanding of EPAs had little impact on the proportion of fellows they deemed ready to graduate.

The proportion of fellows who met the minimum supervision level threshold expected for graduation for the scholarship EPA is shown in Figure 4. The minimum graduation level of supervision threshold for this EPA is level 2, “trusted to develop and conduct scholarly activities with direct oversight and frequent coaching.”⁶ After 2 years of training, the proportion of fellows meeting this threshold was similar across all subspecialties (64%–68%). After 3 years, nearly all fellows met the minimum supervision level threshold for graduation.

In 2014, the ABP convened the Subspecialty Clinical Training and Certification (SCTC) Initiative to examine “…the current model of pediatrics subspecialty fellowship training and certification with emphasis on competency-based clinical training and with recommending changes in the current requirements, if warranted.”² Included in the recommendations was a process by which the ABP would consider modifications for subspecialty training. These included the need to have a framework for competency assessment, with EPAs being a potential framework, and a measurement component to determine the outcomes of training and readiness for practice. In this study, we used EPA supervision level thresholds to determine fellow readiness for graduation after 2 and 3 years of subspecialty training. As pediatric subspecialty fellowships are currently structured, we found that the level of supervision ratings from CCCs, FPDs, and fellows themselves indicated that most fellows did not meet graduation thresholds after 2 years of training. Although this proportion increased as fewer clinical EPA supervision level thresholds were met, the need for direct supervision upon fellowship completion is not acceptable for patient safety and optimal care.

It can be argued that CCCs assigned the level of supervision rating after 2 years of training knowing that the fellow had an additional year of fellowship. However, we found that fellow self-assessments were consistent with those of the CCCs, and most fellows indicated not feeling ready for unsupervised clinical practice after 2 years of training. Although CCC ratings might have been influenced by knowing the fellow has an additional year of training, it is less likely that this would have affected fellows’ self-assessment. In addition, even though many fellowships have not fully integrated EPAs into their program of assessment, CCC understanding of EPAs did not markedly impact our findings.

We used the minimum supervision level expected for graduation identified by FPDs because, ultimately, it is the FPD, with input from the CCC, who verifies that a trainee is competent to graduate. We included fellow supervision ratings in which the FPD was or was not a member of the CCC because we previously showed a minimal effect of FPD membership on the CCC on these ratings.¹⁶ It should be emphasized that these graduation thresholds are low and indicate the need for continued clinical supervision after training.¹² Of the 68 clinical common and subspecialty-specific EPAs, the most frequent minimum level of supervision threshold identified for graduation is level 3 (57%), in which direct supervision is required for some complex cases, followed by level 4 (37%) and level 2 (6%; Table 1). No supervision level threshold for graduation was set at level 5, unsupervised practice. As previously described, at the time of establishing these minimum thresholds for graduation, FPDs were comfortable graduating some fellows needing either direct or indirect supervision for some clinical EPAs.¹¹ Even with such low graduation thresholds, we demonstrated a low proportion of fellows meeting them after 2 years of training.

The National Academies of Sciences, Engineering, and Medicine recently recommended reducing pediatric subspecialty training to 2 years as one component to augment the pathway of physicians pursuing a subspecialty to increase the workforce nationally.³ This sentiment has been echoed by subspecialists and trainees alike.^4,17 Experts have suggested that such a reduction in duration of training would increase the lifetime earning potential of pediatric subspecialists and provide another reason for more trainees to choose this path, because trainee debt may be a contributor to their subspecialty decision-making.^18,19 The only current model of 2-year fellowship training with ABP certification is PHM. However, it is argued that PHM was uniquely set up for a 2-year fellowship given the amount of inpatient care during pediatric residency training, allowing time in PHM fellowship to hone existing skills, deepen prior knowledge, and focus on scholarship.

Currently, the ACGME’s subspecialty program requirement is for fellows to engage in a minimum of 12 months of scholarly/research time.²⁰ The ABPs SCTC Initiative affirmed the importance of the scholarly requirement stating, “…scholarship is a core value in subspecialty training because scholarly activity serves to teach fellows to be critical thinkers and evidence-based practitioners and to analyze, interpret, and apply research evidence at the point of care.”² In a survey of FPDs, Abramson et al reported that 25% of fellowship programs included at least 21 months free of or with reduced clinical time to allow trainees to engage in scholarship, thereby providing 12 months of clinical time (plus 3 months for vacation).²¹ Notably, in 7 subspecialties, fellows have 8 months or more of clinical time in their first year of training with very few programs providing substantial clinical time in the third year of fellowship (unpublished data). Reducing the length of fellowship training would require curricular restructuring, and it is unknown what impact such modification would have on the pediatric physician-scientist pathway. There may be other unintended consequences of consolidating the current 3 years of fellowship into 2. For example, given concerns regarding physician trainee burnout, consolidating clinical time may exacerbate existing burnout trends.²² 

The ACGME recently revised the pediatric residency program requirements, and this could impact fellow preparedness to enter subspecialty training. A survey conducted by Boyer et al indicated that many FPDs believe that pediatric residents are not sufficiently prepared to enter subspecialty training.²³ The effect of the revised training requirements will need to be evaluated once the changes are implemented.

Of note, this study was partially conducted during the height of the COVID-19 pandemic. In a separate analysis, we demonstrated that fellows graduating during the pandemic had higher EPA supervision level ratings than those graduating before the pandemic.²⁴ This suggests fellows would be more ready to graduate during this time.

This study has limitations. As noted above, the 2-year EPA supervision level ratings may be biased by the fact that CCCs know that the current fellowship training model is 3 years in duration, so there is no expectation of fellow graduation readiness after 2 years. That being said, not all trainees were meeting graduation thresholds even at 3 years, and the thresholds used in this investigation were not “unsupervised practice.” Another limitation was the lack of structured and standardized training provided to FPDs, CCCs, and fellows regarding the use of the EPA rating scales. However, the proportions did not differ based on the CCC understanding of EPAs, and a previous study suggested that, in the subspecialties, substantial training was not required.²⁵ FPDs could have discussed the CCC EPA supervision ratings with their fellows, but they would have had to print the online forms to do so. FPDs were also unaware of which fellows completed the self-assessments. Individuals who felt comfortable using EPAs for fellow assessment may or may not be using them in the same way as other assessors, but the accumulated strong validity evidence for these supervision level scales suggests a shared mental model among most users.^9,14 The number of fellow self-assessments was low, but the close correlation seen with CCC assessments is notable. Finally, it is possible that the fellows and training programs not participating in this longitudinal evaluation were different, although the sample represented programs of varying sizes located in all regions of the United States.

Our study indicates that as fellowships are currently structured, many pediatric subspecialty fellows would not be considered competent to graduate after 2 years of training using an EPA assessment framework. If shortening pediatric subspecialty fellowship training is sought, substantial fellowship curricular changes and modification of certification requirements would be needed to ensure that all trainees are ready to provide safe and evidence-based care after 2 years of training.

We thank Marzia Hazara, Beth King, Dennis West, Anna Sombrio, and Avery Ulrich for their help with this project.

ABP

American Board of Pediatrics

ACGME

Accreditation Council for Graduate Medical Education

APPD SPIN

Association of Pediatric Program Directors’ Subspecialty Pediatrics Investigator Network

CCC

Clinical Competency Committee

EPA

Entrustable Professional Activity

FPD

fellowship program director

IRB

institutional review board

PHM

pediatric hospital medicine

SCTC

Subspecialty Clinical Training and Certification.