BACKGROUND AND OBJECTIVES

Entrustable Professional Activities (EPAs) were developed to assess pediatric fellows. We previously showed that fellowship program directors (FPDs) may graduate fellows who still require supervision. How this compares with their expectations for entrustment of practicing subspecialists is unknown.

METHODS

We surveyed US FPDs in 14 pediatric subspecialties through the Subspecialty Pediatrics Investigator Network between April and August 2017. For each of 7 common pediatric subspecialty EPAs, we compared the minimum level of supervision that FPDs required for graduation with the level they expected of subspecialists for safe and effective practice using the Friedman rank sum test and paired t test. We compared differences between subspecialties using linear regression.

RESULTS

We collected data from 660 FPDs (response rate 82%). For all EPAs, FPDs did not require fellows to reach the level of entrustment for graduation that they expected of subspecialists to practice (P < .001). FPDs expected the least amount of supervision for the EPAs consultation and handovers. Mean differences between supervision levels for graduation and practice were smaller for clinical EPAs (consultation, handovers, lead a team) when compared with nonclinical EPAs (quality improvement, management, lead the profession and scholarship; P = .001) and were similar across nearly all subspecialties.

CONCLUSIONS

Fellowship graduates may need continued development of clinical and nonclinical skills in their early practice period, underscoring a need for continued assessment and mentoring. Graduation readiness must be based on clear requirements, with alignment of FPD expectations and regulatory standards, to ensure quality care for patients.

What’s Known on This Subject:

Entrustable Professional Activities and supervision scales are used to assess trainees. Program directors may graduate fellows who require supervision, but how that compares with their perceptions about entrustment of practicing subspecialists is not known.

What This Study Adds:

For the common pediatric subspecialty Entrustable Professional Activities, program directors report that they would graduate fellows who do not meet the level of entrustment expected of practicing subspecialists, reinforcing the need for assessment and mentoring of early career subspecialists.

Medical education has transitioned over the last 15 years to an outcome-based framework, and Entrustable Professional Activities (EPAs) have emerged as an important tool to assess physicians.1,2  EPAs are units of observable work and describe the important routine tasks expected of a physician in practice.3  The concept of “entrustable” aligns with the ability of the individual to perform the activity safely and effectively without supervision.2,3  EPAs are used in some training programs for assessment and curriculum development.48  In US pediatric fellowship training, 7 common EPAs that share content relevant to all pediatric subspecialties have been developed (Table 1).9,10  We previously developed supervision scales for these EPAs10,11  and obtained strong validity evidence for their use in assessing the competency of pediatric fellows.11 

TABLE 1

Common Pediatric Subspecialty EPAs and Level of Supervision Scales

EPALevel of Supervision
Apply public health principles and QI methods to improve population health 1. Trusted to observe only 
 2. Trusted to contribute with direct supervision and coaching as a member of a collaborative effort to improve care at the institutional level 
 3. Trusted to contribute without direct coaching as a member of a collaborative effort to improve care at the institutional level 
 4. Trusted to lead collaborative efforts to improve care for populations and systems at the institutional level 
 5. Trusted to lead collaborative efforts to improve care at the level of populations and systems at the regional and/or national level 
Provide consultation to other health providers caring for children and adolescents and refer patients requiring further consultation to other subspecialty providers if necessary (consultation) 1. Trusted to observe only 
 2. Trusted to execute with direct supervision and coaching 
 3. Trusted to execute with indirect supervision and discussion of information conveyed for selected simple and complex cases 
 4. Trusted to execute with indirect supervision and may require discussion of information conveyed but only for selected complex cases 
 5. Trusted to execute independently without supervision 
Contribute to the fiscally sound, equitable and collaborative management of a health care workplace (management) 1. Trusted to observe only 
 2. Trusted to perform with direct supervision and coaching with supervisor verifying work product for accuracy 
 3. Trusted to perform with supervisor serving as a consultant for all tasks 
 4. Trusted to perform with supervisor serving as a consultant but only for complex tasks 
 5. Trusted to perform without supervision 
Facilitate handovers to another health care provider either within or across settings (handovers) 1. Trusted to observe only 
 2. Trusted to execute with direct supervision and coaching 
 3. Trusted to execute with indirect supervision with verification of information after the handover for selected simple and complex cases 
 4. Trusted to execute with indirect supervision with verification of information after the handover for selected complex cases 
 5. Trusted to execute independently without supervision 
Lead an interprofessional health care team (lead team) 1. Trusted to participate only 
 2. Trusted to lead with direct supervision and coaching 
 3. Trusted to lead with supervisor occasionally present to provide advice 
 4. Trusted to lead without supervisor present but requires coaching to improve member and team performance 
 5. Trusted to lead without supervision to improve member and team performance 
Lead within the subspecialty profession (lead profession) 1. Trusted to observe only 
 2. Trusted to contribute to advocacy and public education activities for the subspecialty profession with direct supervision and coaching at the institutional level 
 3. Trusted to contribute to advocacy and public education activities for the subspecialty profession with indirect supervision at the institutional level 
 4. Trusted to mentor others and lead advocacy and public education activities for the subspecialty profession at the institutional level 
 5. Trusted to lead advocacy and public education activities for the subspecialty profession at the regional and/or national level 
Engage in scholarly activities through discovery, application, and dissemination of new knowledge (scholarship) 1. Trusted to assist in scholarly activities with direct supervision 
 2. Trusted to develop and conduct scholarly activities with direct oversight and frequent coaching 
 3. Trusted to develop and conduct scholarly activities with occasional coaching 
 4. Trusted to lead scholarly activities and provide coaching to others at the local institution 
 5. Trusted to lead and coach others in scholarly activities through national and/or international networks 
EPALevel of Supervision
Apply public health principles and QI methods to improve population health 1. Trusted to observe only 
 2. Trusted to contribute with direct supervision and coaching as a member of a collaborative effort to improve care at the institutional level 
 3. Trusted to contribute without direct coaching as a member of a collaborative effort to improve care at the institutional level 
 4. Trusted to lead collaborative efforts to improve care for populations and systems at the institutional level 
 5. Trusted to lead collaborative efforts to improve care at the level of populations and systems at the regional and/or national level 
Provide consultation to other health providers caring for children and adolescents and refer patients requiring further consultation to other subspecialty providers if necessary (consultation) 1. Trusted to observe only 
 2. Trusted to execute with direct supervision and coaching 
 3. Trusted to execute with indirect supervision and discussion of information conveyed for selected simple and complex cases 
 4. Trusted to execute with indirect supervision and may require discussion of information conveyed but only for selected complex cases 
 5. Trusted to execute independently without supervision 
Contribute to the fiscally sound, equitable and collaborative management of a health care workplace (management) 1. Trusted to observe only 
 2. Trusted to perform with direct supervision and coaching with supervisor verifying work product for accuracy 
 3. Trusted to perform with supervisor serving as a consultant for all tasks 
 4. Trusted to perform with supervisor serving as a consultant but only for complex tasks 
 5. Trusted to perform without supervision 
Facilitate handovers to another health care provider either within or across settings (handovers) 1. Trusted to observe only 
 2. Trusted to execute with direct supervision and coaching 
 3. Trusted to execute with indirect supervision with verification of information after the handover for selected simple and complex cases 
 4. Trusted to execute with indirect supervision with verification of information after the handover for selected complex cases 
 5. Trusted to execute independently without supervision 
Lead an interprofessional health care team (lead team) 1. Trusted to participate only 
 2. Trusted to lead with direct supervision and coaching 
 3. Trusted to lead with supervisor occasionally present to provide advice 
 4. Trusted to lead without supervisor present but requires coaching to improve member and team performance 
 5. Trusted to lead without supervision to improve member and team performance 
Lead within the subspecialty profession (lead profession) 1. Trusted to observe only 
 2. Trusted to contribute to advocacy and public education activities for the subspecialty profession with direct supervision and coaching at the institutional level 
 3. Trusted to contribute to advocacy and public education activities for the subspecialty profession with indirect supervision at the institutional level 
 4. Trusted to mentor others and lead advocacy and public education activities for the subspecialty profession at the institutional level 
 5. Trusted to lead advocacy and public education activities for the subspecialty profession at the regional and/or national level 
Engage in scholarly activities through discovery, application, and dissemination of new knowledge (scholarship) 1. Trusted to assist in scholarly activities with direct supervision 
 2. Trusted to develop and conduct scholarly activities with direct oversight and frequent coaching 
 3. Trusted to develop and conduct scholarly activities with occasional coaching 
 4. Trusted to lead scholarly activities and provide coaching to others at the local institution 
 5. Trusted to lead and coach others in scholarly activities through national and/or international networks 

Previous studies have suggested that graduates of pediatric training programs may not achieve the competence necessary for unsupervised practice in their new positions.1214  In US pediatric residency programs, residents did not uniformly achieve the competence or entrustment necessary for unsupervised practice at graduation.12,13  In a survey of recent fellowship graduates, 12% believed they did not receive appropriate clinical training to prepare for their first position, develop clinical independence or manage the complexity of procedures and patient care.15  Self-perceived deficiencies in clinical care and practice management were reported by early career pediatric hospitalists.14  Burnout has also been associated with self-perceived gaps in nonclinical competencies, such as practice management.16 

We previously showed that US fellowship program directors (FPDs) would graduate fellows who required supervision for the common subspecialty EPAs, requiring the least amount of supervision for clinically oriented EPAs, with few differences across the subspecialties.17  How that relates to their expectations for practicing subspecialists for safe and effective outcomes is unknown. Through the Subspecialty Pediatrics Investigator Network (SPIN),18  we compared FPD beliefs about the minimum level of supervision expected for a subspecialist in practice with their requirements for graduation. We hypothesized that there would be gaps between the 2 supervision levels, which would be smallest for EPAs directly related to clinical care.

SPIN is a collaboration between the American Board of Pediatrics (ABP), Association of Pediatric Program Directors (APPD) Fellowship Committee, APPD Longitudinal Educational Assessment Research Network, and Council of Pediatric Subspecialties, and it includes 1 to 2 representatives from each of the pediatric subspecialties with ABP certification.18  As described in our previous report,17  we sent a confidential survey using LimeSurvey between April 2017 and August 2017 to FPDs of the Accreditation Council for Graduate Medical Education (ACGME)–accredited US fellowship programs for the 14 subspecialties with ABP certification at the time of the study. After a literature review,911 ,19,20  we designed a survey that was reviewed and edited by SPIN representatives, all of whom have graduate medical education experience (Supplemental Fig 3). SPIN representatives chose 2 to 4 faculty, previously serving as FPDs in each subspecialty, to pilot-test the survey. Their feedback was used to modify the survey and develop the final version. The study was determined to be exempt by the University of Illinois at Chicago Institutional Review Board.

We set a goal to have at least 75% of all FPDs in each subspecialty participate. We sent the original invitation and 4 reminders by e-mail to prospective participants. SPIN representatives then contacted and requested all nonresponders via e-mail to complete the survey. Representatives had access to FPD names and e-mail addresses but not to the participant’s responses.

The 7 common pediatric subspecialty EPAs and supervision scales are shown in Table 1. The scales are based on 5 levels of supervision, with higher levels meaning less supervision. In the survey introduction, we described the concept of EPAs and the functions of each, entrustability of a subspecialist (when a subspecialist should be able to perform most of the functions defined by the EPA, resulting in a safe and effective outcome), and assignment of the supervision level (based on trust and not necessarily on direct observation of all associated functions; Supplemental Fig 3). For each of the EPAs, we asked the following:

  1. “For this EPA, what do you believe is the minimum level of supervision a fellow must achieve to successfully complete fellowship?”

  2. “For this EPA, if a fellow did not achieve at least this minimal level of supervision, would you still allow him or her to graduate?”

  3. “For this EPA, what is the lowest level in which you would consider that a practicing subspecialist (and not necessarily a trainee) should be able to perform most of the activities described above resulting in a safe and effective outcome?”

Data are described as n (%) for categorical survey items and median and percentiles. We previously described the minimum level of supervision for graduation as that reported by the FPDs to successfully complete the fellowship.17  However, if the FPD indicated that they would allow the fellow to graduate even if the fellow did not achieve that level of supervision, it was set at 1 level below (adjusted minimum level of supervision). On the basis of consensus of SPIN representatives and other experts in graduate medical education, we defined the minimum level of supervision for graduation and practice as the level that at least 80% of FPDs would not drop below, a boundary that aligns with previously published data.13,17  We tested for differences in the distribution of responses to the graduation and practice items for each EPA using nonparametric within-subject Friedman rank sum tests. For comparison, we divided the EPAs into clinical (consultation, handovers, and lead team) and nonclinical (quality improvement [QI], management, lead profession, and scholarship). We compared mean differences between graduation and practice responses for each EPA and for clinical versus nonclinical EPAs using paired t tests and linear regression models fitted to the difference scores with a fixed effect of subspecialty. We used pediatric endocrinology as the reference level for the subspecialty effect because its mean FPD responses were nearest the overall mean across EPAs. A P value <.05 was considered significant. Data analyses were conducted by using R 3.6 (R Core Team, Vienna, Austria) and lme4 package.21 

We collected data from 660 of 802 FPDs with an overall response rate of 82%. All subspecialties, except for infectious diseases, met the goal of 75% for participation (Supplemental Table 3). We previously described characteristics of the FPDs, programs, and minimum expected supervision levels for graduation.17  The adjusted minimum level of supervision that FPDs deemed necessary for fellows to successfully graduate was compared with the level they expected for subspecialists to practice for each EPA in Fig 1. The unadjusted graduation data for graduation are included in Supplemental Table 4. As we previously showed,17  to graduate, the majority of FPDs did not require fellows to be trusted to practice without supervision or coaching for any EPA. To practice as a subspecialist, FPDs expected the least amount of supervision for the EPAs consultation and handovers; however, even for those EPAs, a significant number of FPDs (49% and 35%, respectively) did not report unsupervised practice as being necessary for safe and effective outcomes. For the other EPAs, the majority of FPDs did not regard unsupervised practice as necessary for a subspecialist to practice.

FIGURE 1

Minimum level of supervision reported by FPDs to be required for graduation and for entrustment of subspecialists for safe and effective practice for each of the common pediatric subspecialty EPAs. Graduation levels are adjusted for FPD willingness to graduate fellows even if they did not meet the minimum level. The boxplot shows median (dark line), interquartile range (box), and one-and-one-half interquartile ranges from the box (whiskers), with outliers plotted as points. Light gray represents graduation; dark gray represents entrustment for safe and effective practice by a subspecialist (N = 660 FPDs).

FIGURE 1

Minimum level of supervision reported by FPDs to be required for graduation and for entrustment of subspecialists for safe and effective practice for each of the common pediatric subspecialty EPAs. Graduation levels are adjusted for FPD willingness to graduate fellows even if they did not meet the minimum level. The boxplot shows median (dark line), interquartile range (box), and one-and-one-half interquartile ranges from the box (whiskers), with outliers plotted as points. Light gray represents graduation; dark gray represents entrustment for safe and effective practice by a subspecialist (N = 660 FPDs).

Close modal

The adjusted minimum level of supervision to graduate a fellow and for a subspecialist to practice for each EPA are shown in Fig 2. For graduation, FPDs deemed indirect supervision acceptable for patient care EPAs such as consultation (level 3) and handovers (level 4), but would allow fellows to graduate who required even more supervision for the other EPAs. For practice, FPDs expected the least amount of supervision for consultation and handovers. However, even for these EPAs, FPDs found it acceptable that practicing subspecialists may require indirect supervision for select complex patients (level 4). For the remaining EPAs, FPDs believed that practicing subspecialists may require consultation for all tasks and occasional advice (management and lead team) or even direct supervision (QI, lead profession, and scholarship).

FIGURE 2

Minimum supervision levels for graduation and for entrustment for safe and effective practice for which at least 80% of FPDs would not want a lower level for the common pediatric subspecialty EPAs. Graduation levels are adjusted for FPD willingness to graduate fellows even if they did not meet the minimum level. Open bars indicate supervision level for graduation and gray bars indicate that for safe and effective practice (N = 660 FPDs).

FIGURE 2

Minimum supervision levels for graduation and for entrustment for safe and effective practice for which at least 80% of FPDs would not want a lower level for the common pediatric subspecialty EPAs. Graduation levels are adjusted for FPD willingness to graduate fellows even if they did not meet the minimum level. Open bars indicate supervision level for graduation and gray bars indicate that for safe and effective practice (N = 660 FPDs).

Close modal

For all EPAs, FPDs did not require fellows to reach the level of entrustment at graduation that they expected of subspecialists to practice for safe and effective outcomes (Fig 1, Table 2). The smallest differences between the levels were observed for consultation and handovers EPAs whereas the largest differences were for lead profession, management, and lead team. The mean differences between graduation and practice were smaller for clinical EPAs when compared with nonclinical EPAs, even when adjusted for subspecialty (P = .001). Compared with pediatric endocrinology (Table 2), the magnitude of the difference between graduation and practice was significantly greater for pediatric pulmonology for management (P = .049) and handovers (P = .03) EPAs but less for pediatric nephrology for QI (P = .03), pediatric emergency medicine for consultation (P = .02), and critical care and neonatology for lead team (P = .01 and .02, respectively). There were no other significant differences among the other subspecialties for the other EPAs (P > .05).

TABLE 2

Statistically Significant Mean Differences Between the Minimum Levels of Supervision That FPDs Report That They Require for Graduation and Expect for Practice for Each of the Common Pediatric Subspecialty EPAs by Subspecialty, Compared With Pediatric Endocrinology

EPAOverall Mean Difference Between Practice and Graduation (95% CI)Mean Differences Between Minimum Level of Supervision for Practice and Graduation for Subspecialties Significantly Different From Pediatric Endocrinology (95% CI)a
QI 0.59 (0.53 to 0.65) Nephrology: 0.34 (0.11 to 0.57) 
Consultation 0.38 (0.32 to 0.44) Emergency medicine: 0.09 (−0.12 to 0.29) 
Management 0.95 (0.87 to 1.03) Pulmonology: 1.33 (1.03 to 1.62) 
Handovers 0.35 (0.29 to 0.40) Pulmonology: 0.59 (0.37 to 0.80) 
Lead team 0.79 (0.72 to 0.87) Critical care: 0.51 (0.26 to 0.75) 
 — Neonatology: 0.56 (0.35 to 0.77) 
Lead profession 1.02 (0.95 to 1.09) — 
Scholarship 0.55 (0.48 to 0.61) — 
EPAOverall Mean Difference Between Practice and Graduation (95% CI)Mean Differences Between Minimum Level of Supervision for Practice and Graduation for Subspecialties Significantly Different From Pediatric Endocrinology (95% CI)a
QI 0.59 (0.53 to 0.65) Nephrology: 0.34 (0.11 to 0.57) 
Consultation 0.38 (0.32 to 0.44) Emergency medicine: 0.09 (−0.12 to 0.29) 
Management 0.95 (0.87 to 1.03) Pulmonology: 1.33 (1.03 to 1.62) 
Handovers 0.35 (0.29 to 0.40) Pulmonology: 0.59 (0.37 to 0.80) 
Lead team 0.79 (0.72 to 0.87) Critical care: 0.51 (0.26 to 0.75) 
 — Neonatology: 0.56 (0.35 to 0.77) 
Lead profession 1.02 (0.95 to 1.09) — 
Scholarship 0.55 (0.48 to 0.61) — 

N = 660. CI, confidence interval; —, not applicable.

a

Pediatric endocrinology had closest to average differences and served as reference subspecialty; P < .05.

In our study, FPDs did not require fellows to reach the level of entrustment at graduation that they expected of subspecialists to practice for safe and effective outcomes for any of the common pediatric subspecialty EPAs. We previously showed that FPDs did not require fellows to be entrusted for unsupervised practice at graduation for these EPAs.17  By adjusting for FPD willingness to graduate fellows even if they did not achieve the minimum level of supervision, the amount of supervision deemed necessary for new graduates was even greater than in our previous report. Our findings are consistent with data on general pediatrics EPAs demonstrating that expectations for unsupervised practice at the time of graduation vary by EPAs and programs.13  Additionally, they align with the variability in the reported ACGME milestone levels that fellows achieve at graduation.22  The fact that fellowship graduates may still require supervision conflicts with requirements that they be prepared for unsupervised practice.23  Our findings, especially if supported by actual fellow performance data, have important implications for decision-making around future fellowship graduation requirements, the mentorship and support structure for new graduates, and structured learning and assessment as part of maintenance of certification. Educational leaders and regulatory bodies need to reevaluate fellowship program graduation standards to align the structure and curricula of fellowship programs with the desired outcomes outlined in the EPA framework. This reevaluation should be viewed through the lens of lifelong learning so that future approaches are developed in a way to best facilitate transitions between fellowship and practice to optimize both education and patient outcomes.

FPDs expected the least amount of supervision for graduation and subspecialty practice for EPAs that involve patient care. The gap between these expectations was also smallest for these EPAs, which likely reflects prioritization of what is most important for the immediate needs of patients and their safety. However, even for these EPAs, FPDs reported that they would graduate fellows who do not meet the minimum levels of entrustment required for safe and effective practice. The gap was smallest for activities such as consultation and handovers, but it was larger for leading a team, despite the known importance of this skill to patient outcomes.24 

For nonclinical EPAs, FPDs reported allowing graduation of fellows who require direct supervision and coaching, which exceeds expectations for oversight of practicing subspecialists for effective outcomes. The need for continued coaching is not surprising given the years of experience that are needed to attain expertise in these activities after graduation. Although there may be some debate as to whether these nonclinical activities should be included in the EPA schema, they were determined by a consensus of subspecialty experts to be an important part of the framework because the EPAs facilitate development of a shared mental model of trainee performance. In addition, there is strong validity evidence to support the supervision scales used in this investigation for these nonclinical activities.11 

The magnitude of the gaps between the level of supervision expected for graduation and entrustment as a subspecialist may align with the importance that FPDs place on the activity, outcomes of their training program, or postgraduate careers of their fellows. Additionally, some programs may lack the resources, expertise, time, or opportunity to train fellows sufficiently to levels of unsupervised practice in domains such as QI,25  practice management,26,27  leadership,28,29  and research.30  The gap for each EPA was similar between most subspecialties. The few differences may reflect variability in the importance that those subspecialists place on a given activity. For example, for leading a team, the gap was smaller in critical care and neonatology, perhaps reflecting the importance of teamwork in the critical care environment.24  Similarly, consultation may play a more prominent role in the care delivered by emergency medicine physicians. The reasons for other differences, such as the larger gap in pulmonology for handovers, are unclear. Qualitative studies would be helpful to delineate them.

Our findings support the need for continued, formalized mentorship to continue development of clinical proficiency for recent graduates in their new positions to ensure quality patient care. Graduates who have demonstrated competence within their training programs may need to care for new patient populations within unfamiliar complex systems of health care delivery. In academic centers, training is often provided for administrative tasks (eg, billing) for new physicians, but confirmation of clinical proficiency (eg, review and oversight of initial clinical activities) is variable. Addressing the gaps that new graduates perceive in their competency could minimize the stress and burnout that they experience in the transition process.16  An initial period of supervision similar to that required in training programs, consisting of observation and assessment (with feedback) by supervisors, should be considered until graduates demonstrate competency in their new activities.31  A formalized handover process from training program to practice would facilitate identification of areas in which support of clinical skills should be provided. Identification and training of supervisors to assess and mentor new graduates is also needed, and they should meet regularly and develop individualized plans. Collaboration with the ABP to create learning activities for maintenance of certification can help address important skills gaps.

Our study also underscores the need for mentorship and training for early career subspecialists for professional development in areas such as scholarship, QI, and leadership. Formal mentorship is vital for the career development of physicians in academic medicine,3234  and new graduates should be assigned a mentor. Elements of effective mentoring programs include mentor training, an oversight committee, contracts between mentor and mentees, formal curricula for mentees, adequate funding, and protected time.34  Development of expertise in QI and leadership skills may require additional training either within or outside the institution. Training in QI is offered by a number of organizations,35  including the Institute for Healthcare Improvement, the American Institute for Healthcare Quality, the Academic Pediatric Association, and the American Association of Medical Colleges (AAMC). Most academic health centers provide leadership development programs; however, access may be limited.36  Formal leadership training programs and seminars are offered by professional organizations, such as the AAMC,37  the APPD (eg, Leadership in Educational Academic Development [LEAD]),38  the Academic Pediatric Association, the American Academy of Pediatrics, and individual subspecialty organizations (eg, the American Thoracic Society, the American Gastroenterological Association, the Society of Hospital Medicine). Additionally, the AAMC offers specific leadership programs for early career women and underrepresented minorities.

This investigation demonstrated a gap in skills between FPD expectations of fellow performance at the time of graduation and the skills expected for practice. Additionally, there was variability between FPDs in their expectations because many were willing to graduate fellows below an expected minimum level. One way to close these gaps is to create a shared mental model for what is to be achieved as a fellow and design structured learning and assessment during the early years of maintenance of certification to achieve the skill level needed in practice. Setting standards for EPA-based assessment tools is a complex process that includes the judgment of expert assessors.20  We chose to survey FPDs who are responsible for verifying fellow readiness for graduation, but other stakeholders must be involved in setting these standards39  (eg, division chiefs, chairs, subspecialists in academic and private practice settings, trainees, patients, families, and regulatory organizations) because of the potential impact on the need for assessment and support after graduation.

There are several limitations to our study. We previously collected validity evidence for the use of EPAs for fellow assessment11 ; however, it is possible there was confusion in the FPD construal of the supervision scales and concept of entrustment for subspecialists. For graduation, we asked FPDs to assign the supervision level on the basis of what they would prospectively trust fellows to do, not what they directly observed them to do. Because institutional and accrediting bodies require supervision for these activities, FPDs may overestimate the amount of supervision needed for graduation. We adhered to principles of survey design,40  but it is possible that our questions did not adequately cover the construct of interest. There is potential for self-reporting and nonresponder bias; however, the latter is mitigated by the high response rate. In the survey, we asked about the expected minimum supervision level, which is important in setting standards but does not address FPD expectations for the majority of fellows and practicing subspecialists. It is possible that FPD expectations could have changed since the study was completed; however, that is unlikely, given the lack of intercurrent studies on EPAs in pediatric fellowship programs, modification of assessment tools, or changes in requirements by accrediting organizations. Our ongoing longitudinal EPA study should help to determine if fellow workplace performance during training is consistent with FPD expectations at graduation.

This study has important implications for educators, supervising physicians, and regulatory agencies, suggesting a need to align the expected outcomes of fellowship training programs with the skills required for safe and effective practice. It underscores the importance of setting standards to ensure fellow competence on graduation. EPAs represent one tool for addressing the ongoing learning that needs to occur in the early years of practice to meet expectations for practicing subspecialists and throughout the continuum to both acquire needed skills and maintain them. The period of time for which early career subspecialists require enhanced support is unclear and is an area for potential future research.

Drs Mink and Schwartz conceptualized and designed the study, coordinated and supervised the data collection, analyzed and interpreted the data, and critically revised the manuscript for important intellectual content; Drs Carraccio and Herman conceptualized and designed the study, analyzed and interpreted the data, and critically reviewed the manuscript for important intellectual content; Dr Weiss contributed to the study design, collected, analyzed and interpreted the data, and drafted the initial manuscript; Dr Turner contributed to the study design, collected, analyzed, and interpreted the data, and critically reviewed the manuscript for important intellectual content; Drs Aye, Fussell, Mahan, McGann, Kesselheim, Myers, and Stafford contributed to the study design, collected and interpreted the data, and critically reviewed the manuscript for important intellectual content; Drs Chess, Curran, Dammann, High, Hsu, Pitts, Sauer, and Srivastava contributed to the study design, collected data, and critically reviewed the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported in part by the American Board of Pediatrics Foundation.

COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2021-053258.

The content is solely the responsibility of the authors and does not necessarily represent the official views of the American Board of Pediatrics or the American Board of Pediatrics Foundation.

Mr Turner’s current affiliation is American Board of Pediatrics, Chapel Hill, NC.

     
  • AAMC

    American Association of Medical Colleges

  •  
  • ABP

    American Board of Pediatrics

  •  
  • ACGME

    Accreditation Council for Graduate Medical Education

  •  
  • APPD

    Association of Pediatric Program Directors

  •  
  • EPA

    Entrustable Professional Activity

  •  
  • FPD

    fellowship program director

  •  
  • QI

    quality improvement

  •  
  • SPIN

    Subspecialty Pediatrics Investigator Network

1
ten Cate
O
,
Scheele
F
.
Competency-based postgraduate training: can we bridge the gap between theory and clinical practice?
Acad Med
.
2007
;
82
(
6
):
542
547
2
Carraccio
C
,
Englander
R
,
Gilhooly
J
, et al
.
Building a framework of entrustable professional activities, supported by competencies and milestones, to bridge the educational continuum
.
Acad Med
.
2017
;
92
(
3
):
324
330
3
Ten Cate
O
.
Nuts and bolts of entrustable professional activities
.
J Grad Med Educ
.
2013
;
5
(
1
):
157
158
4
Moore
D
,
Young
CJ
,
Hong
J
.
Implementing entrustable professional activities: the yellow brick road towards competency-based training?
ANZ J Surg
.
2017
;
87
(
12
):
1001
1005
5
Boyce
P
,
Spratt
C
,
Davies
M
,
McEvoy
P
.
Using entrustable professional activities to guide curriculum development in psychiatry training
.
BMC Med Educ
.
2011
;
11
:
96
6
Mulder
H
,
Ten Cate
O
,
Daalder
R
,
Berkvens
J
.
Building a competency-based workplace curriculum around entrustable professional activities: the case of physician assistant training
.
Med Teach
.
2010
;
32
(
10
):
e453
e459
7
Warm
EJ
,
Held
JD
,
Hellmann
M
, et al
.
Entrusting observable practice activities and milestones over the 36 months of an internal medicine residency
.
Acad Med
.
2016
;
91
(
10
):
1398
1405
8
O’Dowd
E
,
Lydon
S
,
O’Connor
P
,
Madden
C
,
Byrne
D
.
A systematic review of 7 years of research on entrustable professional activities in graduate medical education, 2011-2018
.
Med Educ
.
2019
;
53
(
3
):
234
249
9
American Board of Pediatrics
.
Entrustable professional activities for subspecialties
.
Available at: www.abp.org/subspecialty-epas. Accessed May 15, 2021
10
Mink
RB
,
Myers
AL
,
Turner
DA
,
Carraccio
CL
.
Competencies, milestones, and a level of supervision scale for entrustable professional activities for scholarship
.
Acad Med
.
2018
;
93
(
11
):
1668
1672
11
Mink
RB
,
Schwartz
A
,
Herman
BE
, et al;
Steering Committee of the Subspecialty Pediatrics Investigator Network (SPIN)
.
Validity of level of supervision scales for assessing pediatric fellows on the common pediatric subspecialty entrustable professional activities
.
Acad Med
.
2018
;
93
(
2
):
283
291
12
Li
ST
,
Tancredi
DJ
,
Schwartz
A
, et al;
Association of Pediatric Program Directors (APPD) Longitudinal Educational Assessment Research Network (LEARN) Validity of Resident Self-Assessment Group
.
Competent for unsupervised practice: use of pediatric residency training milestones to assess readiness
.
Acad Med
.
2017
;
92
(
3
):
385
393
13
Schumacher
DJ
,
West
DC
,
Schwartz
A
, et al;
Association of Pediatric Program Directors Longitudinal Educational Assessment Research Network General Pediatrics Entrustable Professional Activities Study Group
.
Longitudinal assessment of resident performance using Entrustable Professional Activities
.
JAMA Netw Open
.
2020
;
3
(
1
):
e1919316
14
Librizzi
J
,
Winer
JC
,
Banach
L
,
Davis
A
.
Perceived core competency achievements of fellowship and non-fellowship-trained early career pediatric hospitalists
.
J Hosp Med
.
2015
;
10
(
6
):
373
379
15
Freed
GL
,
Dunham
KM
,
Moran
LM
,
Spera
L
,
McGuinness
GA
,
Stevenson
DK
.
Pediatric subspecialty fellowship clinical training project: recent graduates and midcareer survey comparison
.
Pediatrics
.
2014
;
133
(
suppl 2
):
S70
S75
16
Westerman
M
,
Teunissen
PW
,
Fokkema
JP
, et al
.
The transition to hospital consultant and the influence of preparedness, social support, and perception: a structural equation modelling approach
.
Med Teach
.
2013
;
35
(
4
):
320
327
17
Turner
DA
,
Schwartz
A
,
Carraccio
C
, et al;
Subspecialty Pediatrics Investigator Network (SPIN)
.
Continued supervision for the common pediatric subspecialty entrustable professional activities may be needed following fellowship graduation
.
Acad Med
.
2021
;
96
(
7S
):
S22
S28
18
Mink
R
,
Schwartz
A
,
Carraccio
C
, et al;
Steering Committee of the Subspecialty Pediatrics Investigator Network
.
Creating the Subspecialty Pediatrics Investigator Network [published correction appears in J Pediatr. 2019;207:269]
.
J Pediatr
.
2018
;
192
:
3
4.e2
19
Downing
SM
.
Validity: on meaningful interpretation of assessment data
.
Med Educ
.
2003
;
37
(
9
):
830
837
20
Norcini
JJ
.
Setting standards on educational tests
.
Med Educ
.
2003
;
37
(
5
):
464
469
21
Bates
D
,
Maechler
M
,
Bolker
B
,
Walker
S
.
Fitting linear mixed-effects models using Ime4
.
J Stat Softw
.
2015
;
67
(
1
):
1
48
22
Hamstra
SJ
,
Yamazaki
K
,
Shah
H
, et al
.
Milestones national report 2019
.
23
Accreditation Council for Graduate Medical Education
.
Program requirements for graduate medical education
.
24
Schmutz
JB
,
Meier
LL
,
Manser
T
.
How effective is teamwork really? The relationship between teamwork and performance in healthcare teams: a systematic review and meta-analysis
.
BMJ Open
.
2019
;
9
(
9
):
e028280
25
Mann
KJ
,
Craig
MS
,
Moses
JM
.
Quality improvement educational practices in pediatric residency programs: survey of pediatric program directors
.
Acad Pediatr
.
2014
;
14
(
1
):
23
28
26
Austin
RE
,
von Schroeder
HP
.
How accurate are we? A comparison of resident and staff physician billing knowledge and exposure to billing education during residency training
.
Can J Surg
.
2019
;
62
(
5
):
340
346
27
Andreae
MC
,
Dunham
K
,
Freed
GL
.
Inadequate training in billing and coding as perceived by recent pediatric graduates
.
Clin Pediatr (Phila)
.
2009
;
48
(
9
):
939
944
28
Sadowski
B
,
Cantrell
S
,
Barelski
A
,
O’Malley
PG
,
Hartzell
JD
.
Leadership training in graduate medical education: a systematic review
.
J Grad Med Educ
.
2018
;
10
(
2
):
134
148
29
Kumar
B
,
Swee
ML
,
Suneja
M
.
Leadership training programs in graduate medical education: a systematic review
.
BMC Med Educ
.
2020
;
20
(
1
):
175
30
Weiss
P
,
Myers
AL
,
McGann
KA
, et al
.
Funding sources and perceived financial insecurity in pediatric subspecialty fellowship programs
.
Acad Pediatr
.
2019
;
19
(
7
):
815
821
31
Ulmer
U
,
Wolman
DM
,
Johns
MME
;
Institute of Medicine of the National Academies
.
Resident Duty Hours: Enhancing Sleep, Supervision, and Safety
.
Washington, DC
:
The National Academies Press
;
2009
32
Alvira
CM
,
Steinhorn
RH
,
Balistreri
WF
, et al
.
Enhancing the development and retention of physician-scientists in academic pediatrics: strategies for success
.
J Pediatr
.
2018
;
200
:
277
284
33
Steiner
JF
.
Promoting mentorship in translational research: should we hope for Athena or train mentor?
Acad Med
.
2014
;
89
(
5
):
702
704
34
Kashiwagi
DT
,
Varkey
P
,
Cook
DA
.
Mentoring programs for physicians in academic medicine: a systematic review
.
Acad Med
.
2013
;
88
(
7
):
1029
1037
35
Karasick
AS
,
Nash
DB
.
Training in quality and safety: the current landscape
.
Am J Med Qual
.
2015
;
30
(
6
):
526
538
36
Lucas
R
,
Goldman
EF
,
Scott
AR
,
Dandar
V
.
Leadership development programs at academic health centers: results of a national survey
.
Acad Med
.
2018
;
93
(
2
):
229
236
37
AAMC
.
Leadership development
.
2021
.
38
Trimm
F
,
Caputo
G
,
Bostwick
S
, et al
.
Developing leaders in pediatric graduate medical education: the APPD LEAD Program
.
Acad Pediatr
.
2015
;
15
(
2
):
143
146
39
Yudkowsky
R
,
Downing
S
,
Tekian
A
.
Standard setting
. In:
Downing
S
,
Yudkowsky
R
, eds.
Assessment in Health Professions Education
.
New York, NY
:
Taylor and Francis
;
2009
:
119
148
40
Rickards
G
,
Magee
C
,
Artino
AR
 Jr
.
You can’t fix by analysis what you’ve spoiled by design: developing survey instruments and collecting validity evidence
.
J Grad Med Educ
.
2012
;
4
(
4
):
407
410

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.