Achieving Entrustable Professional Activities During Fellowship

SPIN is a collaboration between the American Board of Pediatrics (ABP), Association of Pediatric Program Directors (APPD) Fellowship Committee, APPD Longitudinal Educational Assessment Research Network, and Council of Pediatric Subspecialties, and it includes 1 to 2 representatives from each of the pediatric subspecialties with ABP certification.¹⁸  As described in our previous report,¹⁷  we sent a confidential survey using LimeSurvey between April 2017 and August 2017 to FPDs of the Accreditation Council for Graduate Medical Education (ACGME)–accredited US fellowship programs for the 14 subspecialties with ABP certification at the time of the study. After a literature review,^{9–11 ,19,20}  we designed a survey that was reviewed and edited by SPIN representatives, all of whom have graduate medical education experience (Supplemental Fig 3). SPIN representatives chose 2 to 4 faculty, previously serving as FPDs in each subspecialty, to pilot-test the survey. Their feedback was used to modify the survey and develop the final version. The study was determined to be exempt by the University of Illinois at Chicago Institutional Review Board.

We set a goal to have at least 75% of all FPDs in each subspecialty participate. We sent the original invitation and 4 reminders by e-mail to prospective participants. SPIN representatives then contacted and requested all nonresponders via e-mail to complete the survey. Representatives had access to FPD names and e-mail addresses but not to the participant’s responses.

The 7 common pediatric subspecialty EPAs and supervision scales are shown in Table 1. The scales are based on 5 levels of supervision, with higher levels meaning less supervision. In the survey introduction, we described the concept of EPAs and the functions of each, entrustability of a subspecialist (when a subspecialist should be able to perform most of the functions defined by the EPA, resulting in a safe and effective outcome), and assignment of the supervision level (based on trust and not necessarily on direct observation of all associated functions; Supplemental Fig 3). For each of the EPAs, we asked the following:

1. “For this EPA, what do you believe is the minimum level of supervision a fellow must achieve to successfully complete fellowship?”

2. “For this EPA, if a fellow did not achieve at least this minimal level of supervision, would you still allow him or her to graduate?”

3. “For this EPA, what is the lowest level in which you would consider that a practicing subspecialist (and not necessarily a trainee) should be able to perform most of the activities described above resulting in a safe and effective outcome?”

Data are described as n (%) for categorical survey items and median and percentiles. We previously described the minimum level of supervision for graduation as that reported by the FPDs to successfully complete the fellowship.¹⁷  However, if the FPD indicated that they would allow the fellow to graduate even if the fellow did not achieve that level of supervision, it was set at 1 level below (adjusted minimum level of supervision). On the basis of consensus of SPIN representatives and other experts in graduate medical education, we defined the minimum level of supervision for graduation and practice as the level that at least 80% of FPDs would not drop below, a boundary that aligns with previously published data.^13,17  We tested for differences in the distribution of responses to the graduation and practice items for each EPA using nonparametric within-subject Friedman rank sum tests. For comparison, we divided the EPAs into clinical (consultation, handovers, and lead team) and nonclinical (quality improvement [QI], management, lead profession, and scholarship). We compared mean differences between graduation and practice responses for each EPA and for clinical versus nonclinical EPAs using paired t tests and linear regression models fitted to the difference scores with a fixed effect of subspecialty. We used pediatric endocrinology as the reference level for the subspecialty effect because its mean FPD responses were nearest the overall mean across EPAs. A P value <.05 was considered significant. Data analyses were conducted by using R 3.6 (R Core Team, Vienna, Austria) and lme4 package.²¹ 

We collected data from 660 of 802 FPDs with an overall response rate of 82%. All subspecialties, except for infectious diseases, met the goal of 75% for participation (Supplemental Table 3). We previously described characteristics of the FPDs, programs, and minimum expected supervision levels for graduation.¹⁷  The adjusted minimum level of supervision that FPDs deemed necessary for fellows to successfully graduate was compared with the level they expected for subspecialists to practice for each EPA in Fig 1. The unadjusted graduation data for graduation are included in Supplemental Table 4. As we previously showed,¹⁷  to graduate, the majority of FPDs did not require fellows to be trusted to practice without supervision or coaching for any EPA. To practice as a subspecialist, FPDs expected the least amount of supervision for the EPAs consultation and handovers; however, even for those EPAs, a significant number of FPDs (49% and 35%, respectively) did not report unsupervised practice as being necessary for safe and effective outcomes. For the other EPAs, the majority of FPDs did not regard unsupervised practice as necessary for a subspecialist to practice.

The adjusted minimum level of supervision to graduate a fellow and for a subspecialist to practice for each EPA are shown in Fig 2. For graduation, FPDs deemed indirect supervision acceptable for patient care EPAs such as consultation (level 3) and handovers (level 4), but would allow fellows to graduate who required even more supervision for the other EPAs. For practice, FPDs expected the least amount of supervision for consultation and handovers. However, even for these EPAs, FPDs found it acceptable that practicing subspecialists may require indirect supervision for select complex patients (level 4). For the remaining EPAs, FPDs believed that practicing subspecialists may require consultation for all tasks and occasional advice (management and lead team) or even direct supervision (QI, lead profession, and scholarship).

For all EPAs, FPDs did not require fellows to reach the level of entrustment at graduation that they expected of subspecialists to practice for safe and effective outcomes (Fig 1, Table 2). The smallest differences between the levels were observed for consultation and handovers EPAs whereas the largest differences were for lead profession, management, and lead team. The mean differences between graduation and practice were smaller for clinical EPAs when compared with nonclinical EPAs, even when adjusted for subspecialty (P = .001). Compared with pediatric endocrinology (Table 2), the magnitude of the difference between graduation and practice was significantly greater for pediatric pulmonology for management (P = .049) and handovers (P = .03) EPAs but less for pediatric nephrology for QI (P = .03), pediatric emergency medicine for consultation (P = .02), and critical care and neonatology for lead team (P = .01 and .02, respectively). There were no other significant differences among the other subspecialties for the other EPAs (P > .05).

In our study, FPDs did not require fellows to reach the level of entrustment at graduation that they expected of subspecialists to practice for safe and effective outcomes for any of the common pediatric subspecialty EPAs. We previously showed that FPDs did not require fellows to be entrusted for unsupervised practice at graduation for these EPAs.¹⁷  By adjusting for FPD willingness to graduate fellows even if they did not achieve the minimum level of supervision, the amount of supervision deemed necessary for new graduates was even greater than in our previous report. Our findings are consistent with data on general pediatrics EPAs demonstrating that expectations for unsupervised practice at the time of graduation vary by EPAs and programs.¹³  Additionally, they align with the variability in the reported ACGME milestone levels that fellows achieve at graduation.²²  The fact that fellowship graduates may still require supervision conflicts with requirements that they be prepared for unsupervised practice.²³  Our findings, especially if supported by actual fellow performance data, have important implications for decision-making around future fellowship graduation requirements, the mentorship and support structure for new graduates, and structured learning and assessment as part of maintenance of certification. Educational leaders and regulatory bodies need to reevaluate fellowship program graduation standards to align the structure and curricula of fellowship programs with the desired outcomes outlined in the EPA framework. This reevaluation should be viewed through the lens of lifelong learning so that future approaches are developed in a way to best facilitate transitions between fellowship and practice to optimize both education and patient outcomes.

FPDs expected the least amount of supervision for graduation and subspecialty practice for EPAs that involve patient care. The gap between these expectations was also smallest for these EPAs, which likely reflects prioritization of what is most important for the immediate needs of patients and their safety. However, even for these EPAs, FPDs reported that they would graduate fellows who do not meet the minimum levels of entrustment required for safe and effective practice. The gap was smallest for activities such as consultation and handovers, but it was larger for leading a team, despite the known importance of this skill to patient outcomes.²⁴ 

For nonclinical EPAs, FPDs reported allowing graduation of fellows who require direct supervision and coaching, which exceeds expectations for oversight of practicing subspecialists for effective outcomes. The need for continued coaching is not surprising given the years of experience that are needed to attain expertise in these activities after graduation. Although there may be some debate as to whether these nonclinical activities should be included in the EPA schema, they were determined by a consensus of subspecialty experts to be an important part of the framework because the EPAs facilitate development of a shared mental model of trainee performance. In addition, there is strong validity evidence to support the supervision scales used in this investigation for these nonclinical activities.¹¹ 

The magnitude of the gaps between the level of supervision expected for graduation and entrustment as a subspecialist may align with the importance that FPDs place on the activity, outcomes of their training program, or postgraduate careers of their fellows. Additionally, some programs may lack the resources, expertise, time, or opportunity to train fellows sufficiently to levels of unsupervised practice in domains such as QI,²⁵  practice management,^26,27  leadership,^28,29  and research.³⁰  The gap for each EPA was similar between most subspecialties. The few differences may reflect variability in the importance that those subspecialists place on a given activity. For example, for leading a team, the gap was smaller in critical care and neonatology, perhaps reflecting the importance of teamwork in the critical care environment.²⁴  Similarly, consultation may play a more prominent role in the care delivered by emergency medicine physicians. The reasons for other differences, such as the larger gap in pulmonology for handovers, are unclear. Qualitative studies would be helpful to delineate them.

Our findings support the need for continued, formalized mentorship to continue development of clinical proficiency for recent graduates in their new positions to ensure quality patient care. Graduates who have demonstrated competence within their training programs may need to care for new patient populations within unfamiliar complex systems of health care delivery. In academic centers, training is often provided for administrative tasks (eg, billing) for new physicians, but confirmation of clinical proficiency (eg, review and oversight of initial clinical activities) is variable. Addressing the gaps that new graduates perceive in their competency could minimize the stress and burnout that they experience in the transition process.¹⁶  An initial period of supervision similar to that required in training programs, consisting of observation and assessment (with feedback) by supervisors, should be considered until graduates demonstrate competency in their new activities.³¹  A formalized handover process from training program to practice would facilitate identification of areas in which support of clinical skills should be provided. Identification and training of supervisors to assess and mentor new graduates is also needed, and they should meet regularly and develop individualized plans. Collaboration with the ABP to create learning activities for maintenance of certification can help address important skills gaps.

Our study also underscores the need for mentorship and training for early career subspecialists for professional development in areas such as scholarship, QI, and leadership. Formal mentorship is vital for the career development of physicians in academic medicine,^32–34  and new graduates should be assigned a mentor. Elements of effective mentoring programs include mentor training, an oversight committee, contracts between mentor and mentees, formal curricula for mentees, adequate funding, and protected time.³⁴  Development of expertise in QI and leadership skills may require additional training either within or outside the institution. Training in QI is offered by a number of organizations,³⁵  including the Institute for Healthcare Improvement, the American Institute for Healthcare Quality, the Academic Pediatric Association, and the American Association of Medical Colleges (AAMC). Most academic health centers provide leadership development programs; however, access may be limited.³⁶  Formal leadership training programs and seminars are offered by professional organizations, such as the AAMC,³⁷  the APPD (eg, Leadership in Educational Academic Development [LEAD]),³⁸  the Academic Pediatric Association, the American Academy of Pediatrics, and individual subspecialty organizations (eg, the American Thoracic Society, the American Gastroenterological Association, the Society of Hospital Medicine). Additionally, the AAMC offers specific leadership programs for early career women and underrepresented minorities.

This investigation demonstrated a gap in skills between FPD expectations of fellow performance at the time of graduation and the skills expected for practice. Additionally, there was variability between FPDs in their expectations because many were willing to graduate fellows below an expected minimum level. One way to close these gaps is to create a shared mental model for what is to be achieved as a fellow and design structured learning and assessment during the early years of maintenance of certification to achieve the skill level needed in practice. Setting standards for EPA-based assessment tools is a complex process that includes the judgment of expert assessors.²⁰  We chose to survey FPDs who are responsible for verifying fellow readiness for graduation, but other stakeholders must be involved in setting these standards³⁹  (eg, division chiefs, chairs, subspecialists in academic and private practice settings, trainees, patients, families, and regulatory organizations) because of the potential impact on the need for assessment and support after graduation.

There are several limitations to our study. We previously collected validity evidence for the use of EPAs for fellow assessment¹¹ ; however, it is possible there was confusion in the FPD construal of the supervision scales and concept of entrustment for subspecialists. For graduation, we asked FPDs to assign the supervision level on the basis of what they would prospectively trust fellows to do, not what they directly observed them to do. Because institutional and accrediting bodies require supervision for these activities, FPDs may overestimate the amount of supervision needed for graduation. We adhered to principles of survey design,⁴⁰  but it is possible that our questions did not adequately cover the construct of interest. There is potential for self-reporting and nonresponder bias; however, the latter is mitigated by the high response rate. In the survey, we asked about the expected minimum supervision level, which is important in setting standards but does not address FPD expectations for the majority of fellows and practicing subspecialists. It is possible that FPD expectations could have changed since the study was completed; however, that is unlikely, given the lack of intercurrent studies on EPAs in pediatric fellowship programs, modification of assessment tools, or changes in requirements by accrediting organizations. Our ongoing longitudinal EPA study should help to determine if fellow workplace performance during training is consistent with FPD expectations at graduation.

This study has important implications for educators, supervising physicians, and regulatory agencies, suggesting a need to align the expected outcomes of fellowship training programs with the skills required for safe and effective practice. It underscores the importance of setting standards to ensure fellow competence on graduation. EPAs represent one tool for addressing the ongoing learning that needs to occur in the early years of practice to meet expectations for practicing subspecialists and throughout the continuum to both acquire needed skills and maintain them. The period of time for which early career subspecialists require enhanced support is unclear and is an area for potential future research.

AAMC

American Association of Medical Colleges

ABP

American Board of Pediatrics

ACGME

Accreditation Council for Graduate Medical Education

APPD

Association of Pediatric Program Directors

EPA

Entrustable Professional Activity

FPD

fellowship program director

QI

quality improvement

SPIN

Subspecialty Pediatrics Investigator Network