Defining the Essential Components of a Teaching Service

In our study, we used the modified Delphi method to generate consensus regarding the essential components of resident teaching services. The Delphi method is defined as a technique “for achieving consensual agreement among expert panelists through repeated iterations of anonymized opinions and of proposed compromise statements from the group moderator.”¹⁹  This type of method is often used to synthesize expert opinions to provide recommendations, especially for topics for which evidence is limited.^20,21 

The Delphi method consists of 6 stages: (1) identifying a research problem, (2) conducting a literature search, (3) developing a questionnaire, (4) coordinating anonymous iterative questionnaire rounds, (5) providing group feedback between rounds, and (6) summarizing the findings.^22,23 

To create the initial list of potential components of teaching services, we conducted a focus group with ∼20 rotation directors at a single large pediatric residency program. During this session, we had participants brainstorm criteria of an ideal teaching service. We also sought the opinions of 10 pediatric residents from each postgraduate year via an open-ended e-mail free-text response. Lastly, we conducted a literature search to further supplement our list of candidate components. We reviewed both narrative pieces^{8–11,24–26}  as well as studies that were used to evaluate teaching services^{9,11,14,17,18,27–32}  to prompt more ideas. The author team subsequently reviewed the candidate characteristics generated by these methods and eliminated redundant concepts as well those with limited generalizability.

The author team then developed a questionnaire listing the selected potential components. The components fell under 2 domains: attending characteristics and system characteristics. We performed cognitive interviewing with 4 attending physicians and adapted the survey accordingly for clarity. We prospectively determined that we would complete a maximum of 5 rounds using the Delphi methodology.

We sought to form a diverse group of pediatricians to serve on our expert panel, including residents, hospitalists, and other pediatric subspecialists. We included representatives from different regions of North America and representatives who worked with trainees from small (<30 residents), medium (30–60 residents), and large (>60 residents) residency programs. Inclusion criteria for our expert panel included candidates who (1) had interest and experience in medical education, ideally holding local, regional, and/or national leadership positions, and (2) worked clinically as an inpatient attending with pediatric residents. Additionally, we sought to include at least 1 current pediatric resident. To assemble the expert panel, we asked the presidents and/or educational leaders of several prominent pediatric organizations via e-mail to nominate 1 to 2 expert panelists who met those criteria. We received nominations from the American Academy of Pediatrics (AAP), the AAP Section on Pediatric Trainees, the AAP Section on Hospital Medicine Subcommittee on Pediatric Educators, the American Board of Pediatrics, the Academic Pediatric Association, the American Pediatric Society, the Association of Medical School Pediatric Department Chairs, the Association of Pediatric Program Directors, the Council on Medical Student Education in Pediatrics, and the Council of Pediatric Subspecialties. We invited all nominees who met our inclusion criteria to participate. We subsequently invited 3 additional pediatricians who met our inclusion criteria to join our panel, allowing for increased diversity regarding geographic location and subspecialty focus.

We sent the panelists an e-mail invitation to participate with a description of the time commitment, study procedures, and inclusion criteria. If a participant declined to participate or felt that he or she did not meet inclusion criteria, we then asked the nominating organization for a different nominee. We continued this process until we had at least one participant from each of the participating organizations. Panelists were blinded to each other’s identities throughout the study, ensuring confidential responses and independent contributions.

In our initial questionnaire, we included an introduction that explained the purpose of the study, the definition of a teaching service, and an explanation of the panelist’s role. The panelists were presented with a list of candidate components that could be integrated into a teaching service and were asked to rate each element on a 4-point rating scale: (1) recommend as an essential component (referred to in this article as “essential”); (2) recommend, but not essential (referred to in this article as “recommended”); (3) neutral; or (4) do not recommend (referred to in this article as “not recommended”). Panelists were also given the opportunity to use free-text comments to suggest both alternative wording and additional elements to be included in subsequent iterative rounds. On the basis of consensus research literature, we predefined consensus as at least 70% agreement with 1 response within the 4-point scale.³³  The study team decided to classify any items that did not ultimately meet consensus in a single category, but met at least 70% combined agreement between both essential and recommended, as the latter (recommended). We felt these items should not be overlooked despite not meeting consensus and decided that including them in the recommended category would be the most conservative approach.

After completion of each round, the author team analyzed the collated, deidentified responses. Characteristics that reached consensus were removed from subsequent rounds. On the basis of panelist qualitative responses, all other characteristics fell into 1 of the following categories: (1) inclusion in the subsequent round with no adjustments, (2) inclusion in the subsequent round with clarifying adjustments to wording, or (3) removal from the subsequent survey due to a perceived lack of universal applicability. Panelists’ comments were reviewed by the author team to identify common and salient recommendations; the author team reached agreement on appropriate revisions and additional candidate characteristics to be included in subsequent rounds.

After each round, all results, including the deidentified raw quantitative and qualitative data as well as the outcomes of the postround analysis, were disseminated to all panelists. Panelists were encouraged to review these results before completing the subsequent round’s questionnaire, which aided in moving the group toward consensus while maintaining panelist anonymity.

We also distributed a subsequent survey, asking the panelists to rate (1) the effort and (2) the impact related to characteristics that reached consensus as recommended in the first 4 rounds, with the goal of creating an action priority matrix, a visual tool used to prioritize a group of ideas, solutions, or actions across 2 dimensions: effort and impact.³⁴  The action priority matrix, a diagramming technique most often used in business, helps to prioritize action items, allowing for the efficient use of time and resources; by plotting effort and impact, action items can be placed in 1 of 4 quadrants: (1) quick wins (high impact, low effort), (2) major projects (high impact, high effort), (3) fill-ins (low impact, low effort), and (4) thankless tasks (low impact, high effort). In our study, effort and impact were rated on a 5-point scale, with 1 being the lowest and 5 being the highest scores. The author team chose not to study the impact and effort of characteristics that reached consensus as essential because it was felt that, in our study, we had already deemed these characteristics to be inherently necessary to all teaching services.

The institutional review board of the principal investigator’s hospital deemed the study exempt from review. The survey was distributed via e-mail and administered by using Qualtrics online surveys (Qualtrics, Provo, UT). All 5 questionnaires were sent between December 2018 and June 2019, and all data were collected during this time interval.

The focus group with rotation directors yielded 22 candidate components, free-text responses from pediatric residents yielded 30 candidate components, and the literature search yielded 56 candidate components. The author team reviewed all 108 items and grouped the characteristics into 2 broad categories: attending characteristics and system characteristics. After elimination of redundant and nongeneralizable components, 47 candidate components remained; these components were all included in the first-round survey by using the modified Delphi method.

The final expert panel included diversity regarding geographic location, size of respective residency program, and pediatric specialty (Table 1). All invited panelists (n = 18; 15 nominated and 3 invited by the study team) who met criteria agreed to participate in our study. There was 100% participation in all 5 surveys, which included 4 rounds by using the modified Delphi method (Fig 1) and a fifth survey evaluating the effort and impact related to characteristics that reached consensus as recommended.

Fourteen attending physician characteristics and 7 system characteristics reached consensus as essential components of teaching services (Table 2). Twenty-one items reached consensus as recommended by having at least 70% agreement with this response, whereas an additional 4 items (R-A10, R-S3, R-S5, and R-S9) reached consensus by having at least 70% combined agreement between essential and recommended by the end of 4 rounds (Table 3). Among the 25 total items that reached consensus as recommended, 11 items were attending characteristics and 14 items were system characteristics. We created an action priority matrix on the basis of the results from the fifth survey (Fig 2). Each point on the figure represents the mean of 18 ratings of effort and impact. The implementation of attending characteristics was perceived to require less effort than the implementation of system characteristics (mean effort for attending characteristics: 2.95; mean effort for system characteristics: 3.69) but led to similar impact (mean impact of attending characteristics: 3.57; mean impact of system characteristics: 3.47).

Three items reached consensus as neutral, 1 item reached consensus as not recommended, and 1 item did not reach consensus by the end of 4 iterative rounds (Table 4). All 5 of these items were systems characteristics.

In this study, we used the Delphi consensus methodology to define both the (1) essential and (2) recommended components of a teaching service. During pediatric residency, the majority of training occurs in the inpatient setting, making these findings valuable for most institutions, divisions, and residency programs.

Not surprisingly, many of the essential components elucidated by this study are also described in the ACGME Common Program Requirements. For example, there was consensus in our study that attending physicians should receive training around teaching (E-A1). This concept, which has been well described in previous literature,³⁵  is also recommended by the ACGME, which states that “faculty pursue faculty development designed to enhance their skills at least annually…intended to describe structured programming developed for the purpose of enhancing transference of knowledge, skill, and behavior from the educator to the learner.” Our findings will hopefully reinforce the importance of these common components.

Conversely, there were a few components that reached consensus as essential that are less well detailed in the ACGME requirements; these components are focused on resident feedback and the solicitation of attending physician feedback. Although the ACGME states that “evaluation must be documented at the completion of the assignment” and that “faculty members should provide feedback frequently throughout the course of each rotation,”⁷  from our results, we provide even more granular guidance on these tasks, recommending that attending physicians set expectations with residents (E-A3), provide timely verbal feedback to residents (EA-11), complete required written evaluations regarding residents in a timely manner (E-A12), provide narrative comments in written evaluations (E-A13), and solicit feedback from residents regarding their own performance (EA-14). Therefore, this tool not only highlights the importance of the ACGME requirements but also emphasizes the essential components that are not outlined by accrediting bodies.

Twenty-five items reached consensus as recommended. The items on this list are less represented in the ACGME Common Program Requirements and could serve as a resource for those who wish to improve the quality of their teaching services. For example, a particular division may establish 2 inpatient teams: a teaching service with trainees and a nonteaching service without trainees. This division could optimize its teaching service by intentionally selecting attending physicians on the basis of teaching ability (R-A2) or interest in being a teaching attending (R-A3), 2 components that reached consensus as recommended.

Notably, many of the topics that reached consensus as recommended could be achieved through faculty development efforts, which would also satisfy the essential component that faculty receive training around teaching (E-A1). Furthermore, by receiving training around teaching, faculty may be better prepared to set individualized learning goals with residents (R-A7) and devote attention to linking clinical experiences with trainee career goals (R-A8), 2 additional components that reached consensus as recommended.

Several recommended components address the increasing workload of today’s learners. These include the integration of care team assistants (R-S10), the development of strategies to address excessive documentation demands (R-S11), the use of patient caps (R-S12), and the use of nonresident solutions to offload patients in case of particularly high censuses (R-S13). Importantly, this list also addresses the increasing workload of attending physicians by suggesting the development of a mechanism to protect attending physicians from other major responsibilities when on service (R-S8). Because burnout and work compression remain prevalent among both physicians in training^36,37  and those in practice,³⁸  it becomes imperative that programs continue to consider strategies to limit workload expansion.

Interestingly, the only item that reached consensus as not recommended was that teaching services should exclude fellows (D-S1). Although anecdotally, concerns have been raised that fellows hinder resident autonomy, fellows have been shown to serve as valuable resident teachers.³⁹  Fellows often have a better understanding of residents’ current knowledge base and what skills they still need to acquire to perform well, in comparison with attending physicians who are further away from training. This finding supports inclusion of fellow supervisors in inpatient teaching services. This is increasingly relevant because pediatric hospital medicine is now recognized as a subspecialty,⁴⁰  which will undoubtedly lead to more hospitalist fellowship programs and hospitalist fellows in inpatient teaching services.

Importantly, implementing any of these components will take varied amounts of effort, which will further vary between programs. To better understand the variability in perceived effort and impact of these interventions, we surveyed the expert panelists on their perceptions of the effort and impact of components that reached consensus as recommended, allowing us to generate an action priority matrix. Interestingly, on average, attending characteristics were more likely than system characteristics to be rated as having higher impact with lower effort, placing them in the quick wins category. This suggests that key stakeholders may want to consider starting with attending characteristics when implementing new initiatives. That being said, although systems-based change may initially require more effort to implement, these changes may allow for more sustainable improvement and should be closely considered. It is important to note that impact and effort will likely vary from program to program and service to service, and we therefore suggest that each program and division complete its own individualized action priority matrix before instituting changes.

Limitations to our study include those inherent to the Delphi methodology. Although care was taken to be thorough in constructing our initial list of candidate components, it is possible that some potential components in the literature were missed; thus, we may have unintentionally guided the panel in our survey design. Additionally, although we made a significant effort to build a pool of diverse panelists on the basis of subspecialty, program size, and geography, our findings may not represent the opinion of all pediatric medical education experts. Because our panel included only pediatricians and was geared toward inpatient services, it may be difficult to apply these findings to other specialties, particularly ones in which training primarily takes place in the ambulatory, emergency department, or operating room settings. Our final lists include a select number of components that, in their current state, are nonobservable, and therefore further refinement to these criteria is required before these lists can be used as an evaluative tool. The Delphi methodology is also inherently affected by subject bias, although this should dually be considered a benefit because it allows for the development of consensus.¹⁶ 

The results of this study suggest a framework that programs can use when creating, redesigning, or evaluating teaching services at their institutions. In our framework, we outline components essential to teaching services that we suggest should be universally implemented along with nonessential but recommended components that should be carefully considered and implemented pending resources and relevance to the given clinical environment. The focus of future research should be on validating this list and creating a tool that can be implemented to more generalizable settings.

We thank the members of our medical education expert panel, which included Drs Susan Bannister, Susan Bostwick, April Buchanan, Ann E. Burke, Douglas Carlson, H. Barrett Fromme, Melvin Heyman, Heather Howell, Jennifer Kesselheim, Su-Ting Li, Leah Mallory, Jay Mehta, Michael S. Ryan, Suzanne Stewart, David A. Turner, Jacqueline Walker, Michael Weisgerber, and Eric Zwemer.

AAP

American Academy of Pediatrics

ACGME

Accreditation Council for Graduate Medical Education