Advancing the Agenda in Pediatric Mental Health Education

Calls for pediatric residency training programs to prepare future pediatricians to address MH problems began in the 1970s.^10,11 Nearly 2 decades later (1997), the Accreditation Committee on Graduate Medical Education (ACGME) mandated a 4-week development and behavior (DB) rotation.¹² Recently, the American Board of Pediatrics (ABP) made a call to improve MH training for pediatricians and have held stakeholder meetings of leaders in medical education.^1,13 The ABP also developed an entrustable professional activity (EPA) (ie, units of work that define a profession) to assess and manage patients with common behavioral and/or MH problems.¹⁴ 

Although the EPA clearly targets the assessment and management of MH problems, there is no consensus on which educational interventions are the most important and feasible in preparing future pediatricians to become competent in these skills. That said, residents who completed the mandated 4-week DB rotation are more likely to address MH problems^15,16; however, some believe making DB a rotation may have accidentally separated DB and MH issues when in actuality these are core to pediatrics.¹⁷ A decade after the mandate (2011), the majority of pediatric program directors (PDs) reported that their residents’ knowledge of the diagnosis and treatment of MH problems was average,¹⁸ and pediatric residents describe ongoing discomfort and uncertainty with all aspects of MH visits.¹⁹ 

Pediatric residents want to learn through active learning experiences, and several pediatric residency programs have successfully implemented interventions (eg, simulated patient experiences and role plays) that have impacted practices, such as addressing suicide and substance use disorder.^19,–21 Although more than half of pediatric residency programs train residents in primary care with on-site MH professionals, it is unknown whether this training environment impacts pediatric providers’ behaviors longer-term.²² Understanding which educational interventions are the most important, feasible, and readily implemented will be critical to creating the consensus needed to change the training paradigm.

Our goal in this study was to use the concept-mapping methodology to create a training framework to best prepare the pediatric workforce to provide MH care. Therefore, our objectives in this study were to (1) develop a concept map, gathering ideas on how to best improve training programs; (2) identify which ideas are important and feasible; and (3) assess the degree of consensus among stakeholders regarding the importance and feasibility of ideas generated.

We hypothesized that stakeholders will see an active learning approach as the most important change needed but the least feasible without substantial policy changes and new program initiatives. We also hypothesized there will be discernable differences between groups of stakeholders’ ratings regarding which ideas are more important and feasible. Results from this study will establish an empirically supported framework to guide action planning, program development, curricula, and policy.^23,24 

Concept mapping is a reliable and valid mixed-methods design to generate and organize a comprehensive set of ideas and represent them visually as a map.^23,24 It integrates input from multiple sources with differing content expertise or interest to ensure heterogeneity of perspectives. The process involves brainstorming ideas, structuring ideas (sorting and rating), and representing the ideas visually (see the Supplemental Information for details). All data were collected via a Web-based system by using Concept Systems software created for this process (Concept Systems Global Max).²⁵ This study was deemed exempt by our institutional review board.

The research team included a pediatrician (C.G.) with expertise in implementing and teaching MH in primary care, an expert in policy (W.T.) who created the concept-mapping methodology, an academic child and adolescent psychiatrist (J.T.W.), and a pediatrician with expertise in medical education (S.B.).

The research team identified potential participants to represent the variety of stakeholders relevant to MH training using nonrandom purposive sampling to maximize participation and help ensure heterogeneity of perspectives.²³ There were 2 groups of participants: an invited group and a core group. The large and anonymous invited group brainstormed ideas and later rated them for importance and feasibility. The core group performed these tasks and sorted ideas.

C.G. identified stakeholders involved in MH initiatives and parents of children with chronic physical and/or MH problems in the ABP’s Family Leadership Advisory Group. J.T.W. and S.B. identified chairs and PDs in both disciplines who could recruit faculty and trainees. Together, the team identified academic and community clinicians of both disciplines working in a variety of geographic locations (CO, NY, SC, AL, OH, MD, WA, CA, PA, and IL) and practice settings.

We e-mailed and invited all 117 identified potential participants to participate in this project as part of the invited group. They were encouraged to share the invitation with colleagues and Listservs if deemed appropriate, using a snowball sampling approach. The core group of 76 participants was chosen from the initial list but mainly represented key stakeholders in national efforts around pediatric MH, educators, and trainees.

Invited participants brainstormed ideas by completing the following focus statement: “To prepare future pediatricians for their role in caring for children and adolescents with mental and behavioral health conditions, residency training needs to...” Statements collected from a national workshop on the “10 best ways” to enhance pediatric residency training in MH were added to brainstormed responses.¹³ Ideas were synthesized manually via content analysis by team members, who coded statements, grouped statements with similar codes, and created an aggregate statement to represent all statements in each code. The final set of statements was reviewed by members of the research team and core group to ensure the final set was comprehensive.

Core group participants were asked to categorize the statements according to their view of their meaning or theme. After grouping similar statements into piles, they were asked to label each pile.

Core and invited participants rated the relative importance and feasibility of the ideas using a 5-point Likert-type scale for importance (1 = relatively unimportant; 5 = extremely important) and feasibility (1 = not at all feasible; 5 = extremely feasible).

Invited participants answered demographic questions: practice location (urban, rural, or suburban), practice type (academic, private, community, or not in professional practice), profession (pediatrician, psychiatrist, psychologist, social worker, nurse, nurse practitioner, policy maker, program developer, or other), years in practice (trainee, 0–5 years, 6–10 years, 11–15 years, 16–20 years, or >20 years), and if they were a parent (yes or no).

Using Concept Systems multidimensional scaling (MDS) and hierarchical cluster analysis created a point map, cluster map, bivariate “go zone,” and pattern match. The point map displayed the location of ideas (represented by a number); ideas with similar meanings were located closer to each other. The cluster map reflected the results of the cluster analysis and displayed the clustered ideas by enclosing them in polygon-shaped figures. Several cluster solutions were examined, ensuring merged ideas represented a single concept. The go-zone bivariate plot of idea ratings created a visual of ideas rated above average for both importance and feasibility. The pattern-match graph looked at the relationship between each cluster’s average ratings of importance compared with feasibility.

The research team conducted an initial review of data as the foundation for a facilitated videoconferencing session of a subgroup of the core group to review, interpret, and discuss use of results (Supplemental Information). The facilitator asked participants to read the ideas in each cluster and then generate a phrase to describe the set of statements. A discussion would follow until the group came to a consensus for an acceptable label for each cluster. The session ended by discussing which ideas should be easily implemented and where there were opportunities for innovation.

Descriptive statistics summarized the demographics and average statement and cluster ratings by using means and SDs. Pearson correlations measured correlations of a cluster’s importance and feasibility ratings. Paired-sample t tests compared averages between categories of the demographic variables of interest. Post hoc analyses using a Bonferroni correction factor of 40 accounted for multiple analyses for both ratings (ie, [7 clusters + overall average] × 5 demographic associations). We compared pediatricians to MH professionals by combining psychiatrists, psychologists, and social workers; nurses, nurse practitioners, and others were not included in the analyses for profession because their discipline was unknown. Practice location was dichotomized to urban versus rural and suburban. The years in practice category was dichotomized between trainees and all others. Academic practices were compared with private or community practices or respondents not in practice.

Snowball sampling yielded 244 participants who generated 279 brainstormed ideas that were supplemented by 97 statements from the workshop. These 376 ideas were edited, and compound statements were split to a final set of 499 distinct statements (Fig 1). Content analysis synthesized statements to a final set of 99. Two hundred and sixteen participants rated ideas for importance and 209 for feasibility. Of the participants who completed importance ratings, half work in pediatrics, more than half were trainees, and approximately two-thirds worked in urban and academic practices (Table 1). One-third were parents, and 56 of the 65 parents identified as professionals in the field. Of the 36% of respondents who answered “other” to profession, 86% were also trainees and likely answered “other” because they had not yet officially finished training in their discipline. Fifty-three percent (40 of 76) of the participants invited to the core group participated in sorting.

The final cluster map yielded 7 clusters (Fig 2). Table 2 lists the top 5 statements in each cluster ordered by their importance ratings.

The modalities for MH training cluster (8 ideas) includes teaching modalities needed to help trainees build clinical skills; creating longitudinal MH experiences was rated the most important and adapting curricula the most feasible.

The prioritization of MH cluster includes 18 ideas around training priorities and creating a culture in which MH is destigmatized and integrated into current clinical and teaching contexts, including collaborating with MH professionals. The most important statements were around culture change “…where MH is valued, de-stigmatized, and an integral part of all aspects of care,” integrating MH into the continuity clinic and providing clear expectations of the future pediatrician in MH care. Creating clear expectations of the pediatrician was the most feasible.

The systems-based practice cluster includes 10 ideas and refers to the larger system of MH and knowledge and use of resources in one’s practice and community. Using existing resources into clinical care was the most important and feasible statement.

The self-awareness and/or relationship building cluster includes 8 ideas and focuses on the internal and/or personal goals for the trainee, such as becoming more self-aware of their own stigma and biases. Promoting resilience and teaching coping skills was 1 of the most important and feasible statements.

The remaining 3 clusters describe the clinical skills identified as needed for pediatricians. The training in clinical assessment of patients cluster includes 24 ideas reflecting different aspects of assessment, including history taking and using screening tools. The training in treatment cluster includes 9 ideas about psychosocial and pharmacologic treatment options deemed appropriate for pediatricians. The diagnosis-specific skills cluster includes 21 ideas and describes skill development on specific MH diagnoses, including attention-deficit/hyperactivity disorder (ADHD), anxiety, and depression and/or self-harm, which were rated the most important.

Approximately one-third of all ideas were in the go zone with high importance and feasibility; the majority being clinical skills (Fig 3). Ideas with high importance yet rated below average in feasibility disproportionately come from the prioritization of MH cluster. See Supplemental Table 3 for each statement by cluster and zone.

The pattern match (Fig 4) reflects the lack of correlation (r < 0.9) for most clusters’ ratings of importance and feasibility. Statistically significant differences between importance and feasibility ratings were found for prioritization of MH, systems-based practice, training in clinical assessment of patients, and diagnosis-specific skills (P < .001 for each comparison).

Respondents living in suburban and rural areas rated each cluster higher in importance (P < .004) and feasibility (P = .004) compared with those living in an urban setting. MH professionals rated feasibility of prioritization of MH higher than pediatricians (4.3 vs 3.8 [95% confidence interval −0.73 to −0.18]; P = .04) but otherwise were in agreement with pediatricians. Trainees rated systems-based practice higher in importance (4.1 vs 3.7 [95% confidence interval −0.63 to −0.23]; P = .004) and every cluster higher in feasibility, although training in treatment did not reach statistical significance after post-hoc analysis.

After decades of efforts to address the gap in MH training,¹⁰ this is the first study to our knowledge that empirically gathered a comprehensive set of ideas on how to achieve this important goal. This 7-domain MH training framework, including 99 ideas generated from almost 250 participants, informs the future of training in pediatrics. The core concepts in the go zone are clinical skills addressing common MH problems that define the future pediatrician’s role in caring for children with MH problems. As hypothesized, there were common discrepancies between importance and feasibility in ideas and clusters and some differences between stakeholders on importance and feasibility ratings.

These data are more detailed than the American Academy of Pediatrics (AAP) 2009 MH competencies and the ABP’s EPA and include specific guidance not only on clinical skills but also approaches to training, professional development, and MH collaboration.^14,26 These data also come from multiple stakeholders with different perspectives, providing a more holistic view than previous studies. For instance, although having a faculty champion was most important to PDs at the national workshop,¹³ it was less important in these data. Stakeholders may feel a single faculty champion may not alone be able to move forward without culture and policy change and PDs may not be in the front line and know what is needed for change.

The discord between importance and feasibility ratings may explain why advancing the agenda in MH training has been so challenging. Culture change was the most important idea but 1 of the least feasible, and few studies have successfully addressed practice cultures.^27,28 This may explain why despite the development of several MH resources and curricula,^29,–31 there has been little progress in changing pediatricians’ practices.⁸ Additionally, culture change regarding the future pediatrician’s scope of practice involves multilevel interventions that address stigma, attitudes, and systems changes.⁵ However, only implementing highly feasible ideas will not catalyze change. For instance, implementing existing curricula (ie, from the Reach Institute³² or AAP³⁰), rated above average in feasibility, without addressing culture change may impede uptake, as demonstrated by 1 learning collaborative.^27,28 

Systemic barriers still exist that will make it challenging to implement highly important ideas. Despite half of training programs having an on-site MH professional in the continuity clinic,^13,22 there is no financial structure to support them in a role deemed as most important: providing direct supervision and consulting, advising, and collaborating with faculty and trainees.³³ Feasibility of integrating MH into the continuity clinic for longitudinal experiences with patients is further limited because the ACGME recently decreased the number of required sessions residents need to spend in the continuity clinic.¹² 

MH professionals rated prioritization of MH higher in importance, which is not surprising but is promising because most of the statements in this cluster will require their collaboration to become feasible. Previously, pediatric residents have identified many organizational barriers to integrating MH into their continuity clinic, which may explain why they rated systems-based practice higher in importance.¹⁹ Each cluster being rated higher in feasibility may be because they are not involved with the implementation of educational interventions and may not realize the resources needed for success. Participants working in rural and suburban settings rated all clusters higher for importance and feasibility compared with those in urban locations, which may reflect a greater need and urgency.³⁴ Not only is there a shortage of MH professionals, but there is also a maldistribution, with the majority practicing in urban locations. This may explain their importance ratings but may not explain feasibility, and this warrants further study to inform implementation throughout the country.

Arguably, concepts in the go zone should be implemented immediately by programs by using resources that already exist. More specifically, addressing depression and better dissemination of existing underused resources are both in the go zone. Faculty can teach residents by reviewing the AAP’s guidelines for adolescent depression³⁵ and watching a sample motivational interview where a provider engages an adolescent with depression, which can be found on the AAP website.²⁹ Adhering to these guidelines and implementing depression screening in the continuity clinic will then provide experiential opportunities. However, once depression screening is implemented, trainees will need longitudinal experiences and mentorship from experienced practitioners to respond to positive screen results and appropriately manage their patients. Although addressing anxiety was in the go zone, there are currently no clinical guidelines for pediatricians, which may impede implementation.

Developing and studying interventions to make important ideas feasible are needed to catalyze change. Not only do programs need to engage local and national leadership for culture change, but attitudes around MH need to be addressed through strategies that facilitate reflection on stigma and biases.³⁶ The ABP recently engaged 13 PD–department chair dyads at a stakeholder meeting around MH training, which is promising, but also needs to reach all programs and leadership.³⁷ Changing training priorities to include MH to the same degree as other common childhood illness will not happen without changing ACGME requirements. Mandated time in specific areas (inpatient versus continuity clinic) needs to reflect the changing epidemiology of pediatrics.³ Reconsidering the decrease in required continuity clinic sessions can provide residents with more longitudinal experiences and maximize exposure to MH professionals because half of continuity clinics have one on-site.²² Lastly, clinical guidelines for anxiety and all skills listed in the go zone need to be developed.

Despite using purposive and snowball sampling, it is possible not all important stakeholders were reached, which could limit generalizability. The sample only included 24 MH professionals, 35 participants in nonacademic institutes, and 47 participants who had >10 years’ experience, making it possible that we did not have a large enough sample to compare differences between these stakeholders, and input from them may have been under-represented. However, a pooled study analysis of 69 studies that assessed the rigor of concept mapping found the average sample size for the first Web-based rating to be 112.8 and 75.6 for the second rating. Our sample sizes of 217 and 209 were much larger, suggesting we had an adequate absolute number of participants to ensure heterogeneity of perspectives.²⁴ Because similar statements were synthesized, if only 1 stakeholder of a certain demographic responded with an idea that was different from others, it would have been included as 1 of the 99 final ideas used in this project. Future study should include input from payers and children with MH problems who were not included in this sample. Lastly, there are small differences in ratings between statements, suggesting that merely focusing on the go-zone statements may not be the best strategy, as illustrated throughout this discussion.

The study results can drive future planning in advancing MH training in pediatrics. Ideas that were deemed feasible and important can help drive curricular interventions by using resources that already exist. Strategies are needed for culture and policy change, and innovation is needed to make important ideas more feasible, such as creating and studying sustainable models integrating MH professionals into training programs. Specifically, we recommend the following:

1. embrace MH as an integral part of all aspects of care to change culture and reduce stigma;

2. implement existing curricula, such as from the AAP or Reach Institute, with an emphasis on ADHD, anxiety, depression, and suicide and/or self-harm;

3. integrate MH care into continuity clinics by creating systems for screening for early risk factors, performing diagnostic assessments, and providing first-line treatment;

4. ensure trainees have longitudinal experiences with children with MH problems, and track these experiences;

5. train pediatric role models to precept residents;

6. provide on-site MH professionals to precept trainees alongside pediatric attending physicians;

7. train MH professionals in the most effective models of working with their pediatric colleagues; and

8. advocate for financial structures and/or incentives to enable pediatricians and MH professionals to address the return on investing in integrated care models.

We acknowledge all participants in the invited and core group. We also acknowledge Dr Julia A. McMillan, who provided feedback throughout the process and was part of the interpretation session; Drs John Duby, Lawrence Wissow, Marshall Land, Elena Mann, and David Kaye for their participation in the interpretation of the results session; and Dr Paul Christos for his statistical support.

AAP

American Academy of Pediatrics

ABP

American Board of Pediatrics

ACGME

Accreditation Committee on Graduate Medical Education

ADHD

attention-deficit/hyperactivity disorder

DB

development and behavior

EPA

entrustable professional activity

MDS

multidimensional scaling

MH

mental health

PD

program director