Video Abstract

Video Abstract

Close modal
BACKGROUND AND OBJECTIVES:

There is an urgent need to advance mental health (MH) education and/or training in pediatric residency programs, yet no consensus on how to achieve this. We created an operational framework from ideas provided by a diverse group of stakeholders on how to advance MH education.

METHODS:

Concept-mapping methodology was used, which involves brainstorming ideas by completing a focus prompt, sorting ideas into groups, and rating them for importance and feasibility. Multidimensional scaling and hierarchical cluster analysis grouped ideas into clusters. Average importance and feasibility were calculated for each statement and cluster and compared statistically in each cluster and between subgroups.

RESULTS:

Ninety-nine ideas were generated. Sorted ideas yielded a 7-cluster concept map: (1) modalities for MH training, (2) prioritization of MH, (3) systems-based practice, (4) self-awareness and/or relationship building, (5) training in clinical assessment of patients, (6) training in treatment, and (7) diagnosis-specific skills. Two hundred and sixteen participants rated ideas for importance and 209 for feasibility. Four clusters had a statistically significant difference between their importance and feasibility ratings (P < .001). Suburban and rural area respondents (versus urban) rated clusters higher in importance and feasibility (P < .004), trainees rated all clusters higher in feasibility than practicing clinicians, and MH professionals rated prioritization of MH higher in feasibility (3.42 vs 2.98; P < .001).

CONCLUSIONS:

This comprehensive set of ideas, especially those rated highly in both importance and feasibility, should inform curricular and policy initiatives. Differences between importance and feasibility may explain why there has been little progress in this field.

What’s Known on This Subject:

Despite decades of calls to improve training so that pediatricians can care for children with mental health problems, current training programs do not prepare them to do so.

What This Study Adds:

This multistakeholder analysis provides a complete set of ideas, represented visually as a concept map, to improve mental health education in pediatric training programs. Ideas are rated for importance and feasibility to guide action planning.

There is an urgent need for pediatric workforce development to train providers to care for children with mental health (MH) problems, yet there is little understanding on how best to address this need.1 The burden of pediatric MH problems is a national public health crisis because MH problems are the most costly pediatric condition, increasing in prevalence, and if untreated lead to adult morbidity and higher health care costs.2,4 MH conditions affect 1 in 5 children; yet only half are recognized, and only another half receive appropriate services.5,6 Although pediatricians are currently the only source of care for more than one-third of children with MH problems,7 the majority of pediatricians are not routinely inquiring and/or screening for them and cite their lack of training as a barrier.8,9 

Calls for pediatric residency training programs to prepare future pediatricians to address MH problems began in the 1970s.10,11 Nearly 2 decades later (1997), the Accreditation Committee on Graduate Medical Education (ACGME) mandated a 4-week development and behavior (DB) rotation.12 Recently, the American Board of Pediatrics (ABP) made a call to improve MH training for pediatricians and have held stakeholder meetings of leaders in medical education.1,13 The ABP also developed an entrustable professional activity (EPA) (ie, units of work that define a profession) to assess and manage patients with common behavioral and/or MH problems.14 

Although the EPA clearly targets the assessment and management of MH problems, there is no consensus on which educational interventions are the most important and feasible in preparing future pediatricians to become competent in these skills. That said, residents who completed the mandated 4-week DB rotation are more likely to address MH problems15,16; however, some believe making DB a rotation may have accidentally separated DB and MH issues when in actuality these are core to pediatrics.17 A decade after the mandate (2011), the majority of pediatric program directors (PDs) reported that their residents’ knowledge of the diagnosis and treatment of MH problems was average,18 and pediatric residents describe ongoing discomfort and uncertainty with all aspects of MH visits.19 

Pediatric residents want to learn through active learning experiences, and several pediatric residency programs have successfully implemented interventions (eg, simulated patient experiences and role plays) that have impacted practices, such as addressing suicide and substance use disorder.19,21 Although more than half of pediatric residency programs train residents in primary care with on-site MH professionals, it is unknown whether this training environment impacts pediatric providers’ behaviors longer-term.22 Understanding which educational interventions are the most important, feasible, and readily implemented will be critical to creating the consensus needed to change the training paradigm.

Our goal in this study was to use the concept-mapping methodology to create a training framework to best prepare the pediatric workforce to provide MH care. Therefore, our objectives in this study were to (1) develop a concept map, gathering ideas on how to best improve training programs; (2) identify which ideas are important and feasible; and (3) assess the degree of consensus among stakeholders regarding the importance and feasibility of ideas generated.

We hypothesized that stakeholders will see an active learning approach as the most important change needed but the least feasible without substantial policy changes and new program initiatives. We also hypothesized there will be discernable differences between groups of stakeholders’ ratings regarding which ideas are more important and feasible. Results from this study will establish an empirically supported framework to guide action planning, program development, curricula, and policy.23,24 

Concept-Mapping Overview

Concept mapping is a reliable and valid mixed-methods design to generate and organize a comprehensive set of ideas and represent them visually as a map.23,24 It integrates input from multiple sources with differing content expertise or interest to ensure heterogeneity of perspectives. The process involves brainstorming ideas, structuring ideas (sorting and rating), and representing the ideas visually (see the Supplemental Information for details). All data were collected via a Web-based system by using Concept Systems software created for this process (Concept Systems Global Max).25 This study was deemed exempt by our institutional review board.

Research Team

The research team included a pediatrician (C.G.) with expertise in implementing and teaching MH in primary care, an expert in policy (W.T.) who created the concept-mapping methodology, an academic child and adolescent psychiatrist (J.T.W.), and a pediatrician with expertise in medical education (S.B.).

Participants

The research team identified potential participants to represent the variety of stakeholders relevant to MH training using nonrandom purposive sampling to maximize participation and help ensure heterogeneity of perspectives.23 There were 2 groups of participants: an invited group and a core group. The large and anonymous invited group brainstormed ideas and later rated them for importance and feasibility. The core group performed these tasks and sorted ideas.

C.G. identified stakeholders involved in MH initiatives and parents of children with chronic physical and/or MH problems in the ABP’s Family Leadership Advisory Group. J.T.W. and S.B. identified chairs and PDs in both disciplines who could recruit faculty and trainees. Together, the team identified academic and community clinicians of both disciplines working in a variety of geographic locations (CO, NY, SC, AL, OH, MD, WA, CA, PA, and IL) and practice settings.

We e-mailed and invited all 117 identified potential participants to participate in this project as part of the invited group. They were encouraged to share the invitation with colleagues and Listservs if deemed appropriate, using a snowball sampling approach. The core group of 76 participants was chosen from the initial list but mainly represented key stakeholders in national efforts around pediatric MH, educators, and trainees.

Invited participants brainstormed ideas by completing the following focus statement: “To prepare future pediatricians for their role in caring for children and adolescents with mental and behavioral health conditions, residency training needs to...” Statements collected from a national workshop on the “10 best ways” to enhance pediatric residency training in MH were added to brainstormed responses.13 Ideas were synthesized manually via content analysis by team members, who coded statements, grouped statements with similar codes, and created an aggregate statement to represent all statements in each code. The final set of statements was reviewed by members of the research team and core group to ensure the final set was comprehensive.

Sorting

Core group participants were asked to categorize the statements according to their view of their meaning or theme. After grouping similar statements into piles, they were asked to label each pile.

Rating

Core and invited participants rated the relative importance and feasibility of the ideas using a 5-point Likert-type scale for importance (1 = relatively unimportant; 5 = extremely important) and feasibility (1 = not at all feasible; 5 = extremely feasible).

Invited participants answered demographic questions: practice location (urban, rural, or suburban), practice type (academic, private, community, or not in professional practice), profession (pediatrician, psychiatrist, psychologist, social worker, nurse, nurse practitioner, policy maker, program developer, or other), years in practice (trainee, 0–5 years, 6–10 years, 11–15 years, 16–20 years, or >20 years), and if they were a parent (yes or no).

Using Concept Systems multidimensional scaling (MDS) and hierarchical cluster analysis created a point map, cluster map, bivariate “go zone,” and pattern match. The point map displayed the location of ideas (represented by a number); ideas with similar meanings were located closer to each other. The cluster map reflected the results of the cluster analysis and displayed the clustered ideas by enclosing them in polygon-shaped figures. Several cluster solutions were examined, ensuring merged ideas represented a single concept. The go-zone bivariate plot of idea ratings created a visual of ideas rated above average for both importance and feasibility. The pattern-match graph looked at the relationship between each cluster’s average ratings of importance compared with feasibility.

The research team conducted an initial review of data as the foundation for a facilitated videoconferencing session of a subgroup of the core group to review, interpret, and discuss use of results (Supplemental Information). The facilitator asked participants to read the ideas in each cluster and then generate a phrase to describe the set of statements. A discussion would follow until the group came to a consensus for an acceptable label for each cluster. The session ended by discussing which ideas should be easily implemented and where there were opportunities for innovation.

Descriptive statistics summarized the demographics and average statement and cluster ratings by using means and SDs. Pearson correlations measured correlations of a cluster’s importance and feasibility ratings. Paired-sample t tests compared averages between categories of the demographic variables of interest. Post hoc analyses using a Bonferroni correction factor of 40 accounted for multiple analyses for both ratings (ie, [7 clusters + overall average] × 5 demographic associations). We compared pediatricians to MH professionals by combining psychiatrists, psychologists, and social workers; nurses, nurse practitioners, and others were not included in the analyses for profession because their discipline was unknown. Practice location was dichotomized to urban versus rural and suburban. The years in practice category was dichotomized between trainees and all others. Academic practices were compared with private or community practices or respondents not in practice.

Snowball sampling yielded 244 participants who generated 279 brainstormed ideas that were supplemented by 97 statements from the workshop. These 376 ideas were edited, and compound statements were split to a final set of 499 distinct statements (Fig 1). Content analysis synthesized statements to a final set of 99. Two hundred and sixteen participants rated ideas for importance and 209 for feasibility. Of the participants who completed importance ratings, half work in pediatrics, more than half were trainees, and approximately two-thirds worked in urban and academic practices (Table 1). One-third were parents, and 56 of the 65 parents identified as professionals in the field. Of the 36% of respondents who answered “other” to profession, 86% were also trainees and likely answered “other” because they had not yet officially finished training in their discipline. Fifty-three percent (40 of 76) of the participants invited to the core group participated in sorting.

The final cluster map yielded 7 clusters (Fig 2). Table 2 lists the top 5 statements in each cluster ordered by their importance ratings.

The modalities for MH training cluster (8 ideas) includes teaching modalities needed to help trainees build clinical skills; creating longitudinal MH experiences was rated the most important and adapting curricula the most feasible.

The prioritization of MH cluster includes 18 ideas around training priorities and creating a culture in which MH is destigmatized and integrated into current clinical and teaching contexts, including collaborating with MH professionals. The most important statements were around culture change “…where MH is valued, de-stigmatized, and an integral part of all aspects of care,” integrating MH into the continuity clinic and providing clear expectations of the future pediatrician in MH care. Creating clear expectations of the pediatrician was the most feasible.

The systems-based practice cluster includes 10 ideas and refers to the larger system of MH and knowledge and use of resources in one’s practice and community. Using existing resources into clinical care was the most important and feasible statement.

The self-awareness and/or relationship building cluster includes 8 ideas and focuses on the internal and/or personal goals for the trainee, such as becoming more self-aware of their own stigma and biases. Promoting resilience and teaching coping skills was 1 of the most important and feasible statements.

The remaining 3 clusters describe the clinical skills identified as needed for pediatricians. The training in clinical assessment of patients cluster includes 24 ideas reflecting different aspects of assessment, including history taking and using screening tools. The training in treatment cluster includes 9 ideas about psychosocial and pharmacologic treatment options deemed appropriate for pediatricians. The diagnosis-specific skills cluster includes 21 ideas and describes skill development on specific MH diagnoses, including attention-deficit/hyperactivity disorder (ADHD), anxiety, and depression and/or self-harm, which were rated the most important.

Approximately one-third of all ideas were in the go zone with high importance and feasibility; the majority being clinical skills (Fig 3). Ideas with high importance yet rated below average in feasibility disproportionately come from the prioritization of MH cluster. See Supplemental Table 3 for each statement by cluster and zone.

The pattern match (Fig 4) reflects the lack of correlation (r < 0.9) for most clusters’ ratings of importance and feasibility. Statistically significant differences between importance and feasibility ratings were found for prioritization of MH, systems-based practice, training in clinical assessment of patients, and diagnosis-specific skills (P < .001 for each comparison).

Respondents living in suburban and rural areas rated each cluster higher in importance (P < .004) and feasibility (P = .004) compared with those living in an urban setting. MH professionals rated feasibility of prioritization of MH higher than pediatricians (4.3 vs 3.8 [95% confidence interval −0.73 to −0.18]; P = .04) but otherwise were in agreement with pediatricians. Trainees rated systems-based practice higher in importance (4.1 vs 3.7 [95% confidence interval −0.63 to −0.23]; P = .004) and every cluster higher in feasibility, although training in treatment did not reach statistical significance after post-hoc analysis.

After decades of efforts to address the gap in MH training,10 this is the first study to our knowledge that empirically gathered a comprehensive set of ideas on how to achieve this important goal. This 7-domain MH training framework, including 99 ideas generated from almost 250 participants, informs the future of training in pediatrics. The core concepts in the go zone are clinical skills addressing common MH problems that define the future pediatrician’s role in caring for children with MH problems. As hypothesized, there were common discrepancies between importance and feasibility in ideas and clusters and some differences between stakeholders on importance and feasibility ratings.

These data are more detailed than the American Academy of Pediatrics (AAP) 2009 MH competencies and the ABP’s EPA and include specific guidance not only on clinical skills but also approaches to training, professional development, and MH collaboration.14,26 These data also come from multiple stakeholders with different perspectives, providing a more holistic view than previous studies. For instance, although having a faculty champion was most important to PDs at the national workshop,13 it was less important in these data. Stakeholders may feel a single faculty champion may not alone be able to move forward without culture and policy change and PDs may not be in the front line and know what is needed for change.

The discord between importance and feasibility ratings may explain why advancing the agenda in MH training has been so challenging. Culture change was the most important idea but 1 of the least feasible, and few studies have successfully addressed practice cultures.27,28 This may explain why despite the development of several MH resources and curricula,29,31 there has been little progress in changing pediatricians’ practices.8 Additionally, culture change regarding the future pediatrician’s scope of practice involves multilevel interventions that address stigma, attitudes, and systems changes.5 However, only implementing highly feasible ideas will not catalyze change. For instance, implementing existing curricula (ie, from the Reach Institute32 or AAP30), rated above average in feasibility, without addressing culture change may impede uptake, as demonstrated by 1 learning collaborative.27,28 

Systemic barriers still exist that will make it challenging to implement highly important ideas. Despite half of training programs having an on-site MH professional in the continuity clinic,13,22 there is no financial structure to support them in a role deemed as most important: providing direct supervision and consulting, advising, and collaborating with faculty and trainees.33 Feasibility of integrating MH into the continuity clinic for longitudinal experiences with patients is further limited because the ACGME recently decreased the number of required sessions residents need to spend in the continuity clinic.12 

MH professionals rated prioritization of MH higher in importance, which is not surprising but is promising because most of the statements in this cluster will require their collaboration to become feasible. Previously, pediatric residents have identified many organizational barriers to integrating MH into their continuity clinic, which may explain why they rated systems-based practice higher in importance.19 Each cluster being rated higher in feasibility may be because they are not involved with the implementation of educational interventions and may not realize the resources needed for success. Participants working in rural and suburban settings rated all clusters higher for importance and feasibility compared with those in urban locations, which may reflect a greater need and urgency.34 Not only is there a shortage of MH professionals, but there is also a maldistribution, with the majority practicing in urban locations. This may explain their importance ratings but may not explain feasibility, and this warrants further study to inform implementation throughout the country.

Arguably, concepts in the go zone should be implemented immediately by programs by using resources that already exist. More specifically, addressing depression and better dissemination of existing underused resources are both in the go zone. Faculty can teach residents by reviewing the AAP’s guidelines for adolescent depression35 and watching a sample motivational interview where a provider engages an adolescent with depression, which can be found on the AAP website.29 Adhering to these guidelines and implementing depression screening in the continuity clinic will then provide experiential opportunities. However, once depression screening is implemented, trainees will need longitudinal experiences and mentorship from experienced practitioners to respond to positive screen results and appropriately manage their patients. Although addressing anxiety was in the go zone, there are currently no clinical guidelines for pediatricians, which may impede implementation.

Developing and studying interventions to make important ideas feasible are needed to catalyze change. Not only do programs need to engage local and national leadership for culture change, but attitudes around MH need to be addressed through strategies that facilitate reflection on stigma and biases.36 The ABP recently engaged 13 PD–department chair dyads at a stakeholder meeting around MH training, which is promising, but also needs to reach all programs and leadership.37 Changing training priorities to include MH to the same degree as other common childhood illness will not happen without changing ACGME requirements. Mandated time in specific areas (inpatient versus continuity clinic) needs to reflect the changing epidemiology of pediatrics.3 Reconsidering the decrease in required continuity clinic sessions can provide residents with more longitudinal experiences and maximize exposure to MH professionals because half of continuity clinics have one on-site.22 Lastly, clinical guidelines for anxiety and all skills listed in the go zone need to be developed.

Despite using purposive and snowball sampling, it is possible not all important stakeholders were reached, which could limit generalizability. The sample only included 24 MH professionals, 35 participants in nonacademic institutes, and 47 participants who had >10 years’ experience, making it possible that we did not have a large enough sample to compare differences between these stakeholders, and input from them may have been under-represented. However, a pooled study analysis of 69 studies that assessed the rigor of concept mapping found the average sample size for the first Web-based rating to be 112.8 and 75.6 for the second rating. Our sample sizes of 217 and 209 were much larger, suggesting we had an adequate absolute number of participants to ensure heterogeneity of perspectives.24 Because similar statements were synthesized, if only 1 stakeholder of a certain demographic responded with an idea that was different from others, it would have been included as 1 of the 99 final ideas used in this project. Future study should include input from payers and children with MH problems who were not included in this sample. Lastly, there are small differences in ratings between statements, suggesting that merely focusing on the go-zone statements may not be the best strategy, as illustrated throughout this discussion.

The study results can drive future planning in advancing MH training in pediatrics. Ideas that were deemed feasible and important can help drive curricular interventions by using resources that already exist. Strategies are needed for culture and policy change, and innovation is needed to make important ideas more feasible, such as creating and studying sustainable models integrating MH professionals into training programs. Specifically, we recommend the following:

  1. embrace MH as an integral part of all aspects of care to change culture and reduce stigma;

  2. implement existing curricula, such as from the AAP or Reach Institute, with an emphasis on ADHD, anxiety, depression, and suicide and/or self-harm;

  3. integrate MH care into continuity clinics by creating systems for screening for early risk factors, performing diagnostic assessments, and providing first-line treatment;

  4. ensure trainees have longitudinal experiences with children with MH problems, and track these experiences;

  5. train pediatric role models to precept residents;

  6. provide on-site MH professionals to precept trainees alongside pediatric attending physicians;

  7. train MH professionals in the most effective models of working with their pediatric colleagues; and

  8. advocate for financial structures and/or incentives to enable pediatricians and MH professionals to address the return on investing in integrated care models.

Dr Green was the principal investigator for this study, was involved in all aspects of the study from its creation, developed and pilot tested the focus prompt, recruited stakeholders and acquired data from them through the World Wide Web, edited and synthesized the final set of statements, analyzed and interpreted data, facilitated the formal data interpretation session, and wrote the first draft of the article; Dr Trochim was involved with all aspects of the project’s design, including developing the focus prompt, editing and synthesizing statements, creating the maps and graphs, analyzing and interpreting the data, and critically revising the manuscript for important intellectual content; Dr Walkup contributed to the design by helping in the creation of the focus prompt, recruiting stakeholders, editing and synthesizing the final set of statements, interpreting data, and critically revising the manuscript for important intellectual content; Dr Bostwick helped develop and pilot test the focus statement and identify potential participants, reviewed the final set of statements to ensure they were comprehensive and the initial interpretation of data, and critically revised the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Weill Cornell Medicine Clinical and Translational Science Center KL2 Scholars Program (National Institutes of Health, National Center for Advancing Translational Sciences grant KL2-TR-002385). Funded by the National Institutes of Health (NIH).

We acknowledge all participants in the invited and core group. We also acknowledge Dr Julia A. McMillan, who provided feedback throughout the process and was part of the interpretation session; Drs John Duby, Lawrence Wissow, Marshall Land, Elena Mann, and David Kaye for their participation in the interpretation of the results session; and Dr Paul Christos for his statistical support.

AAP

American Academy of Pediatrics

ABP

American Board of Pediatrics

ACGME

Accreditation Committee on Graduate Medical Education

ADHD

attention-deficit/hyperactivity disorder

DB

development and behavior

EPA

entrustable professional activity

MDS

multidimensional scaling

MH

mental health

PD

program director

1
McMillan
JA
,
Land
M
 Jr
,
Leslie
LK
.
Pediatric residency education and the behavioral and mental health crisis: a call to action.
Pediatrics
.
2017
;
139
(
1
):
e20162141
2
Copeland
WE
,
Shanahan
L
,
Davis
M
,
Burns
BJ
,
Angold
A
,
Costello
EJ
.
Increase in untreated cases of psychiatric disorders during the transition to adulthood.
Psychiatr Serv
.
2015
;
66
(
4
):
397
403
3
Houtrow
AJ
,
Larson
K
,
Olson
LM
,
Newacheck
PW
,
Halfon
N
.
Changing trends of childhood disability, 2001-2011.
Pediatrics
.
2014
;
134
(
3
):
530
538
4
Perrin
JM
,
Asarnow
JR
,
Stancin
T
,
Melek
SP
,
Fritz
GK
.
Mental health conditions and health care payments for children with chronic medical conditions.
Acad Pediatr
.
2019
;
19
(
1
):
44
50
5
American Academy of Child and Adolescent Psychiatry Committee on Health Care Access and Economics Task Force on Mental Health
.
Improving mental health services in primary care: reducing administrative and financial barriers to access and collaboration [published correction appears in Pediatrics. 2009;123(6):1611].
Pediatrics
.
2009
;
123
(
4
):
1248
1251
6
Whitney
DG
,
Peterson
MD
.
US national and state-level prevalence of mental health disorders and disparities of mental health care use in children [published online ahead of print February 11, 2019].
JAMA Pediatr
. doi:10.1001/jamapediatrics.2018.5399
7
Anderson
LE
,
Chen
ML
,
Perrin
JM
,
Van Cleave
J
.
Outpatient visits and medication prescribing for US children with mental health conditions.
Pediatrics
.
2015
;
136
(
5
). Available at: www.pediatrics.org/cgi/content/full/136/5/e1178
8
Stein
RE
,
Storfer-Isser
A
,
Kerker
BD
, et al
.
Beyond ADHD: how well are we doing?
Acad Pediatr
.
2016
;
16
(
2
):
115
121
9
Horwitz
SM
,
Storfer-Isser
A
,
Kerker
BD
, et al
.
Barriers to the identification and management of psychosocial problems: changes from 2004 to 2013.
Acad Pediatr
.
2015
;
15
(
6
):
613
620
10
Haggerty
RJ
.
The changing role of the pediatrician in child health care.
Am J Dis Child
.
1974
;
127
(
4
):
545
549
11
Leslie
L
,
Rappo
P
,
Abelson
H
, et al
.
Final report of the FOPE II pediatric generalists of the future workgroup.
Pediatrics
.
2000
;
106
(
5
):
1199
1223
12
Accreditation Council on Graduate Medical Education
. ACGME program requirements for graduate medical education in pediatrics. 2016. Available at: www.acgme.org/Portals/0/PFAssets/ProgramRequirements/320_pediatrics_2017-07-01.pdf?ver=2017-06-30-083432-507. Accessed November 3, 2016, and August 15, 2017
13
McMillan
JA
,
Land
ML
,
Rodday
AM
,
Wills
K
,
Green
CM
,
Leslie
LK
.
Report of a joint Association of Pediatric Program Directors-American Board of Pediatrics workshop: preparing future pediatricians for the mental health crisis.
J Pediatr
.
2018
;
201
:
285
291
14
American Board of Pediatrics
. Entrustable professional activities. Available at: https://www.abp.org/entrustable-professional-activities-epas. Accessed February 3, 2016
15
Green
C
,
Storfer-Isser
A
,
Stein
REK
, et al
.
Which pediatricians comanage mental health conditions?
Acad Pediatr
.
2017
;
17
(
5
):
479
486
16
Stein
RE
,
Storfer-Isser
A
,
Kerker
BD
, et al
.
Does length of developmental behavioral pediatrics training matter?
Acad Pediatr
.
2017
;
17
(
1
):
61
67
17
Stein
RE
.
Are we on the right track? Examining the role of developmental behavioral pediatrics.
Pediatrics
.
2015
;
135
(
4
):
589
591
18
Green
C
,
Hampton
E
,
Ward
MJ
,
Shao
H
,
Bostwick
S
.
The current and ideal state of mental health training: pediatric program director perspectives.
Acad Pediatr
.
2014
;
14
(
5
):
526
532
19
Hampton
E
,
Richardson
JE
,
Bostwick
S
,
Ward
MJ
,
Green
C
.
The current and ideal state of mental health training: pediatric resident perspectives.
Teach Learn Med
.
2015
;
27
(
2
):
147
154
20
Schram
P
,
Harris
SK
,
Van Hook
S
, et al
.
Implementing adolescent screening, brief intervention, and referral to treatment (SBIRT) education in a pediatric residency curriculum.
Subst Abus
.
2015
;
36
(
3
):
332
338
21
Fallucco
EM
,
Conlon
MK
,
Gale
G
,
Constantino
JN
,
Glowinski
AL
.
Use of a standardized patient paradigm to enhance proficiency in risk assessment for adolescent depression and suicide.
J Adolesc Health
.
2012
;
51
(
1
):
66
72
22
Shahidullah
JD
,
Kettlewell
PW
,
Palejwala
MH
, et al
.
Behavioral health training in pediatric residency programs: a national survey of training directors.
J Dev Behav Pediatr
.
2018
;
39
(
4
):
292
302
23
Trochim
W
,
Kane
M
.
Concept mapping: an introduction to structured conceptualization in health care.
Int J Qual Health Care
.
2005
;
17
(
3
):
187
191
24
Rosas
SR
,
Kane
M
.
Quality and rigor of the concept mapping methodology: a pooled study analysis.
Eval Program Plann
.
2012
;
35
(
2
):
236
245
25
Concept Systems
.
The Concept System Global Max. (Build 2016.046.12) [Web-based platform]
.
Ithaca, NY
:
Concept Systems
;
2016
. Available at: www.conceptsystemsglobal.com. Accessed July 8, 2019
26
Committee on Psychosocial Aspects of Child and Family Health and Task Force on Mental Health
.
Policy statement–the future of pediatrics: mental health competencies for pediatric primary care.
Pediatrics
.
2009
;
124
(
1
):
410
421
27
Baum
RA
,
King
MA
,
Wissow
LS
.
Outcomes of a statewide learning collaborative to implement mental health services in pediatric primary care.
Psychiatr Serv
.
2019
;
70
(
2
):
123
129
28
King
MA
,
Wissow
LS
,
Baum
RA
.
The role of organizational context in the implementation of a statewide initiative to integrate mental health services into pediatric primary care.
Health Care Manage Rev
.
2018
;
43
(
3
):
206
217
29
American Academy of Pediatrics
. Mental health initiatives: implementing mental health priorities in practice. Available at: https://www.aap.org/en-us/advocacy-and-policy/aap-health-initiatives/Mental-Health/Pages/depression.aspx. Accessed May 17, 2019
30
American Academy of Pediatrics
. Mental health initiatives: residency curriculum. 2014. Available at: www.aap.org/en-us/advocacy-and-policy/aap-health-initiatives/Mental-Health/Pages/Residency-Curriculum.aspx. Accessed January 16, 2015
31
American Academy of Pediatrics Task Force on Mental Health
. Addressing mental health care in primary care: a clinician’s toolkit. 2010. Available at: https://www.aap.org/en-us/advocacy-and-policy/aap-health-initiatives/Mental-Health/Pages/Addressing-Mental-Health-Concerns-in-Primary-Care-A-Clinicians-Toolkit.aspx. Accessed July 8, 2019
32
The REACH Institute
. Child and adolescent training institute in evidence-based psychotherapies. Available at: www.thereachinstitute.org/services/for-healthcare-organizations/staff-training/child-adolescent-training-in-evidence-based-psychotherapies-catie-1. Accessed October 21, 2016
33
Ader
J
,
Stille
CJ
,
Keller
D
,
Miller
BF
,
Barr
MS
,
Perrin
JM
.
The medical home and integrated behavioral health: advancing the policy agenda.
Pediatrics
.
2015
;
135
(
5
):
909
917
34
Thomas
CR
,
Holzer
CE
 III
.
The continuing shortage of child and adolescent psychiatrists.
J Am Acad Child Adolesc Psychiatry
.
2006
;
45
(
9
):
1023
1031
35
Zuckerbrot
RA
,
Cheung
A
,
Jensen
PS
,
Stein
REK
,
Laraque
D
;
GLAD-PC Steering Group
.
Guidelines for Adolescent Depression in Primary Care (GLAD-PC): part I. Practice preparation, identification, assessment, and initial management.
Pediatrics
.
2018
;
141
(
3
):
e20174081
36
Avery
J
,
Zerbo
E
.
Improving psychiatry residents’ attitudes toward individuals diagnosed with substance use disorders.
Harv Rev Psychiatry
.
2015
;
23
(
4
):
296
300
37
American Board of Pediatrics and The National Academy of Sciences
. Engineering, medicine behavioral and mental health stakeholder meeting: “preparing future pediatricians to meet the behavioral and mental health needs of children”. 2018. Available at: https://www.abp.org/sites/abp/files/pdf/abpf-nasem-agenda-book-2018.pdf. Accessed August 2, 2018

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data