BACKGROUND AND OBJECTIVES

Because of severe and protracted shortages of pediatric behavioral health (BH) specialists, collaboration between pediatric primary care practitioners (PCPs) and BH specialists has the potential to increase access to BH services by expanding the BH workforce. In a previous study, we demonstrated that phase 1 of a behavioral health integration program (BHIP) enrolling 13 independently owned, community-based pediatric practices was associated with increased access to BH services while averting substantial cost increases and achieving high provider self-efficacy and professional satisfaction. The current study was undertaken to assess whether the initial access findings were replicated over 4 subsequent implementation phases and to explore the practicality of broad dissemination of the BHIP model.

METHODS

After phase 1, BHIP was extended over 4 subsequent phases in a stepped-wedge design to 46 additional pediatric practices, for a total cohort of 59 practices (354 PCPs serving >300 000 patients). Program components comprised BH education and consultation and support for integrated practice transformation; these components facilitated on-site BH services by an interprofessional BH team. Outcomes were assessed quarterly, preprogram and postprogram launch.

RESULTS

Across combined phases 1 to 5, BHIP was associated with increased primary care access to BH services (screening, psychotherapy, PCP BH visits, psychotropic prescribing) and performed well across 7 standard implementation outcome domains (acceptability, appropriateness, feasibility, fidelity, adoption, penetration, and sustainability). Emergency BH visits and attention-deficit/hyperactivity disorder prescribing were unchanged.

CONCLUSIONS

These findings provide further support for the potential of integrated care to increase access to BH services in pediatric primary care.

An estimated 1 in 5 youth in the United States has a functionally impairing psychiatric disorder,1  yet less than half of these youth receive appropriate treatment.2  Absent or inadequate treatment of child psychiatric disorders leads to significant morbidity and mortality across the life span.3  Because of the severe, protracted shortage and maldistribution of child-trained behavioral health (BH) specialists,4,5  for several decades, both the American Academy of Pediatrics 611  and the American Academy of Child and Adolescent Psychiatry 1113  have endorsed the management of mild and/or moderate presentations of common psychiatric disorders (anxiety, depression, and attention-deficit/hyperactivity disorder [ADHD]) in pediatric primary care, thereby conserving BH specialists for the management of severe and/or complex disorders. Yet despite abundant supports,1421  pediatric primary care practitioners (PCPs) have made modest progress in routinely identifying and treating psychiatric disorders,22  consistently citing training, confidence, time, and reimbursement challenges.23 

Collaborative models of care have the potential to attenuate the challenges of managing BH problems in primary care by creating partnerships between PCPs and BH specialists. Collaborative models span a broad spectrum of delivery modalities, from minimal collaboration at a distance to full collaboration in an integrated medical home.24  Although integrated collaborative models have a robust evidence base for improving medical and BH outcomes and reducing costs in adult populations,2528  empirical support in pediatric populations is more limited yet promising. Randomized clinical trials of pediatric integrated collaborative care targeted at specific child and adolescent psychiatric disorders have shown significant improvement in clinical symptoms as well as secondary outcomes (eg, function, quality of life, patient satisfaction, and cost).2932 

Although the potential of integrated collaborative care has been demonstrated in both adult and pediatric primary care settings, evidence of the practicality and impact of large-scale dissemination of these models has been sparse and largely limited to adult populations.33  Therefore, in 2013 we sequentially undertook the development, pilot implementation, and broad dissemination of a behavioral health integration program (BHIP) in an 84-practice pediatric primary care association in Massachusetts. The BHIP is an integrated collaborative care model aimed to increase access to BH care in pediatric primary care settings, in which the “primary care advantage”34  can be realized. Consistent with American Academy of Child and Adolescent Psychiatry recommendations,12  the BHIP comprises 3 key components: BH education, BH consultation, and clinical and operational support for integrated practice transformation. These components were designed to facilitate on-site delivery of BH services by an integrated interprofessional team composed of PCPs, behavioral health clinicians (BHCs) (primarily clinical social workers), and medical home care coordinators (CCs).

We previously reported that the BHIP pilot implementation (phase 1, 13 practices) was associated with increased BH integration and increased access to BH services (screening, assessment, psychotropic prescribing, and psychotherapy) while averting substantial cost increases and achieving high provider BH self-efficacy and professional satisfaction.35  In the current study, a stepped-wedge design was used to ascertain whether the favorable phase 1 access-to-care findings were replicated across 4 subsequent implementation phases enrolling 46 additional pediatric practices. A secondary aim was to examine the practicality of the broad dissemination of BHIP as measured by 7 standard implementation outcome domains (acceptability, appropriateness, feasibility, fidelity, adoption, penetration, and sustainability).36 

The eligible BHIP population comprises 84 practice members of a statewide association of community-based, independently owned pediatric practices affiliated with an academic medical center. The practices comprise >400 PCPs serving >350 000 patients.

After notification of the BHIP launch and participation requirements through usual communication channels, to date, 78 practices (93%) have voluntarily agreed to participate. Program participation requires that practices (1) designate ≥1 PCP (physician, nurse practitioner, and/or physician assistant [BH champions]) and ≥1 additional clinical (BHC) and/or office staff to attend the education component and disseminate learned information throughout the practice; (2) use the consultation component as needed; (3) engage in learning about integrated practice transformation; and (4) provide on-site clinical BH services. Figure 1 outlines goals and action items guiding BHIP development and implementation.

FIGURE 1

BH integration in pediatric primary care driver diagram. a An integrated team at the pediatric primary care site comprising PCPs, BHCs, and CCs, supported by CAP and clinical and operational consultants.

FIGURE 1

BH integration in pediatric primary care driver diagram. a An integrated team at the pediatric primary care site comprising PCPs, BHCs, and CCs, supported by CAP and clinical and operational consultants.

Close modal

A stepped-wedge design enables graduated enrollment (10–15 practices per year) of the 78 participating practices into consecutive BHIP phases. Enrollment was nonrandomized; rather, practices entered phases in accordance with their readiness to meet all participation requirements. BHIP enrollment began in July 2013; once a practice is enrolled, participation is ongoing. Because the program is consistent with our institution’s definition of a quality improvement program, the need for individual informed consent was waived.

The sample for this report comprises the first 5 BHIP implementation phases (59 practices, with 354 PCPs [70% physicians, 29% nurse practitioners, and 1% physician assistants], serving >300 000 patients). The 59 practices range in size from 1 to 34 PCPs serving 464 to 28 369 patients; 62% are located in inner core and/or regional urban centers and 38% in mature and/or maturing suburbs.37  Overall, the practices’ patients broadly represent the sociodemographic characteristics of Massachusetts (71% white, 9% Black, 12% Hispanic, and 7% Asian American; mean per capita income: $41 794).38  Approximately 75% of the practices’ patients are commercially insured; the remainder are insured by Medicaid. Characteristics of each implementation phase are presented in Table 1.

TABLE 1

BHIP Phase Characteristics

Phase 1Phase 2Phase 3Phase 4Phase 5
Phase start date July 2013 September 2014 June 2015 June 2016 June 2017 
No. practices per phase 13 10 14 12 10 
Average No. PCPs per practice 10.4 (6.9 MDs, 3.5 NPs) 5.1 (3.1 MDs, 2.0 NPs and/or PAs) 4.4 (3.4 MDs, 1.0 NPs) 5.9 (4.5 MDs, 1.4 NPs) 6.4 (3.7 MDs, 2.7 NPs) 
Average practice patient panel 7765 4037 3195 4726 5012 
Average No. patients per PCP 747 792 726 801 783 
Practice locationa 50% urban; 50% suburban 60% urban; 40% suburban 72% urban; 28% suburban 67% urban; 33% suburban 60% urban; 40% suburban 
Average percentage of white residents in practice communities38  81.8 78.4 75.8 80.5 78.6 
Average percentage of patients with public insurance 20.2 13.2 24.4 23.4 27.7 
Average No. CCs per practice 1.0 1.0 1.0 1.0 1.0 
Average No. BHCs per practice 1.9 0.4 0.6 0.4 0.7 
Average No. PCPs in practice enrolled in the learning community 2.5 2.7 1.6 1.8 2.2 
Average percentage of practices enrolled in the child psychiatry consultation programb 100 100 86 92 100 
Percentage of practices >20 miles from affiliated academic medical center 53.9 50.0 21.5 58.4 70.0 
Percentage of practices located in high or severe child psychiatry shortage areac 50.0 40.0 21.5 41.7 80.0 
Phase 1Phase 2Phase 3Phase 4Phase 5
Phase start date July 2013 September 2014 June 2015 June 2016 June 2017 
No. practices per phase 13 10 14 12 10 
Average No. PCPs per practice 10.4 (6.9 MDs, 3.5 NPs) 5.1 (3.1 MDs, 2.0 NPs and/or PAs) 4.4 (3.4 MDs, 1.0 NPs) 5.9 (4.5 MDs, 1.4 NPs) 6.4 (3.7 MDs, 2.7 NPs) 
Average practice patient panel 7765 4037 3195 4726 5012 
Average No. patients per PCP 747 792 726 801 783 
Practice locationa 50% urban; 50% suburban 60% urban; 40% suburban 72% urban; 28% suburban 67% urban; 33% suburban 60% urban; 40% suburban 
Average percentage of white residents in practice communities38  81.8 78.4 75.8 80.5 78.6 
Average percentage of patients with public insurance 20.2 13.2 24.4 23.4 27.7 
Average No. CCs per practice 1.0 1.0 1.0 1.0 1.0 
Average No. BHCs per practice 1.9 0.4 0.6 0.4 0.7 
Average No. PCPs in practice enrolled in the learning community 2.5 2.7 1.6 1.8 2.2 
Average percentage of practices enrolled in the child psychiatry consultation programb 100 100 86 92 100 
Percentage of practices >20 miles from affiliated academic medical center 53.9 50.0 21.5 58.4 70.0 
Percentage of practices located in high or severe child psychiatry shortage areac 50.0 40.0 21.5 41.7 80.0 

BHCs include clinical social workers, clinical psychologists, and mental health counselors. MD, medical doctor; NP, nurse practitioner; PA, physician assistant.

a

Urban: metropolitan core communities, streetcar suburbs, major regional urban centers, and subregional urban centers; suburban: mature suburban towns, established suburbs and Cape Cod towns, maturing New England towns, and country suburbs.37 

b

MCPAP.39 

c

One to 46 child psychiatrists per 100 000 population.5 

BH Teams

The practice-based BH teams comprise PCPs, BHCs hired by practices after program launch, and practices’ medical home CCs. These on-site teams are supported by the off-site BHIP clinical and/or operational team, which in turn is supported by the executive leadership of the affiliated entities and an external partnership with the Massachusetts Child Psychiatry Access Program (MCPAP)39  (Table 2).

TABLE 2

Key BHIP Team Members, Collaborating Partners, and Executive Leadership

Personnel and/or AffiliationRoleFunding Source
BHIP pediatric practice teams   
 PCPs; n = 354 Practice-based BH services (screening, focused assessment, guided self-management, psychopharmacology, and triage to specialty care) Billed revenues 
 BHCs; n = 37 Practice-based BH services (focused assessment and focused psychotherapy) Billed revenues 
 CCs; n = 59 Practice-based care coordination and/or case management Practice revenues 
BHIP clinical and/or operational team   
 Program manager (clinical psychologist); 1.0 FTE Overall BHIP oversight; learning-community faculty; and operational consultation to practices Grant funding 
 Associate medical director for BH, pediatric practice association; medical codirector, MCPAP (CAP); 0.55 FTE Overall BHIP clinical oversight; lead learning-community faculty; regional MCPAP clinical oversight; clinical consultation to PCPs Grant funding (0.2 FTE)MCPAP funding (0.35 FTE) 
 Integration managers (clinical social workers): 2.0 FTE Learning-community faculty; clinical consultation to BHCs; operational consultation to practices Grant funding 
 Quality improvement consultant: 0.1 FTE Educational and operational support Grant funding 
 Other learning-community faculty from affiliated academic medical center (CAPs [n = 2], developmental behavioral pediatricians [n = 2]) Learning-community faculty per scheduled sessions Grant funding 
MCPAP team   
 Other clinical consultants to PCPs (CAPs [n = 7], program coordinators and/or resource specialists [n = 3]) Clinical consultation per scheduled shifts; consultation scheduling; and resource consultation MCPAP funding 
Executive leadership   
 Chief executive officer, pediatric practice association (pediatrician) Overall pediatric association oversight Contributed 
 Medical director, pediatric practice association (pediatrician) Overall pediatric practice association clinical oversight Contributed 
 Psychiatrist-in-chief, affiliated academic medical center (CAP) Overall department of psychiatry oversight Contributed 
Personnel and/or AffiliationRoleFunding Source
BHIP pediatric practice teams   
 PCPs; n = 354 Practice-based BH services (screening, focused assessment, guided self-management, psychopharmacology, and triage to specialty care) Billed revenues 
 BHCs; n = 37 Practice-based BH services (focused assessment and focused psychotherapy) Billed revenues 
 CCs; n = 59 Practice-based care coordination and/or case management Practice revenues 
BHIP clinical and/or operational team   
 Program manager (clinical psychologist); 1.0 FTE Overall BHIP oversight; learning-community faculty; and operational consultation to practices Grant funding 
 Associate medical director for BH, pediatric practice association; medical codirector, MCPAP (CAP); 0.55 FTE Overall BHIP clinical oversight; lead learning-community faculty; regional MCPAP clinical oversight; clinical consultation to PCPs Grant funding (0.2 FTE)MCPAP funding (0.35 FTE) 
 Integration managers (clinical social workers): 2.0 FTE Learning-community faculty; clinical consultation to BHCs; operational consultation to practices Grant funding 
 Quality improvement consultant: 0.1 FTE Educational and operational support Grant funding 
 Other learning-community faculty from affiliated academic medical center (CAPs [n = 2], developmental behavioral pediatricians [n = 2]) Learning-community faculty per scheduled sessions Grant funding 
MCPAP team   
 Other clinical consultants to PCPs (CAPs [n = 7], program coordinators and/or resource specialists [n = 3]) Clinical consultation per scheduled shifts; consultation scheduling; and resource consultation MCPAP funding 
Executive leadership   
 Chief executive officer, pediatric practice association (pediatrician) Overall pediatric association oversight Contributed 
 Medical director, pediatric practice association (pediatrician) Overall pediatric practice association clinical oversight Contributed 
 Psychiatrist-in-chief, affiliated academic medical center (CAP) Overall department of psychiatry oversight Contributed 

PCPs include physicians, nurse practitioners, and physician assistants. BHCs include clinical social workers, clinical psychologists, and mental health counselors. FTE, full-time equivalent.

Education

The core didactic BHIP education component (behavioral health learning community [BHLC]) was delivered by affiliated academic medical center faculty to practice-based BH teams in 10 1- or 2-hour sessions (17 hours total), primarily in the first enrollment year. Most sessions were delivered in person in a geographically central location, with several sessions delivered by televideo. Twenty category 1 continuing medical education and 25 type IV maintenance of certification credits were offered to physician BHLC participants through the affiliated medical school; discipline-specific credits were also offered to other professionals.

Targeted at key BH competencies for pediatricians delineated by the American Academy of Pediatrics,10  the core didactic sessions addressed the purposes and processes of collaborative care; the stepped-care model (Fig 2) universal BH screening40,41 ; focused assessment of BH problems, including the use of symptom rating scales4245 ; phenomenology, etiology, and management of mild and/or moderate presentations of the target disorders (anxiety, depression, and ADHD) and related problems (stress-trauma reactions, disruptive behavior, and suicide); guideline-congruent, first-line medications for target disorders18,46,47 ; focused psychotherapy48 ; guided self-management (patient and/or family), with follow-up for subclinical problems; and referral to specialty BH care for severe, complex, unsafe, and/or refractory presentations. BHCs received 8 hours/year of additional didactic sessions targeted at their specific learning needs. Didactic sessions were supported by experiential learning in the form of optional bimonthly 1-hour interprofessional televideo case conferences.

FIGURE 2

BHIP stepped-care model.

FIGURE 2

BHIP stepped-care model.

Close modal

Consultation

The BHIP consultation component was designed to broaden experiential learning by reinforcing and extending the knowledge acquired in the BHLC to the management of individual patients. Integration managers provided BHCs with ∼1 to 2 hours/month of individual telephonic consultation and 1 hour/month of televideo case consultations. Child and adolescent psychiatrists (CAPs) provided telephone consultations to PCPs on demand 8 hours/day, 5 days/week.

CAP telephonic consultations provided initial psychiatric assessment and treatment suggestions and ongoing medication management support, as indicated. More complex cases were assessed in person by CAPs at the affiliated medical center, with report of findings to PCPs. For severe and/or complex presentations, CAPs facilitated referral to specialty BH care at the affiliated medical center or in the community, per family preference, or provided interim treatment or cotreatment until stable. To provide continuity of learning, in the first 3 years of the BHIP, consultation was provided by CAP BHLC faculty. In subsequent years, to ensure fiscal sustainability, consultation was provided through a partnership between the affiliated department of psychiatry and the MCPAP.39  To standardize CAP consultation, the MCPAP disseminated the BHIP care pathways for anxiety, depression, and ADHD to all MCPAP CAPs and all PCPs in the state.39 

Integrated Practice Support

Program and integration managers and quality improvement consultants provided ∼10 hours/year of in-person or televideo support to PCPs, CCs, and other practice staff. These group sessions addressed clinical and business workflows; billing and revenue cycle management; BHC hiring, contracting, and/or credentialing; crisis plans; linkages to specialty services; electronic health record (EHR) documentation and decision support; and support for practice-individualized quality improvement projects. The sessions were supplemented by ∼3 hours/month of individualized practice-based support.

On-site Clinical BH Services

On-site, billable clinical services comprised BH screening by PCPs; BH assessment and treatment visits to PCPs and BHCs; and PCP prescription of psychotropic medications for anxiety, depression, and ADHD; unbilled BH care coordination was provided by CCs.

BHIP participation data were derived from electronic BHIP and MCPAP use records. BH visits to PCPs and BHCs, psychotropic prescribing by PCPs, and BH visits to hospital emergency departments (EDs) were derived from paid insurance claims from a single large commercial insurance company that shares data with the practice association (Table 3 [billing and diagnostic codes] and Table 4 [prescribed medications]). Screening and BHC use data were compiled from deidentified practice EHRs. Implementation outcomes were operationalized according to standard definitions (Table 5),36  and findings were aggregated from previous BHIP publications35,49  and from the current study.

TABLE 3

Billing (CPT) and Diagnostic (International Classification of Diseases) Codes Included in Analyses

CPT CodesLogical ConjunctionICD Codes (Primary Diagnosis)
Diagnostic evaluation and psychotherapy visits 90791, 90832, 90834, 90837, 90839, 90845, 90846, 90847, 90849, 90853, 90875, 90876, 90880 And Any 
PCP BH visits 99211, 99212, 99213, 99214, 99215 And ICD-9: 290.xx–298.xx, 300.xx–314.xx, 316 
   ICD-10: F1x–F6x, F9x 
ED BH visits 99281, 99282, 99283, 99284, 99285, 99286, 99287, 99288 And ICD-9: 290.xx-298.xx, 300.xx-314.xx, 316 
   ICD-10: F1x–F6x, F9x 
CPT CodesLogical ConjunctionICD Codes (Primary Diagnosis)
Diagnostic evaluation and psychotherapy visits 90791, 90832, 90834, 90837, 90839, 90845, 90846, 90847, 90849, 90853, 90875, 90876, 90880 And Any 
PCP BH visits 99211, 99212, 99213, 99214, 99215 And ICD-9: 290.xx–298.xx, 300.xx–314.xx, 316 
   ICD-10: F1x–F6x, F9x 
ED BH visits 99281, 99282, 99283, 99284, 99285, 99286, 99287, 99288 And ICD-9: 290.xx-298.xx, 300.xx-314.xx, 316 
   ICD-10: F1x–F6x, F9x 

CPT, Current Procedural Terminology; ICD-9, International Classification of Diseases, Ninth Revision; ICD-10, International Classification of Diseases, 10th Revision.

TABLE 4

Guideline-Congruent Medications Included in Analyses

Target DisordersMedication Classes
Medications for anxiety and depression SSRIs 
Medications for ADHD Stimulants, α-agonists, and atomoxetine 
Target DisordersMedication Classes
Medications for anxiety and depression SSRIs 
Medications for ADHD Stimulants, α-agonists, and atomoxetine 

Adapted from the following. (1) Wolraich ML, Hagan JF Jr, Allan C, et al; Subcommittee on Children and Adolescents with Attention-Deficit/Hyperactive Disorder. Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents. Pediatrics. 2019;144(4):e20192528. (2) Walter HJ, Bukstein OG, Abright AR, et al. Clinical practice guideline for the assessment and treatment of children and adolescents with anxiety disorders. J Am Acad Child Adolesc Psychiatry. 2020;59(10):1107–1124. (3) Birmaher B, Brent D, Bernet W, et al; AACAP Work Group on Quality Issues. Practice parameter for the assessment and treatment of children and adolescents with depressive disorders. J Am Acad Child Adolesc Psychiatry. 2007;46(11):1503–1526.

TABLE 5

Implementation Outcomes for the BHIP

Implementation Domain36 BHIP Implementation Evidence35,49  (This Publication)
Acceptability (stakeholders’ satisfaction with the intervention) >90% of surveyed PCPs (n = 66, measured in phases 1–3) reported that BHIP participation achieved the following: 
  Increased their knowledge about symptom rating scales, guided self-management, psychotropic medications, and level-of-care decisions 
  Imparted greater confidence in their ability to manage BH problems 
  Improved the quality of their BH care 
Appropriateness (perceived fit of the intervention with the practice setting) Close match between disorders targeted for BHIP education (anxiety, depression, and ADHD) and those most commonly presenting in PCPs practices 
 After CAP consultation, disposition predominantly to PCPs for ongoing management 
Feasibility (extent to which the intervention can be successfully conducted in the practice setting) 100% of surveyed phase 1 practices (n = 12) achieved all integration readiness domains (leadership, resources, administrative mechanisms, screening, clinical management, family centeredness, care coordination, and quality improvement). 
 100% of phase 1–5 practices (n = 59) participated in BHIP program components (education, consultation, and integrated practice transformation). 
 35% of phase 1–5 PCPs (n = 125) participated in the didactic learning-community sessions. 
 44% of phase 1–5 PCPs (n = 155) used child psychiatry consultation. 
 63% of phase 1–5 practices (n = 37) hired integrated BHCs. 
Fidelity (extent to which the intervention was delivered in the intended way) Congruence of therapy modality with that recommended as appropriate for primary care48  
 Congruence of prescribing with clinical guidelines18,46,47  
Adoption (extent to which the intervention is used by providers) Significant increases in universal BH screening 
 Significant increases in practice-based psychotherapy 
 Significant increases in PCP BH visits 
 Significant increases in PCP psychotropic prescribing for anxiety and depression 
Penetration (level of dissemination of the intervention throughout the target setting) 93% participation to date of all eligible practices (n = 84) in BHIP 
Sustainability (maintenance of the intervention over time) Maintenance of BHIP over 7 continuous years (with external support from affiliated academic medical centers [education component] and the state [consultation component]) 
Implementation Domain36 BHIP Implementation Evidence35,49  (This Publication)
Acceptability (stakeholders’ satisfaction with the intervention) >90% of surveyed PCPs (n = 66, measured in phases 1–3) reported that BHIP participation achieved the following: 
  Increased their knowledge about symptom rating scales, guided self-management, psychotropic medications, and level-of-care decisions 
  Imparted greater confidence in their ability to manage BH problems 
  Improved the quality of their BH care 
Appropriateness (perceived fit of the intervention with the practice setting) Close match between disorders targeted for BHIP education (anxiety, depression, and ADHD) and those most commonly presenting in PCPs practices 
 After CAP consultation, disposition predominantly to PCPs for ongoing management 
Feasibility (extent to which the intervention can be successfully conducted in the practice setting) 100% of surveyed phase 1 practices (n = 12) achieved all integration readiness domains (leadership, resources, administrative mechanisms, screening, clinical management, family centeredness, care coordination, and quality improvement). 
 100% of phase 1–5 practices (n = 59) participated in BHIP program components (education, consultation, and integrated practice transformation). 
 35% of phase 1–5 PCPs (n = 125) participated in the didactic learning-community sessions. 
 44% of phase 1–5 PCPs (n = 155) used child psychiatry consultation. 
 63% of phase 1–5 practices (n = 37) hired integrated BHCs. 
Fidelity (extent to which the intervention was delivered in the intended way) Congruence of therapy modality with that recommended as appropriate for primary care48  
 Congruence of prescribing with clinical guidelines18,46,47  
Adoption (extent to which the intervention is used by providers) Significant increases in universal BH screening 
 Significant increases in practice-based psychotherapy 
 Significant increases in PCP BH visits 
 Significant increases in PCP psychotropic prescribing for anxiety and depression 
Penetration (level of dissemination of the intervention throughout the target setting) 93% participation to date of all eligible practices (n = 84) in BHIP 
Sustainability (maintenance of the intervention over time) Maintenance of BHIP over 7 continuous years (with external support from affiliated academic medical centers [education component] and the state [consultation component]) 

BHC represents a therapist.

BHC psychotherapy visits, PCP BH visits and prescribing rates, and ED BH visits were compared across preimplementation, launch, and continuation periods in the phase 1 to 5 practices (n = 50), for which claims data were available from project onset in July 2013. Screening proportions were compared across the 3 periods in practices (n = 41) that used a common EHR. Outcome measures were summarized quarterly. Preimplementation comprised treatment as usual (usual level of BH screening, PCP BH visits, and psychotropic prescribing in all practices; usual psychotherapy in one phase 1 and one phase 2 practice). Launch (12 months’ duration) comprised delivery of most clinical and operational learning sessions, facilitation of consultative relationships with BHIP and MCPAP teams, and initiation of BH integration support (including the hiring of BHCs). Continuation comprised ongoing delivery of mature program components (which continues in all practices to the present time, including an opportunity for PCPs and BHCs to attend the BHLC in a later phase if they were not able to attend initially).

The results of the phased BHIP implementation were evaluated by using a stepped-wedge analysis, with each phase treated as a cluster to isolate the effect of BHIP implementation on the outcomes of interest (BH screening, BHC psychotherapy visits, PCP BH visits and psychotropic prescribing, and ED BH visits). Odds ratios (binary outcomes) and rate ratios (rate outcomes), adjusted for secular trends, were calculated for the preimplementation period versus the continuation period for all phases combined by using PROC GLIMMIX with time and program implementation as direct effects in each model and phase as a random effect.50,51  All analyses were completed by using SAS version 9.4 (SAS Institute, Inc, Cary, NC).

By the end of the observation period, all 59 practices (100%) and 125 PCPs (35%) had participated in ≥1 session of the BHLC; 97 physicians (27%) had earned continuing medical education credits by completing attendance, quality project, and survey participation requirements. All (100%) practices participated in learning about BH integration, and 100% of BHCs used consultation from integration managers.

Forty-two practices (71%) and 155 PCPs (44%) used the BHIP and/or MCPAP consultation component. Among 563 MCPAP telephone consultations pertaining to 533 patients during the most recent 12-month period for which data were available (third quarter 2018 through second quarter 2019), leading reasons for consultation were medication question (37.8%) and in-person diagnostic (22.0%) and/or medication (21.7%) consultation request, respectively. Leading preliminary diagnoses by CAP consultants were anxiety (30.6%), depression (25.6%), and ADHD (15.9%). Leading dispositions were ongoing management by PCP (55.4%), in-person CAP consultation (31.1%), outpatient BH referral (4.8%), and emergency, inpatient, and/or partial hospitalization BH referral (1.7%).

In the preimplementation phase, only 2 practices had integrated a BHC (3%). By third quarter 2019, 37 BHCs had been integrated into the 59 practices (63%). Larger practices were more likely to integrate BHCs: 77% of practices with ≥3 PCPs vs 13% of practices with 1 to 2 PCPs (P < .001). As a result, 82.4% of all patients in the practices had access to an integrated BHC. Among 9290 unique patients with psychotherapy visits during third quarter 2018 through second quarter 2019, the mean age was 11 years (interquartile range 7–15 years); 53.4% were girls. The median number of visits per patient was 2 (interquartile range 1–5). Leading diagnoses were stress-related (32.6%), anxiety (26.9%), depression (7.1%), ADHD (6.7%), and co-occurring anxiety and depression (6.8%).

Across the combined BHIP phases, universal BH screening increased from 55.6% in the preimplementation period to 73.9% in the continuation period (Fig 3): adjusted odds ratio 1.25 (95% confidence interval [CI]: 1.21–1.29); P < .001. (Apparent declines in BH screening observed during 2017–2018 coincided with screening documentation changes associated with the transition to a new EHR across the association).

FIGURE 3

BH screening at well visits by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period. Temporary declines observed in 2017 to 2018 coincided with transition between EHRs.

FIGURE 3

BH screening at well visits by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period. Temporary declines observed in 2017 to 2018 coincided with transition between EHRs.

Close modal

Across the combined BHIP phases, psychotherapy visits increased from 15 visits per 1000 patient-years in the preimplementation period to 176 in the continuation period (Fig 4): adjusted rate ratio 6.7 (95% CI: 5.8–7.7; P < .001). PCP BH visits increased from 107 visits per 1000 patient-years preimplementation to 177 in the continuation period (Fig 5): adjusted rate ratio 1.2 (95% CI: 1.1–1.3; P < .001). Guideline-congruent ADHD prescribing did not significantly change, from 254 prescriptions per 1000 patient-years preimplementation to 362 in the continuation period (Fig 6): adjusted rate ratio 1.01 (95% CI: 0.96–1.07; P = .60). Guideline-congruent selective serotonin reuptake inhibitor (SSRI) prescribing increased from 57 prescriptions per 1000 patient-years preimplementation to 190 in the continuation period (Fig 6): adjusted rate ratio 1.3 (95% CI: 1.2–1.4; P < .001). ED BH visits did not significantly change (Fig 7): adjusted rate ratio 0.9 (95% CI: 0.8–1.1; P = .46).

FIGURE 4

Psychotherapy visits with BHCs by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

FIGURE 4

Psychotherapy visits with BHCs by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

Close modal
FIGURE 5

BH visits with pediatric PCPs by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

FIGURE 5

BH visits with pediatric PCPs by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

Close modal
FIGURE 6

Medication prescriptions for ADHD, anxiety, and depression by pediatric PCPs by program phase. ADHD medications (stimulants, atomoxetine, and α-agonists) are represented by the red lines; medications for anxiety and depression (SSRIs) are represented by the blue lines. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

FIGURE 6

Medication prescriptions for ADHD, anxiety, and depression by pediatric PCPs by program phase. ADHD medications (stimulants, atomoxetine, and α-agonists) are represented by the red lines; medications for anxiety and depression (SSRIs) are represented by the blue lines. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

Close modal
FIGURE 7

ED Visits for BH Diagnoses by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

FIGURE 7

ED Visits for BH Diagnoses by program phase. The area in white represents the preimplementation period for each phase; dark gray represents the 12-month project launch period; and light gray represents the postlaunch continuation period.

Close modal

Table 5 presents the BHIP implementation findings to date across 7 standard36  outcome domains.

This analysis demonstrates that the promising pilot findings35  from an integrated collaborative BH program implemented in a large association of community-based, independently owned pediatric primary care practices were replicated over multiple consecutive implementations. The outcome findings suggest that BHIP is successful in generating increased access to BH services (screening, psychotherapy, PCP BH visits, and psychotropic prescribing for anxiety and depression) in pediatric primary care. Moreover, the implementation findings suggest that BHIP is practical to widely disseminate in real-world pediatric primary care settings when key supports (education and consultation) are available.

Providing high-quality services for mild and/or moderate psychiatric disorders in pediatric primary care can convey substantial primary care advantages34  for children and families, including greater trust, convenience, and continuity, and lower costs and stigma.34,52  Yet in a recent national survey22  of general pediatricians, only one-half and two-thirds, respectively, reported screening for anxiety or depression in their patients, and only one-quarter reported treating these disorders. Patients who do receive treatment are reported to have markedly different care experiences because of substantial variability in PCP BH expertise.53 

In this context, the BHIP model was developed to diminish the challenges faced by PCPs when identifying, assessing, and managing mild and/or moderate presentations of common psychiatric disorders. The multicomponent BHIP model can be considered to be a hybrid of different levels of collaborative care,24  comprising full integrated collaboration24  (on-site BH team) and basic collaboration at a distance24  (education and consultation). Because educational and consultative activities generally are nonreimbursable, they risk unsustainability without external grant and/or contract funding. Conversely, for the integrated component, revenues generated by BHCs may exceed their costs, whereas the BH activities of CCs generally are included in usual practice-operating expenses. Like BHIP, other collaborative BH care models likely will need to blend multiple sources of funding to ensure sustainability over time; the ability of community pediatric practices to leverage BH resources through partnerships with academic medical centers (for education) and statewide child psychiatry access programs (for consultation) may be key in this regard. The costs of the BHIP partnerships listed in Table 2 in relation to demonstrable benefits of the program will be explored in a separate report.

In terms of access to BH services in primary care, the strongest effect derived from practices’ hiring of new BHCs, who substantially increased on-site access to psychotherapy. However, the significant increases in screening, PCP BH visits, and SSRI prescribing also are notable because a key goal of collaborative BH care is to shift appropriate clinical tasks (eg, screening, focused assessment, and prescribing of first-line medications) from specialty practitioners to PCPs54  so as to conserve the limited specialist workforce for the management of more severe and complex presentations. Significant change in ADHD prescribing was not observed; however, pediatric PCPs generally have sufficient training and comfort in the management of ADHD such that BHIP could not achieve further gains in prescribing for this disorder.22 

Some variability of both processes (integrating BHCs) and outcomes (BH service delivery) across the 5 BHIP phases was observed. Along with phase of entry, key factors potentially influencing variability in model adoption are listed in Table 1; the association of these factors with the magnitude of program effect in the 5 phases will be explored in a separate report.

In the context of an increase in ED BH visits among youth observed nationally,55  and reports of increased ED BH referrals associated with BH care in pediatric settings,56,57  it is notable that ED BH use by BHIP practices remained flat. Although this finding could be considered a positive outcome (because attenuating appropriate ED use would not be a desirable outcome of collaborative care), more detailed examination is necessary to understand the appropriate ED referral rate.

The BHIP model is congruent with key factors (population-based care [screening], measurement-based care [use of symptom rating scales], and evidence-based services [guideline-congruent psychopharmacology]) identified in a recent systematic review31  as the components most commonly associated with effective pediatric collaborative BH programs. Moreover, the cumulative findings from the BHIP to date35,49  have provided promising evidence of implementation effectiveness across multiple domains. These favorable implementation outcomes extend those identified in previous reviews of pediatric collaborative BH programs58,59  and support the practicality of widespread program dissemination, given availability of key external resources (education and consultation).

Despite its successes, BHIP continues to struggle with a number of challenges, including incomplete participation of PCPs in the educational and consultation components because of release-time constraints; incomplete integration of BHCs across the network because of practice financial and/or space constraints; high demand for on-site BH services, which can exceed BHC capacity; limitation of the direct treatment component of the model to mild and/or moderate psychiatric disorders necessitating referral of severe and/or complex cases to specialty BH services with limited availability; briefer and unscheduled visits for BH providers in primary versus specialty care, requiring adjustment of historical BHC workways; limited time for BH team meetings and other nonreimbursable activities; and limited ability to preventively address emerging BH concerns because of lack of insurance reimbursement for management of subdiagnostic presentations. These factors will need to be successfully addressed if collaborative pediatric BH care is to achieve its full potential. For example, in the BHIP, the BHLC is moving to a web-based platform to increase feasibility by enabling on-demand learning; BHC-sharing arrangements are facilitating BHC hiring by smaller pediatric practices; training in the unique aspects of primary care BH is providing support to BHCs; and formal collaborations with community-based BH entities are enhancing access to specialty BH services. Other BHIP quality improvement initiatives are continuing to seek additional solutions to implementation challenges.

The strengths of this analysis include the large study population, high program participation rates, standardized interventions and outcome measures across all study phases, a broad array of quality indicators, availability of EHR and claims data, use of a stepped-wedge design with multiple implementation replications, and lengthy follow-up. Limitations include the nonexperimental design, service-use data limited to a single (albeit largest) commercial payer source (and, as such, potential limited applicability to higher Medicaid populations), inability to disaggregate the individual effects of each BHIP component, and absence of clinical BH patient outcomes. The recent migration of all association practices to a single EHR templated to record symptom severity and functional impairment scores will enable later analysis of the clinical outcomes.

In the context of uncertainty about the benefits of pediatric collaborative care,60  the findings from the BHIP provide further support for the potential of collaborative care to have a favorable impact on access to BH services in pediatric primary care. The findings also provide support for the practicality of wide dissemination of collaborative BH care in real-world settings given key supports. If sustainably funded, widely disseminated, and locally adapted, such models can increase the competencies of PCPs to effectively manage mild to moderate BH problems in the pediatric medical home. By extending the scope of their practice to BH, PCPs can help to alleviate the substantial gap between the millions of youth needing quality BH services and those receiving them.

We acknowledge the practitioners and staff at the participating pediatric practices; BHLC faculty from Boston Children’s Hospital Departments of Psychiatry and Pediatrics; and the BHIP staff, past and present, for their contributions to BHIP.

Drs Walter and Vernacchio conceptualized and designed the study, analyzed and interpreted the data, and drafted the initial manuscript; Drs Focht, Bromberg, Young, and DeMaso and Ms Goodman and Ms Barton conceptualized and designed the study; Ms Correa analyzed and interpreted the data and drafted the initial manuscript; and all authors reviewed and revised the manuscript, approved the final manuscript as submitted, and agree to be accountable for all aspects of the work.

FUNDING: Funded by the Boston Children’s Hospital Payer Provider Quality Initiative, a collaboration between Boston Children’s Hospital, Blue Cross Blue Shield of Massachusetts, Harvard Pilgrim Health Care, Tufts Health Care, and MassHealth.

ADHD

attention-deficit/hyperactivity disorder

BH

behavioral health

BHC

behavioral health clinician

BHIP

behavioral health integration program

BHLC

behavioral health learning community

CAP

child and adolescent psychiatrist

CC

care coordinator

CI

confidence interval

ED

emergency department

EHR

electronic health record

MCPAP

Massachusetts Child Psychiatry Access Program

PCP

primary care practitioner

SSRI

selective serotonin reuptake inhibitor

1
Perou
R
,
Bitsko
RH
,
Blumberg
SJ
, et al;
Centers for Disease Control and Prevention (CDC)
.
Mental health surveillance among children–United States, 2005-2011
.
MMWR Suppl
.
2013
;
62
(
2
):
1
35
2
Whitney
DG
,
Peterson
MD
.
US national and state-level prevalence of mental health disorders and disparities of mental health care use in children
.
JAMA Pediatr
.
2019
;
173
(
4
):
389
391
3
Copeland
WE
,
Wolke
D
,
Shanahan
L
,
Costello
EJ
.
Adult functional outcomes of common childhood psychiatric problems: a prospective, longitudinal study
.
JAMA Psychiatry
.
2015
;
72
(
9
):
892
899
4
McBain
RK
,
Kofner
A
,
Stein
BD
,
Cantor
JH
,
Vogt
WB
,
Yu
H
.
Growth and distribution of child psychiatrists in the United States: 2007-2016
.
Pediatrics
.
2019
;
144
(
6
):
e20191576
5
American Academy of Child and Adolescent Psychiatry
.
Child and adolescent psychiatry workforce crisis
.
2013
.
6
American Academy of Pediatrics Committee on Psychosocial Aspects of Child and Family Health: pediatrics and the psychosocial aspects of child and family health
.
Pediatrics
.
1982
;
70
(
1
):
126
127
7
American Academy of Pediatrics Committee on Psychosocial Aspects of Child and Family Health: the pediatrician and the “new morbidity”
.
Pediatrics
.
1993
;
92
(
5
):
731
733
8
Committee on Psychosocial Aspects of Child and Family Health
.
American Academy of Pediatrics. The new morbidity revisited: a renewed commitment to the psychosocial aspects of pediatric care. Committee on Psychosocial Aspects of Child and Family Health
.
Pediatrics
.
2001
;
108
(
5
):
1227
1230
9
Committee on Psychosocial Aspects of Child and Family Health and Task Force on Mental Health
.
Policy statement--the future of pediatrics: mental health competencies for pediatric primary care
.
Pediatrics
.
2009
;
124
(
1
):
410
421
10
Foy
JM
,
Green
CM
,
Earls
MF
;
Committee on Psychosocial Aspects of Child and Family Health, Mental Health Leadership Work Group
.
Mental health competencies for pediatric practice
.
Pediatrics
.
2019
;
144
(
5
):
e20192757
11
American Academy of Child and Adolescent Psychiatry, Committee on Health Care Access and Economics; American Academy of Pediatrics, Task Force on Mental Health
.
Improving mental health services in primary care: reducing administrative and financial barriers to access and collaboration
.
Pediatrics
.
2009
;
123
(
4
):
1248
1251
12
American Academy of Child and Adolescent Psychiatry, Committee on Collaboration with Medical Professionals
.
A guide to building collaborative mental health care partnerships in pediatric primary care
.
2010
.
13
American Academy of Child and Adolescent Psychiatry, Committee on Collaboration with Medical Professionals, Committee on Healthcare Access Economics, Committee on Community-Based Systems of Care
.
Best principles for integration of child psychiatry into the pediatric health home
.
2012
.
14
Knapp
P
,
Chait
A
,
Pappadopulos
E
,
Crystal
S
,
Jensen
PS
;
T-MAY Steering Group
.
Treatment of maladaptive aggression in youth: CERT guidelines I. Engagement, assessment, and management
.
Pediatrics
.
2012
;
129
(
6
).
15
Scotto Rosato
N
,
Correll
CU
,
Pappadopulos
E
,
Chait
A
,
Crystal
S
,
Jensen
PS
;
Treatment of Maladaptive Aggressive in Youth Steering Committee
.
Treatment of maladaptive aggression in youth: CERT guidelines II. Treatments and ongoing management
.
Pediatrics
.
2012
;
129
(
6
).
16
Zuckerbrot
RA
,
Cheung
A
,
Jensen
PS
,
Stein
REK
,
Laraque
D
;
GLAD-PC Steering Group
.
Guidelines for adolescent depression in primary care (GLAD-PC): part I. Practice preparation, identification, assessment, and initial management
.
Pediatrics
.
2018
;
141
(
3
):
e20174081
17
Cheung
AH
,
Zuckerbrot
RA
,
Jensen
PS
,
Laraque
D
,
Stein
REK
;
GLAD-PC Steering Group
.
Guidelines for adolescent depression in primary care (GLAD-PC): part II. Treatment and ongoing management
.
Pediatrics
.
2018
;
141
(
3
):
e20174082
18
Wolraich
ML
,
Hagan
JF
 Jr
,
Allan
C
, et al;
Subcommittee on Children and Adolescents with Attention-Deficit/Hyperactive Disorder
.
Clinical practice guideline for the diagnosis, evaluation, and treatment of attention-deficit/hyperactivity disorder in children and adolescents
.
Pediatrics
.
2019
;
144
(
4
):
e20192528
19
American Academy of Pediatrics
.
Addressing Mental Health Concerns in Primary Care: A Clinician’s Toolkit (CD-ROM)
.
Elk Grove Village, IL
:
American Academy of Pediatrics
;
2010
20
Foy
JM
, ed.
Mental Health Care of Children and Adolescents – A Guide for Primary Care Clinicians
.
Itasca, IL
:
American Academy of Pediatrics
;
2018
21
Riddle
MA
.
Pediatric Psychopharmacology for Primary Care
.
Elk Grove Village, IL
:
American Academy of Pediatrics
;
2016
22
Stein
RE
,
Storfer-Isser
A
,
Kerker
BD
, et al
.
Beyond ADHD: how well are we doing?
Acad Pediatr
.
2016
;
16
(
2
):
115
121
23
Horwitz
SM
,
Storfer-Isser
A
,
Kerker
BD
, et al
.
Barriers to the identification and management of psychosocial problems: changes from 2004 to 2013
.
Acad Pediatr
.
2015
;
15
(
6
):
613
620
24
SAMHSA-HRSA Center for Integrated Health Solutions
.
CIHS’ standard framework for levels of integrated care
.
25
Woltmann
E
,
Grogan-Kaylor
A
,
Perron
B
,
Georges
H
,
Kilbourne
AM
,
Bauer
MS
.
Comparative effectiveness of collaborative chronic care models for mental health conditions across primary, specialty, and behavioral health care settings: systematic review and meta-analysis
.
Am J Psychiatry
.
2012
;
169
(
8
):
790
804
26
Archer
J
,
Bower
P
,
Gilbody
S
, et al
.
Collaborative care for depression and anxiety problems
.
Cochrane Database Syst Rev
.
2012
;
10
:
CD006525
27
Gillies
D
,
Buykx
P
,
Parker
AG
,
Hetrick
SE
.
Consultation liaison in primary care for people with mental disorders
.
Cochrane Database Syst Rev
.
2015
;
2015
(
9
):
CD007193
28
Reiss-Brennan
B
,
Brunisholz
KD
,
Dredge
C
, et al
.
Association of integrated team-based care with health care quality, utilization, and cost
.
JAMA
.
2016
;
316
(
8
):
826
834
29
Asarnow
JR
,
Rozenman
M
,
Wiblin
J
,
Zeltzer
L
.
Integrated medical-behavioral care compared with usual primary care for child and adolescent behavioral health: a meta-analysis
.
JAMA Pediatr
.
2015
;
169
(
10
):
929
937
30
Yu
H
,
Kolko
DJ
,
Torres
E
.
Collaborative mental health care for pediatric behavior disorders in primary care: does it reduce mental health care costs?
Fam Syst Health
.
2017
;
35
(
1
):
46
57
31
Yonek
J
,
Lee
C-M
,
Harrison
A
,
Mangurian
C
,
Tolou-Shams
M
.
Key components of effective pediatric integrated mental health care models: a systematic review
.
JAMA Pediatr
.
2020
;
174
(
5
):
487
498
32
Richardson
LP
,
McCarty
CA
,
Radovic
A
,
Suleiman
AB
.
Research in the integration of behavioral health for adolescents and young adults in primary care settings: a systematic review
.
J Adolesc Health
.
2017
;
60
(
3
):
261
269
33
Katzelnick
DJ
,
Williams
MD
.
Large-scale dissemination of collaborative care and implications for psychiatry
.
Psychiatr Serv
.
2015
;
66
(
9
):
904
906
34
Foy
JM
;
American Academy of Pediatrics Task Force on Mental Health
.
Enhancing pediatric mental health care: report from the American Academy of Pediatrics Task Force on Mental Health. Introduction
.
Pediatrics
.
2010
;
125
(
suppl 3
):
S69
S74
35
Walter
HJ
,
Vernacchio
L
,
Trudell
EK
, et al
.
Five-year outcomes of behavioral health integration in pediatric primary care
.
Pediatrics
.
2019
;
144
(
1
):
e20183243
36
Proctor
EK
,
Landsverk
J
,
Aarons
G
,
Chambers
D
,
Glisson
C
,
Mittman
B
.
Implementation research in mental health services: an emerging science with conceptual, methodological, and training challenges
.
Adm Policy Ment Health
.
2009
;
36
(
1
):
24
34
37
Massachusetts Metropolitan Area Planning Council
.
Massachusetts community types
.
38
United States Census Bureau
.
QuickFacts, Massachusetts
.
2019
.
Available at: https://www.census.gov/quickfacts/MA. Accessed January 29, 2021
39
Massachusetts Child Psychiatry Access Program
.
Available at: http://mcpap.com. Accessed January 29, 2021
40
Borowsky
IW
,
Mozayeny
S
,
Ireland
M
.
Brief psychosocial screening at health supervision and acute care visits
.
Pediatrics
.
2003
;
112
(
1 pt 1
):
129
133
41
Richardson
LP
,
McCauley
E
,
Grossman
DC
, et al
.
Evaluation of the Patient Health Questionnaire-9 item for detecting major depression among adolescents
.
Pediatrics
.
2010
;
126
(
6
):
1117
1123
42
Wolraich
ML
,
Lambert
W
,
Doffing
MA
,
Bickman
L
,
Simmons
T
,
Worley
K
.
Psychometric properties of the Vanderbilt ADHD diagnostic parent rating scale in a referred population
.
J Pediatr Psychol
.
2003
;
28
(
8
):
559
567
43
Daviss
WB
,
Birmaher
B
,
Melhem
NA
,
Axelson
DA
,
Michaels
SM
,
Brent
DA
.
Criterion validity of the Mood and Feelings Questionnaire for depressive episodes in clinic and non-clinic subjects
.
J Child Psychol Psychiatry
.
2006
;
47
(
9
):
927
934
44
Birmaher
B
,
Khetarpal
S
,
Brent
D
, et al
.
The Screen for Child Anxiety Related Emotional Disorders (SCARED): scale construction and psychometric characteristics
.
J Am Acad Child Adolesc Psychiatry
.
1997
;
36
(
4
):
545
553
45
Mossman
SA
,
Luft
MJ
,
Schroeder
HK
, et al
.
The Generalized Anxiety Disorder 7-item scale in adolescents with generalized anxiety disorder: signal detection and validation
.
Ann Clin Psychiatry
.
2017
;
29
(
4
):
227
234A
46
Walter
HJ
,
Bukstein
OG
,
Abright
AR
, et al
.
Clinical practice guideline for the assessment and treatment of children and adolescents with anxiety disorders
.
J Am Acad Child Adolesc Psychiatry
.
2020
;
59
(
10
):
1107
1124
47
Birmaher
B
,
Brent
D
,
Bernet
W
, et al;
AACAP Work Group on Quality Issues
.
Practice parameter for the assessment and treatment of children and adolescents with depressive disorders
.
J Am Acad Child Adolesc Psychiatry
.
2007
;
46
(
11
):
1503
1526
48
Schuyler
D
.
Prescribing brief psychotherapy
.
Prim Care Companion J Clin Psychiatry
.
2000
;
2
(
1
):
13
15
49
Walter
HJ
,
Kackloudis
G
,
Trudell
EK
, et al
.
Enhancing pediatricians’ behavioral health competencies through child psychiatry consultation and education
.
Clin Pediatr (Phila)
.
2018
;
57
(
8
):
958
969
50
Hussey
MA
,
Hughes
JP
.
Design and analysis of stepped wedge cluster randomized trials
.
Contemp Clin Trials
.
2007
;
28
(
2
):
182
191
51
Hughes
JP
,
Granston
TS
,
Heagerty
PJ
.
Current issues in the design and analysis of stepped wedge trials
.
Contemp Clin Trials
.
2015
;
45
(
pt A
):
55
60
52
Davis
MM
,
Gunn
R
,
Gowen
LK
,
Miller
BF
,
Green
LA
,
Cohen
DJ
.
A qualitative study of patient experiences of care in integrated behavioral health and primary care settings: more similar than different
.
Transl Behav Med
.
2018
;
8
(
5
):
649
659
53
Mayne
SL
,
Ross
ME
,
Song
L
, et al
.
Variations in mental health diagnosis and prescribing across pediatric primary care practices
.
Pediatrics
.
2016
;
137
(
5
):
e20152974
54
World Health Organization
.
Task Shifting to Tackle Health Worker Shortages
.
Geneva, Switzerland
:
World Health Organization
;
2007
55
Kalb
LG
,
Stapp
EK
,
Ballard
ED
,
Holingue
C
,
Keefer
A
,
Riley
A
.
Trends in psychiatric emergency department visits among youth and young adults in the US
.
Pediatrics
.
2019
;
143
(
4
):
e20182192
56
Hacker
KA
,
Penfold
RB
,
Arsenault
LN
,
Zhang
F
,
Soumerai
SB
,
Wissow
LS
.
Effect of pediatric behavioral health screening and colocated services on ambulatory and inpatient utilization
.
Psychiatr Serv
.
2015
;
66
(
11
):
1141
1148
57
Hacker
K
,
Penfold
R
,
Arsenault
LN
,
Zhang
F
,
Soumerai
SB
,
Wissow
LS
.
The impact of the Massachusetts behavioral health child screening policy on service utilization
.
Psychiatr Serv
.
2017
;
68
(
1
):
25
32
58
Platt
RE
,
Spencer
AE
,
Burkey
MD
, et al
.
What’s known about implementing co-located paediatric integrated care: a scoping review
.
Int Rev Psychiatry
.
2018
;
30
(
6
):
242
271
59
Spencer
AE
,
Platt
RE
,
Bettencourt
AF
, et al
.
Implementation of off-site integrated care for children: a scoping review
.
Harv Rev Psychiatry
.
2019
;
27
(
6
):
342
353
60
Wolfe
I
,
Satherley
RM
,
Scotney
E
,
Newham
J
,
Lingam
R
.
Integrated care models and child health: a meta-analysis
.
Pediatrics
.
2020
;
145
(
1
):
e201837

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: Dr DeMaso receives royalties from American Psychiatric Publishing; the other authors have indicated they have no financial relationships relevant to this article to disclose.