Competency of Future Pediatricians Caring for Children With Behavioral and Mental Health Problems

Residents and recent graduates applying for the first time for the initial certifying examination in general pediatrics from January 2018 to May 2018 were offered a B/MH-focused survey at the end of the application. Applicants were informed that participation had no bearing on their application and that results would be only shared in the aggregate. Excluded from this analysis were trainees from residency programs in Canada and trainees in combined pediatrics-psychiatry residency programs. The ABP’s Institutional Review Board of record deemed the survey exempt.

With the B/MH training questions, respondents were inquired about attitudes regarding how competent primary care and subspecialty trainees should be with B/MH concerns and about perceived competence in assessment and treatment of the B/MH issues listed below. Specifically, the survey was used to assess the extent to which respondents agreed that trainees going in to primary care and/or subspecialty care should be competent in the assessment, treatment, and comanagement or referral of patients with common B/MH problems and to assess perceptions of their program and faculty’s commitment to B/MH training (strongly agree, agree, disagree, strongly disagree). Respondents were asked to rate their competence in 7 assessment skills: eliciting parental B/MH concerns; using screening tools to identify B/MH concerns; using disorder-specific rating scales to help with diagnosis; diagnosing ADHD, anxiety, or depression by using criteria in the Diagnostic and Statistical Manual of Mental Health Disorders, Fifth Edition¹² ; and assessing suicidality. Respondents were also asked to rate their competence in 9 treatment skills: using evidence-based communication and engagement strategies; behavioral management counseling; nonmedication strategies for ADHD, depression, or anxiety; dosing ADHD or depression medications; titrating medications; safety counseling; and comanagement with mental health specialists. Competence was measured on a 5-point scale (poor [1], fair [2], good [3], very good [4], and excellent [5]), consistent with previous studies.⁸  The survey development was informed by previous studies.^8,10 

Individual characteristics examined included age, sex, race and/or ethnicity, years since training, planned future clinical role (general pediatrics, subspecialist, hospitalist, or other/unsure), and medical school location (American versus international medical school). Program characteristics were also available from the ABP’s resident tracking database and included the size of the program (small: 0–30 trainees; medium: 31–60 trainees; large: >60 trainees), program type (categorical or combined programs such as medicine-pediatrics), and US Census Bureau regions.

The primary outcome was perceived competence in assessment and treatment of common B/MH conditions. For this analysis, we created binary variables, with high self-reported competence defined as a score of ≥4 (very good and excellent) compared with lower self-reported competence (poor, fair, and good). We chose this method of dichotomization for consistency with previous studies, allowing us to assess whether competence ratings were improved.⁸  We calculated B/MH assessment and treatment composite scores for each respondent with complete data by calculating the mean score for the 7 B/MH assessment submeasures and the 9 treatment submeasures, respectively.

To identify individual and program-level factors associated with high self-reported competence, we calculated differences in baseline characteristics between respondents with high competence in B/MH assessment and treatment and those reporting less competence using χ² tests and t tests, as appropriate. We then examined variation in B/MH assessment and treatment competence across residency programs by calculating the percentage of respondents at each residency program who had composite assessment and treatment scores ≥4. This resulted in program-level assessment and treatment measures on a 0 to 100 scale. To provide stable estimates, we limited this analysis to residency programs with at least 5 respondents. We examined ranges and coefficients of variation (CVs) ([SD/mean] × 100) in small, medium, and large programs to identify the degree of variation (small: CV < 10; moderate: CV 10–<20; high: CV ≥ 20).¹³  We also examined differences in these measures across and within small, medium, and large residency programs using analysis of variance.

To identify factors associated with these program-level assessment and treatment measures, we first developed unadjusted linear regression with residency program as the unit of analysis. We then developed multivariable models, adjusting for program size, geographic region, and program type (categorical pediatrics or medicine-pediatrics) as well as program-level mean values of individual factors (age, sex, race and/or ethnicity, and medical school location). As a sensitivity analysis, we repeated these regression analyses using a more liberal definition of perceived competence, calculating measures on the basis of the percentage of respondents at each residency program who had composite assessment and treatment scores ≥3. All analyses were conducted by using Stata 15 (Stata Corp, College Station, TX),¹⁴  by using 2-sided tests, and by defining P < .05 as statistically significant.

A total of 2086 individuals participated in this survey (response rate: 62.3%). There were no significant differences between respondents and nonrespondents with respect to age, sex, medical school location, training program size, and geographic region (Supplemental Table 3).

Almost all respondents reported that they believed that all residents entering primary care as well as subspecialties should be competent in the assessment, treatment, referral, and comanagement of common childhood B/MH concerns. Specifically, there was high agreement (ie, ≥98.0% agreeing or strongly agreeing with each skill) for residents entering primary care. The majority also agreed that residents going in to pediatric subspecialty fellowships required competence in assessment (96.7%) and referral or comanagement (95.8%) of B/MH conditions, but less (86.3%) agreed they required competence in the treatment of B/MH conditions. More than 90% (n = 1536; 92.6%) agreed or strongly agreed that their training program was committed to ensuring that graduating residents can address B/MH problems, whereas 92.6% (n = 1537) agreed or strongly agreed that faculty and staff promoted the importance of B/MH by showing their concern and incorporating conversations about B/MH into each interaction.

A smaller proportion of respondents self-reported very good or excellent (high) competence with their own B/MH assessment and treatment skills. The distribution of perceived competence across the B/MH assessment and treatment submeasures is shown in Fig 1. Within the assessment domain, eliciting parent and patient concerns about B/MH problems was the most highly rated competency, with 65.6% of respondents (n = 1196) reporting high competence. However, less than half of respondents reported high competence in diagnosing anxiety (43.4%, n = 773). In the treatment domain, the relative distribution of responses was shifted left, with a low of 26.6% (n = 474) of respondents reporting high competence in dosing medications to treat anxiety or depression and a high of 51.7% (n = 943) reporting this degree of competence in using evidence-based tools, such as motivational interviewing, to encourage engagement in treatment.

Overall, one-third (32.8%; n = 595) of respondents had mean B/MH assessment composite scores ≥4. The characteristics of these respondents, relative to those with mean B/MH composite scores <4, are shown in Table 1. Respondents who reported competence with B/MH assessment were older, more frequently men, and graduates of international medical schools. A smaller proportion (18.9%; n = 337) of respondents had mean B/MH treatment composite scores of ≥4. As with assessment competence, respondents reporting treatment competence were older, more frequently men, and graduates of international medical schools; in addition, treatment competence was reported more frequently by respondents who anticipated subspecialist careers. With respect to program-level characteristics, both assessment and treatment competence was reported significantly more often by graduates of smaller residency programs, graduates of programs in the Northeast, and graduates from combined residency programs (eg, medicine-pediatrics).

Of the 271 programs included in our analysis, 158 programs had at least 5 respondents to the survey and were included in our program-level analysis, leaving 153 categorical programs and 5 medicine-pediatrics programs (representing 1467 respondents). Figure 2 reveals the variation in assessment and treatment competence, as well as program-level mean scores, across the 33 small residency programs, 76 medium programs, and 49 large programs. For both of these outcomes, mean program-level scores were highest for small programs and lowest for large programs. Specifically, the program-level assessment mean scores were 43.4% for small programs, 33.6% for medium programs, and 28.7% for large programs; treatment mean scores were 28.7% for small programs, 19.6% for medium, programs and 13.0% for large programs (P < .01 for the difference in means for both measures). However, within each program-size group, we observed substantial variation, with a range from 0% of residents in the program reporting competence to a high of 100% in small and medium programs and a high of 80% in large programs for B/MH assessment skills. For treatment skills, the ranges were similar: 0% to 100% for small programs, 0% to 80% for medium programs, and 0% to 43% for large programs. CVs for assessment skills were 52.5, 59.7, and 46.9 across small, medium, and large residency programs, respectively, indicating high variation across all program-size strata.¹³  CVs for treatment skills were similarly high, with values of 77.5, 78.1, and 77.4 across small, medium, and large residency programs, respectively.

Regression results revealing pairwise comparisons are presented in Table 2. In unadjusted analyses, both medium and large residency programs had significantly lower assessment and treatment program scores than small programs. When adjusting for other program and individual characteristics, these differences remained for B/MH assessment. Specifically, when adjusting for other covariates, medium-sized programs had, on average, program-level assessment scores 8.9 points lower than those of small programs (−8.9; 95% confidence interval [CI] −17.1 to −0.8), whereas large programs had program-level assessment scores 10.3 points lower than those of small programs (−10.3; 95% CI −19.5 to −1.0). For B/MH treatment, there was no significant difference in adjusted scores between small- and medium-sized programs; however, the significant difference between small and large residency programs remained. As shown in Table 2, large programs had mean program-level assessment scores 8.1 points lower than those of small programs (−8.1; 95% CI −15.3 to −0.9). With the exception of lower treatment composite scores in the West relative to the Midwest, we did not identify any other program factors associated with program scores in our multivariable models.

In our sensitivity analysis, when using composite assessment and treatment scores ≥3, significant differences in program-level treatment measure scores remained. When adjusting for other individual- and program-level characteristics, medium-sized residency programs had mean program-level treatment scores 8.6 points lower than those of small programs (−8.6; 95% CI −16.9 to 0.3), whereas large program had mean program-level treatment scores 17.9 points lower than those of small programs (−17.9; 95% CI −27.3 to −8.4). In our sensitivity analysis of program-level assessment measures, differences according to residency program characteristics were not statistically significant (Supplemental Table 4).

In this large national survey, respondents strongly endorsed pediatric trainee competence in B/MH management for residents entering primary care as well as subspecialty careers. Despite this, a small fraction of respondents reported high levels of competence in B/MH assessment, and an even smaller fraction perceived themselves to be competent in treatment. Furthermore, we observed substantial variation in trainee competence across residency training programs. Differences between assessment and treatment remained on sensitivity analyses. These findings have important implications for the ongoing development of pediatric B/MH training initiatives.

Respondent beliefs about which trainees should be competent in addressing mental health problems are encouraging. Of respondents, 86% agreed or strongly agreed that those going into subspecialty care should be competent in the treatment of common childhood B/MH concerns. In a previous survey of PDs, only half agreed or strongly agreed that their graduating residents pursuing subspecialty careers should be competent in B/MH treatment. These data further illustrate the importance of educating all residents and fellows about B/MH.¹⁵ 

Although a relatively small proportion of respondents reported competence in B/MH assessment and treatment, these numbers have improved over time from previous studies.⁸  For instance, 52% of our sample reported very good or excellent competence in diagnosing depression, which is higher than the 24% reported in the previously mentioned national survey of graduating residents conducted in 2007. Similarly, only 17% of respondents previously reported high competence in diagnosing anxiety, as opposed to 44% reporting high competence in this sample.⁸  Perceived competence in providing behavioral counseling also increased from 16% to 40%. Almost 60% of respondents reported high competence in assessing suicidality, and half reported high competence in safety counseling. However, because suicide is the second leading cause of death for children starting at age 10, ongoing efforts to educate pediatric residents in this domain are necessary to make an impact on this common and potentially life-threatening condition.¹⁶ 

National initiatives, such as publication guidelines for depression^17,18  (2007) and maladaptive aggression^19,20  (2012), in primary care may explain why perceived competence has improved over time. ADHD guidelines have existed since the year 2000, and ADHD-related practices and skills have consistently been rated higher than other B/MH conditions in surveys of residents and practicing pediatricians.^8,10,21  That said, less than one-third of respondents reported high levels of competence for B/MH assessment, and less than one-fifth reported high levels of competence for B/MH treatment. Recognizing that >7.7 million children have B/MH conditions, these numbers are disconcertingly low.²  Rather than relying on passive diffusion of guidelines, concerted efforts across the pediatric community to integrate guideline recommendations for B/MH care in pediatric training and practice may further improve B/MH competence.

Respondents’ beliefs about their competence in B/MH assessment as compared with B/MH treatment are noteworthy. Whereas more than half reported having high competence in diagnosing depression, only 1 in 4 reported competence in dosing medications to treat depression. Given the shortage of child and adolescent psychiatrists in the country, pediatricians need to be competent in prescribing evidence-based psychopharmacologic medications. Resources such as the following can be used to guide curriculum development for common evidence-based B/MH treatments: the newly updated AAP policy statement,²²  published guidelines for the treatment of adolescent depression in primary care,¹⁹  and sample motivational interviewing videos on the AAP Web site.^18,23 

We observed substantial differences in mean program-level B/MH assessment and treatment scores across small, medium, and large residency programs, with smaller programs consistently having higher mean scores. These differences add to previous literature about how training experiences differ by program size. In previous research, PDs from smaller programs were more likely to implement learning activities around B/MH.¹⁰  Also, residents who trained at smaller programs more frequently go into primary care and have reported feeling more prepared to do so than those trained in larger programs.²⁴  However, the variation in program-level assessment and treatment competence measures within each size strata is notable. Although, overall, smaller programs had higher composite assessment scores, the range of scores was wide, and CVs were high in all 3 program-size strata. These results reveal opportunities for improvement across residency programs of all sizes; the development of learning collaboratives grouped by residency program size may enable high-performing programs to share best practices with lower-performing programs with similar resources and interests in primary versus subspecialty care.

Limitations of this study include our evaluation of perceived competence, not measured competence, as our outcome variable.²⁵  However, directly observed competence would require time, money, and validated assessment tools that currently do not exist.²⁶  A study of this magnitude with a similar sample size is not feasible. We dichotomized our 5-point Likert scale to exclude the midpoint, “good,” from our definition of high perceived competence. This approach is consistent with a previous study and allowed us to compare rates of competence over time.⁸  Horwitz et al⁸  similarly justified dichotomizing competence in this manner to account for a social response bias because previous research also suggests that practitioners are inclined to over-report their performance.²⁷  Additionally, the Dunning-Kruger effect is a cognitive bias in which individuals are unaware of their lack of competence and will rate their abilities higher than they are.²⁸  Correspondingly, many trainees may be unaware of all of the knowledge and skills needed to appropriately address pediatric BH/MH problems and therefore are likely to overrate their competence.

Because this study was focused on B/MH skills, we do not know how respondents would rate their competence in other pediatric skills. However, previous studies have revealed that a majority of pediatric residents report high perceived competence in dietary assessment and counseling around dietary changes for obesity and that more than two-thirds report very good to excellent perceived preparedness for medical home activiites.^29,30  Furthermore, 62% of pediatricians have reported that they would have benefitted from additional B/MH training compared with less than one-quarter reporting this need for other subspecialty conditions.³¹  Finally, we cannot assume causation for any of our observed associations given our cross-sectional design. These limitations should be interpreted in the context of our study’s strengths, including a large sample of trainees representing all regions of the United States and all residency program sizes.

This large national study of future pediatricians reveals ongoing training needs to improve trainees’ perceived competence in the assessment and treatment of B/MH problems. It is promising that rates of perceived competence have increased over time, suggesting that national or institutional interventions may be having positive impacts. However, the large variation across small, medium, and large programs in trainee B/MH competence also suggests that national standards are needed for all programs to consistently provide effective B/MH training. Interventions may need to be tailored to program size and be designed to reach trainees regardless of future career aspirations.

AAP

American Academy of Pediatrics

ABP

American Board of Pediatrics

ADHD

attention-deficit/hyperactivity disorder

B/MH

behavioral and mental health

CI

confidence interval

CV

coefficient of variation

PD

program director