OBJECTIVES:

Multiple early childhood screenings are recommended, but gaps persist in implementation. Our aim for this project was to improve screening, discussion, referral, and follow-up of development, autism spectrum disorder (ASD), maternal depression, and social determinants of health (SDoH) to 90% by July 2018.

METHODS:

This 1-year national quality improvement collaborative involved 19 pediatric primary care practices. Supported by virtual and in-person learning opportunities, practice teams implemented changes to early childhood screening. Monthly chart reviews were used to assess screening, discussion, referral, and follow-up for development, ASD, maternal depression, and SDoH. Parent surveys were used to assess parent-reported screening and referral and/or resource provision. Practice self-ratings and team surveys were used to assess practice-level changes.

RESULTS:

Participating practices included independent, academic, hospital-affiliated, and multispecialty group practices and community health centers in 12 states. The collaborative met development and ASD screening goals of >90%. Largest increases in screening occurred for maternal depression (27% to 87%; +222%; P < .001) and SDoH (26% to 76%; +231%; P < .001). Statistically significant increases in discussion of results occurred for all screening areas. For referral, significant increases were seen for development (53% to 86%; P < .001) and maternal depression (23% to 100%; P = .008). Parents also reported increased screening and referral and/or resource provision. Practice-level changes included improved systems to support screening.

CONCLUSIONS:

Practices successfully implemented multiple screenings and demonstrated improvement in subsequent discussion, referral, and follow-up steps. Continued advocacy for adequate resources to support referral and follow-up is needed to translate increased screening into improved health outcomes.

Developmental delay,1  maternal depression,2,3  and adverse social determinants of health (SDoH)46  are detrimental to child well-being. Screening for development,1  autism spectrum disorder (ASD)7  and maternal depression3  at well-child visits (WCVs) has been recommended by the American Academy of Pediatrics (AAP) and incorporated into the Bright Futures Recommendations for Preventive Pediatric Health Care (Periodicity Schedule).8  Recommendations also support regular screening for social-emotional problems9  and SDoH10  and emphasize the role of pediatric practices in early identification, referral, and follow-up.

Practices face challenges in implementing recommended screenings simultaneously. Developmental and ASD screening have improved but remain suboptimal1113 ; maternal depression and SDoH screening have been less frequently implemented.14  Well-described gaps persist in referral and follow-up of children with identified problems.15 

Quality improvement (QI) collaboratives have reduced implementation gaps in many areas, including preventive services16,17  and ASD screening.13  We designed a national QI collaborative to support practices in implementing screening recommendations for development, ASD, maternal depression, and SDoH. Aims were to improve screening processes, including screening, discussion, referral, and follow-up. Practices were encouraged to make screening processes family centered.

Practices were invited to apply to this 1-year QI collaborative through AAP chapters and e-mail distribution lists. Forty-two practices applied; 20 were selected by using standardized scoring that included the following criteria: minimum of 10 infant WCVs per month, willingness to form a QI team, organizational and/or leadership support, and experience with at least 1 target screening area. Practices were selected to ensure diversity in type, size, location, and populations served. One practice withdrew before data collection because of competing priorities, resulting in 19 practices from 12 states.

Each practice identified a QI team consisting of a physician leader, 2 staff members, and a parent partner. Practices received stipends to cover project-related expenses; parent partners received stipends to cover time on the project. Maintenance of Certification and Continuing Medical Education credits were available to all physicians who completed project requirements. Practice characteristics are shown in Table 1.

TABLE 1

Baseline Characteristics of Participating Practices (N = 19) and Patients Served

CharacteristicResult
Practice type, n (%)  
 Independent 8 (42) 
 Academic medical center 5 (26) 
 Hospital-based practice 3 (16) 
 Federally qualified health center 2 (11) 
 Multispecialty group practice 1 (5) 
Practice location, n (%)  
 Suburban 8 (42) 
 Urban 8 (42) 
 Rural 3 (16) 
Practice size, n (%)a  
 Small, 1–10 providers 7 (37) 
 Medium, 11–20 providers 8 (42) 
 Large, >20 providers 4 (21) 
No. participating office locations per practice, n (%)b  
 1 13 (68) 
 2 4 (21) 
 3 2 (11) 
Parent involvement in practice, n (%)  
 Advisory group 3 (16) 
 Parent partner 1 (5) 
 No formal role 15 (79) 
Practice with previous QI experience, n (%) 14 (74) 
Insurance coverage of patients served, mean (range), %  
 Medicaid 62 (25–94) 
 Private insurance 33 (0–66) 
 Uninsured 4 (0–12) 
Race and ethnicity of patients served, mean (range), %  
 White, non-Hispanic 41 (2–93) 
 Black, non-Hispanic 22 (1–85) 
 Hispanic 25 (2–83) 
 Asian American 6 (0–50) 
 NHOPI or AI/AN 2 (0–15) 
 Other 7 (0–25) 
Primary language other than English, mean (range), % 29 (1–90) 
CharacteristicResult
Practice type, n (%)  
 Independent 8 (42) 
 Academic medical center 5 (26) 
 Hospital-based practice 3 (16) 
 Federally qualified health center 2 (11) 
 Multispecialty group practice 1 (5) 
Practice location, n (%)  
 Suburban 8 (42) 
 Urban 8 (42) 
 Rural 3 (16) 
Practice size, n (%)a  
 Small, 1–10 providers 7 (37) 
 Medium, 11–20 providers 8 (42) 
 Large, >20 providers 4 (21) 
No. participating office locations per practice, n (%)b  
 1 13 (68) 
 2 4 (21) 
 3 2 (11) 
Parent involvement in practice, n (%)  
 Advisory group 3 (16) 
 Parent partner 1 (5) 
 No formal role 15 (79) 
Practice with previous QI experience, n (%) 14 (74) 
Insurance coverage of patients served, mean (range), %  
 Medicaid 62 (25–94) 
 Private insurance 33 (0–66) 
 Uninsured 4 (0–12) 
Race and ethnicity of patients served, mean (range), %  
 White, non-Hispanic 41 (2–93) 
 Black, non-Hispanic 22 (1–85) 
 Hispanic 25 (2–83) 
 Asian American 6 (0–50) 
 NHOPI or AI/AN 2 (0–15) 
 Other 7 (0–25) 
Primary language other than English, mean (range), % 29 (1–90) 

AI/AN, American Indian or Alaskan native; NHOPI, native Hawaiian or other Pacific Islander.

a

Includes total full-time equivalent physicians, physician assistants, and nurse practitioners.

b

Defined as actively testing and implementing all project areas and submitting monthly data.

Practices were given AAP recommendations for screening for development (9, 18, and 30 months), ASD (18 and 24 months), maternal depression (1, 2, 4, and 6 months), and SDoH (regularly at WCVs). Practices were encouraged to select validated tools from a list18 ; screening tools selected are shown in Supplemental Table 6. Practices tested processes for screening, discussion, referral, and follow-up using paper forms or electronic tablets before or during visits. Screening was encouraged in the patient’s preferred language by using translated written tools and/or interpreters; for families with limited literacy, having staff read questions was encouraged. Social-emotional screening was optional at project initiation, and practices were recommended to add this gradually per current guidance.9  A project advisory committee (PAC) comprising expert faculty in general and developmental-behavioral pediatrics oversaw planning and implementation. The PAC created a key driver diagram (Supplemental Fig 3) that served as the framework for change.

A modified Institute for Healthcare Improvement Breakthrough Series learning collaborative model was used. Learning collaboratives can facilitate implementation of evidence-based practices, have demonstrated effectiveness in improving screening processes,19  and can accelerate health care improvement.20  Participants learn the Model for Improvement, which consists of (1) aims, (2) changes, and (3) measures.21  Plan-do-study-act (PDSA) cycles are used to conduct small tests of change and accelerate learning.

The Improving Screening, Connections with Families, and Referral Networks (I-SCRN) collaborative (July 2017 to July 2018) included 2 full-day in-person learning sessions separated by a 9-month action period consisting of virtual learning, monthly data collection and feedback, and coaching. The first learning session introduced the evidence for screening and featured sessions on implementing screening and discussing results. Before the learning session, QI teams were instructed to observe existing screening processes and use a structured form to document steps. During the learning session, practices created process flow maps and identified opportunities for PDSA cycles.

Didactic and interactive learning continued during the action period. Webinars were focused on referral and follow-up of positive screening results, which practices identified as areas of need. Several webinars were focused on SDoH at participants’ request. Practices could access resources through the AAP’s Quality Improvement Data Aggregator Web site, which included links to screening tools and parent education materials.

During the second learning session, practices that achieved breakthrough improvement assisted in teaching. This session included a parent partner track and topics prioritized by practices, including SDoH, motivational interviewing, and electronic health record (EHR) documentation.

Five practices were selected for a 1-day technical assistance (TA) visit by using standardized scoring that included interest in and clarity of objectives for TA visits. These TA visits included facilitation of PDSA cycles and a virtual session facilitated by a PAC faculty member to engage the whole practice in improvement.

Chart Reviews

Practices were given a structured tool and instructions for reviewing a systematic monthly random sample of 10 charts for each age (9-, 18-, and 30-month WCVs). These ages were chosen to minimize burden, while ensuring that age-specific screenings were measured. Chart reviews were used to assess whether screening, discussion, referral, and follow-up were documented (yes or no) (Supplemental Table 7). Development and SDoH screenings were assessed at the 9-, 18-, and 30-month WCV and were summed to create single monthly measures. Maternal depression measures were used to assess whether screening occurred at least once before the 9-month WCV. ASD measures summed whether screening occurred at 18 and 24 months (by reviewing the previous 24-month WCV during the 30-month chart review). Measures for discussion, referral, and follow-up were collected using the same method. For SDoH, follow-up of referrals was not assessed to minimize burden, acknowledging that closed-loop referral systems for SDoH were uncommon. Practices used the AAP Quality Improvement Data Aggregator data collection system to enter chart reviews.

Parent Surveys

Practices collected baseline and final anonymous surveys from a randomly selected sample (n = 30) of parents or caregivers after WCVs. To minimize burden, surveys were not administered to the same sample used for chart reviews. A postvisit paper survey was designed with input from the PAC and included whether the parent or caregiver had completed a form (yes or no) at that visit about (1) the child’s development, communication, or behavior; (2) family needs or other issues affecting child’s health; and (3) sadness or depression in the mother (for WCVs before 9 months). Parents or caregivers reported whether screening results were discussed (not at all, somewhat, mostly, or completely). Additional items were used to assess whether resources and/or referrals were provided and helpful (not offered any, not at all helpful, a little helpful, somewhat helpful, or very helpful) and whether the visit helped the parent better understand child development (not at all, a little, somewhat, or a great deal). The overall parent satisfaction rating served as a balancing measure (1 = worst; 5 = best care possible). Surveys were available in English and Spanish; a Vietnamese translation was developed by a participating practice. Surveys were collected by practices and entered into an Excel calculator to generate aggregate monthly measures.

Monthly Reports and Self-assessment Ratings

Practices completed monthly reports online (SurveyMonkey Inc, San Mateo, CA). Teams were encouraged to complete these reports together and describe improvements, PDSA cycles, barriers, and TA needs. Coaches used reports to guide virtual learning content and provide individual feedback to practices. Practice teams also completed monthly self-ratings of implementation for screening, discussion, referral, and follow-up in each screening area (1 = not yet started, 2 = beginning implementation, 3 = partially implemented, and 4 = fully implemented).

Practice Team Surveys

Teams completed surveys assessing office systems to identify patients, document and track follow-up, and code and bill for screening (yes or no). Items were used to assess the extent of systems for referral and/or resource connections, including maintaining resource and/or referral lists, connecting with referral partners, using a standardized referral form, matching child and/or family needs to resources, facilitating referrals through “warm handoffs” (handoffs in the presence of the family), and following-up on referrals (not at all, to some extent, to a moderate extent, or to a large extent). The survey was used to assess use of staff training on screening and parent education materials (yes or no). Practices indicated barriers to screening, including lack of time, reimbursement, staff, appropriate screening tools, referrals and/or resources, and leadership support; EHR issues; and caregiver resistance (yes or no). Responses were dichotomized as follows: mostly and completely versus not at all and somewhat and a great deal versus not at all, a little, and somewhat.

Practices were encouraged to annotate and interpret run charts using run chart rules.22  Data for all practices combined were displayed on run charts and used by project staff to examine effects of collaborative interventions. Baseline and final chart review measures for screening and discussion were compared by using the first and last month. Because of the small numbers of positive screening results in monthly chart reviews, data were combined for the first 3 and last 3 months to calculate referral and follow-up measures. Pairwise t tests were used to compare differences in mean measures. Four practices did not complete all 13 months of data collection. To examine the effect of incomplete data, a sensitivity analysis was conducted by using the last observation carried forward method to impute missing data.23 

For parent or caregiver surveys, mean baseline and final percentages were compared by using paired t tests. Baseline and final practice self-ratings from monthly reports were compared by using matched-pairs Wilcoxon rank tests. Practice survey baseline and final responses were compared by using McNemar tests. Analyses were conducted in Excel and Stata version 15 (Stata Corp, College Station, TX). This project was reviewed by the AAP Institutional Review Board and was determined to be exempt.

Screening and discussion measures are displayed on annotated run charts (Figs 1 and 2). All 19 practices completed chart reviews (N = 756) during month 1. Fifteen practices completed 13 months of chart reviews; this group reached the screening and discussion goals of 90% for development, ASD, and maternal depression (Figs 1 and 2). Shifts indicate nonrandom change, are identified by 6 consecutive points above the median, and occurred for all screening and discussion measures except discussion of social-emotional screening.

FIGURE 1

Percentage of children receiving early childhood screenings. Practices reported the percentage of children receiving early childhood screenings in 13 monthly cycles. All 19 practices completed cycles 1 to 6, 18 completed cycles 7 to 11, 17 completed cycle 12, and 15 completed cycle 13. Learning sessions 1 and 2 occurred at months 3.5 and 10.5, respectively. Practices were asked to review 10 charts per month for each age group: 9-, 18-, and 30-month WCVs. The average number of chart reviews per month for each screening area was as follows: development, n = 716; autism, n = 463; maternal depression, n = 263, social determinants, n = 713; and social emotional, n = 229. Shifts indicate nonrandom change, are identified by 6 consecutive points above the median, and are present for all metrics displayed.

FIGURE 1

Percentage of children receiving early childhood screenings. Practices reported the percentage of children receiving early childhood screenings in 13 monthly cycles. All 19 practices completed cycles 1 to 6, 18 completed cycles 7 to 11, 17 completed cycle 12, and 15 completed cycle 13. Learning sessions 1 and 2 occurred at months 3.5 and 10.5, respectively. Practices were asked to review 10 charts per month for each age group: 9-, 18-, and 30-month WCVs. The average number of chart reviews per month for each screening area was as follows: development, n = 716; autism, n = 463; maternal depression, n = 263, social determinants, n = 713; and social emotional, n = 229. Shifts indicate nonrandom change, are identified by 6 consecutive points above the median, and are present for all metrics displayed.

Close modal
FIGURE 2

Percentage of early childhood screenings discussed with parents of caregivers. Practices reported the percentage of early childhood screenings discussed with parents or caregivers. All 19 practices completed cycles 1 to 6, 18 completed cycles 7 to 11, 17 completed cycle 12, and 15 completed cycle 13. Learning sessions 1 and 2 occurred at months 3.5 and 10.5, respectively. Practices were asked to review 10 charts per month for each age group: 9-, 18- and 30-month WCVs. The denominator for each measure is the number of children screened in each area. Average monthly denominators were as follows: development, n = 564; autism, n = 379; maternal depression, n = 159; social determinants, n = 257; social emotional, n = 148. Shifts indicate nonrandom change, are identified by 6 consecutive points above the median, and are present for all metrics displayed, except discussion of social-emotional screening.

FIGURE 2

Percentage of early childhood screenings discussed with parents of caregivers. Practices reported the percentage of early childhood screenings discussed with parents or caregivers. All 19 practices completed cycles 1 to 6, 18 completed cycles 7 to 11, 17 completed cycle 12, and 15 completed cycle 13. Learning sessions 1 and 2 occurred at months 3.5 and 10.5, respectively. Practices were asked to review 10 charts per month for each age group: 9-, 18- and 30-month WCVs. The denominator for each measure is the number of children screened in each area. Average monthly denominators were as follows: development, n = 564; autism, n = 379; maternal depression, n = 159; social determinants, n = 257; social emotional, n = 148. Shifts indicate nonrandom change, are identified by 6 consecutive points above the median, and are present for all metrics displayed, except discussion of social-emotional screening.

Close modal

We compared baseline and final percentages for screening and discussion for practices with complete data (n = 15) (Table 2). Screening increased significantly for development (from 60% to 93%; P = .009), ASD (from 74% to 95%; P = .005), maternal depression (from 27% to 87%; P < .001), and SDoH (from 26% to 76%; P < .001). Discussion increased for development (from 63% to 97%; P = .002), ASD (from 51% to 93%; P < .001), maternal depression (from 46% to 90%; P = .04), and SDoH (from 19% to 73%; P = .001). Referrals increased for development (from 53% to 86%; P < .001) and maternal depression (from 23% to 100%; P = .008). Follow-up of referrals increased for development (from 38% to 62%; P = .017). A sensitivity analysis was conducted by using the last available data for all 19 practices and yielded similar results.

TABLE 2

Mean Results of Baseline and Final Monthly Chart Reviews

DomainMeasureBaseline, %Final, %Baseline DenominatorFinal DenominatorPa
Development Screeningb 60 93 636 629 .009 
 Discussion of screening resultsb 63 97 400 561 .002 
 Referral for positive screen resultc 53 86 159 180 <.001 
 Follow-up on referral statusc 38 62 103 149 .017 
ASD Screeningb 74 95 408 431 .005 
 Discussion of screening resultsb 51 93 317 410 <.001 
 Referral for positive screen resultc 61 73 58 41 .34 
 Follow-up on referral statusc 37 10 25 29 .10 
Maternal depression Screeningb 27 87 258 209 <.001 
 Discussion of screening resultsb 46 90 103 187 .04 
 Referral for positive screen resultc 23 100 17 18 .008 
 Follow-up on referral statusc 65 67 18 .95 
SDoH Screeningb 26 76 636 629 <.001 
 Documentation of strengths and/or protective factorsb 22 35 636 629 .29 
 Discussion of screening resultsb 19 73 151 457 .001 
 Referral and/or linkage for positive resultsc 43 68 32 73 .08 
Social-emotional developmentd Screeningb 50 83 123 339 .10 
 Discussion of screening resultsb 63 92 77 255 .19 
 Referral for positive screen resultc 60 35 36 .39 
 Follow-up on referral statusc 75 20 19 .07 
DomainMeasureBaseline, %Final, %Baseline DenominatorFinal DenominatorPa
Development Screeningb 60 93 636 629 .009 
 Discussion of screening resultsb 63 97 400 561 .002 
 Referral for positive screen resultc 53 86 159 180 <.001 
 Follow-up on referral statusc 38 62 103 149 .017 
ASD Screeningb 74 95 408 431 .005 
 Discussion of screening resultsb 51 93 317 410 <.001 
 Referral for positive screen resultc 61 73 58 41 .34 
 Follow-up on referral statusc 37 10 25 29 .10 
Maternal depression Screeningb 27 87 258 209 <.001 
 Discussion of screening resultsb 46 90 103 187 .04 
 Referral for positive screen resultc 23 100 17 18 .008 
 Follow-up on referral statusc 65 67 18 .95 
SDoH Screeningb 26 76 636 629 <.001 
 Documentation of strengths and/or protective factorsb 22 35 636 629 .29 
 Discussion of screening resultsb 19 73 151 457 .001 
 Referral and/or linkage for positive resultsc 43 68 32 73 .08 
Social-emotional developmentd Screeningb 50 83 123 339 .10 
 Discussion of screening resultsb 63 92 77 255 .19 
 Referral for positive screen resultc 60 35 36 .39 
 Follow-up on referral statusc 75 20 19 .07 

Fifteen practices contributed complete data for development, ASD, maternal depression, and SDoH; 12 practices contributed data for social-emotional development because this was an optional measure at practice initiation.

a

P values displayed are for 2-sided paired t tests comparing baseline and follow-up mean percentages.

b

Screening and discussion measures display the first and last cycle (month) for 15 practices that submitted all 13 cycles.

c

Referral and follow-up measures apply only to those children who had positive screening results. Because the number of positive screening results was low in each monthly sample, referral and follow-up measures display the sum of the first 3 months (baseline) and the sum of the last 3 months (follow-up) because of small numbers for these measures within individual months.

d

Analysis based on 6 practices that indicated they conducted screening on this domain.

Parent or caregiver report of maternal depression screening increased (from 47% to 57%; P = .005), as did completing a family needs form (from 36% to 52%; P = .01; Table 3). More parents or caregivers reported that providers discussed screening results with them (from 80% to 83%), but the change was not significant (P = .11). Parent or caregiver report of referrals and/or resources for development, maternal depression, or family needs also increased (from 79% to 85%; P = .09) but did not reach statistical significance. On overall rating of care, the percentage of parents rating care as above average or best did not change; reasons for care rating were not elicited (92%).

TABLE 3

Baseline and Final Parent or Caregiver Survey Results

Parent-Reported MeasureBaseline Mean, %Final Mean, %Pa
Completed form about child’s development, communication, and/or behavior 72 78 .18 
Completed form about sadness or depression in the child’s mother (<12 mo only) 47 57 .005 
Completed form about family’s needs or other issues affecting child’s health 36 52 .01 
Provider mostly or completely talked about the results of the form (only those who completed a form)b 80 83 .11 
Provider offered a referral or resources to support child’s development, help the child’s mother with sadness or depression, or help family with other needs 79 85 .09 
Care received was above average or bestc 92 92 .99 
Parent-Reported MeasureBaseline Mean, %Final Mean, %Pa
Completed form about child’s development, communication, and/or behavior 72 78 .18 
Completed form about sadness or depression in the child’s mother (<12 mo only) 47 57 .005 
Completed form about family’s needs or other issues affecting child’s health 36 52 .01 
Provider mostly or completely talked about the results of the form (only those who completed a form)b 80 83 .11 
Provider offered a referral or resources to support child’s development, help the child’s mother with sadness or depression, or help family with other needs 79 85 .09 
Care received was above average or bestc 92 92 .99 

Results presented for practices that completed both baseline and final parent surveys (n = 15).

a

P values generated from paired t tests.

b

Response options were dichotomized: mostly and completely versus not at all and somewhat.

c

Response options were dichotomized: 4 (above average) or 5 (best) versus 1 (worst), 2 (below average), or 3 (average).

Practice self-ratings improved for screening, discussion, referral, and follow-up in all areas on a scale of 1 (not yet started) to 4 (fully implemented) (Table 4). The greatest change was seen for SDoH; mean ratings increased for screening (from 2.1 to 3.6), discussion (from 1.6 to 3.4), and referral (from 1.6 to 3.3; all P < .05).

TABLE 4

Practice Self-rating of Implementation Progress on Monthly Report

DomainItemBaselineFinalChangePa
Development Screening 3.6 4.0 0.4 .02 
 Discussion 2.6 3.8 1.2 .003 
 Referral 3.3 4.0 0.7 .006 
 Follow-up 2.6 3.4 0.8 .002 
ASD Screening 3.8 4.0 0.2 .08 
 Discussion 2.9 3.8 0.9 .006 
 Referral 3.6 3.9 0.4 .02 
 Follow-up 2.5 3.3 0.8 .01 
Maternal depression Screening 3.2 3.9 0.7 .009 
 Discussion 2.6 3.6 1.1 .02 
 Referral 2.6 3.6 1.0 .002 
 Follow-up 1.9 2.8 0.9 .01 
SDoH Screening 2.1 3.6 1.5 .03 
 Discussion 1.6 3.4 1.8 .03 
 Referral 1.6 3.3 1.8 .03 
 Follow-up 1.4 2.6 1.2 .006 
Social-emotional development Screening 1.9 2.4 0.6 .05 
 Discussion 1.5 2.3 0.8 .03 
 Referral 1.6 2.2 0.6 .03 
 Follow-up 1.4 1.8 0.4 .03 
DomainItemBaselineFinalChangePa
Development Screening 3.6 4.0 0.4 .02 
 Discussion 2.6 3.8 1.2 .003 
 Referral 3.3 4.0 0.7 .006 
 Follow-up 2.6 3.4 0.8 .002 
ASD Screening 3.8 4.0 0.2 .08 
 Discussion 2.9 3.8 0.9 .006 
 Referral 3.6 3.9 0.4 .02 
 Follow-up 2.5 3.3 0.8 .01 
Maternal depression Screening 3.2 3.9 0.7 .009 
 Discussion 2.6 3.6 1.1 .02 
 Referral 2.6 3.6 1.0 .002 
 Follow-up 1.9 2.8 0.9 .01 
SDoH Screening 2.1 3.6 1.5 .03 
 Discussion 1.6 3.4 1.8 .03 
 Referral 1.6 3.3 1.8 .03 
 Follow-up 1.4 2.6 1.2 .006 
Social-emotional development Screening 1.9 2.4 0.6 .05 
 Discussion 1.5 2.3 0.8 .03 
 Referral 1.6 2.2 0.6 .03 
 Follow-up 1.4 1.8 0.4 .03 

Results are presented for those practices that completed baseline and final monthly narrative reports (n = 16). Practices rated their team’s progress using the following scale: 1 = not yet started, 2 = beginning implementation, 3 = partially implemented, and 4 = fully implemented.

a

P values were calculated on the basis of matched-pairs Wilcoxon rank tests.

TABLE 5

Percentage of Practices Reporting Office System Changes on Practice Survey

Baseline, %Final, %Pa
Office systems in place to support screening processes    
 Prompt the practice team of needed screenings 50 86 .03 
 Identify patients who have missed screenings 21 29 .32 
 Document the results of screenings 79 100 .08 
 Document discussion of screening results with the family 57 100 .01 
 Document needed referrals 79 86 .56 
 Track follow-up on referrals 50 79 .10 
 Document outcomes of referrals 36 79 .03 
 Code and bill for screening 79 100 .08 
Office systems in place to support community connections and follow-upb    
 Maintains an organized list of community resources 29 43 .32 
 Regularly updates community resource list 14 21 .56 
 Has formal connections with referral partners 21 43 .18 
 Uses a standard form to send information to community agencies 21 50 .16 
 Matches child and/or family needs to appropriate referral resources 14 50 .06 
 Uses a specific approach to facilitate referrals 29 29 1.00 
 Follows-up to determine if families have connected with services 14 21 .65 
 Follows-up on the outcome of referrals 21 .32 
Staff training in place for screening    
 Development 93 100 .32 
 ASD 86 100 .16 
 Maternal depression 57 100 .01 
 SDoH 29 100 .002 
 Social-emotional development 43 64 .26 
Patient education resources available on screening areas    
 Development 57 71 .32 
 ASD 21 36 .32 
 Maternal depression 50 .03 
 SDoH 43 .01 
 Social-emotional development 14 43 .10 
Challenges to screening, referral, and follow-upc    
 Lack of time 56 21 .02 
 Lack of appropriate referral sources 54 29 .01 
 Difficulty securing payment for services 53 40 .28 
 EHR issues 41 24 .33 
 Lack of screening tool in appropriate language 33 14 .11 
 Lack of appropriate screening tool 30 <.001 
 Caregiver reluctance 26 31 .57 
 Lack of adequate staff for implementation 20 11 .16 
 Lack of leadership support 11 .22 
Baseline, %Final, %Pa
Office systems in place to support screening processes    
 Prompt the practice team of needed screenings 50 86 .03 
 Identify patients who have missed screenings 21 29 .32 
 Document the results of screenings 79 100 .08 
 Document discussion of screening results with the family 57 100 .01 
 Document needed referrals 79 86 .56 
 Track follow-up on referrals 50 79 .10 
 Document outcomes of referrals 36 79 .03 
 Code and bill for screening 79 100 .08 
Office systems in place to support community connections and follow-upb    
 Maintains an organized list of community resources 29 43 .32 
 Regularly updates community resource list 14 21 .56 
 Has formal connections with referral partners 21 43 .18 
 Uses a standard form to send information to community agencies 21 50 .16 
 Matches child and/or family needs to appropriate referral resources 14 50 .06 
 Uses a specific approach to facilitate referrals 29 29 1.00 
 Follows-up to determine if families have connected with services 14 21 .65 
 Follows-up on the outcome of referrals 21 .32 
Staff training in place for screening    
 Development 93 100 .32 
 ASD 86 100 .16 
 Maternal depression 57 100 .01 
 SDoH 29 100 .002 
 Social-emotional development 43 64 .26 
Patient education resources available on screening areas    
 Development 57 71 .32 
 ASD 21 36 .32 
 Maternal depression 50 .03 
 SDoH 43 .01 
 Social-emotional development 14 43 .10 
Challenges to screening, referral, and follow-upc    
 Lack of time 56 21 .02 
 Lack of appropriate referral sources 54 29 .01 
 Difficulty securing payment for services 53 40 .28 
 EHR issues 41 24 .33 
 Lack of screening tool in appropriate language 33 14 .11 
 Lack of appropriate screening tool 30 <.001 
 Caregiver reluctance 26 31 .57 
 Lack of adequate staff for implementation 20 11 .16 
 Lack of leadership support 11 .22 
a

P values are for the McNemar test.

b

Responses were dichotomized as “to a large extent” versus “to a moderate extent,” “to some extent,” and “not at all.”

c

Challenges and barriers were asked for each screening area separately. Percentages shown are an average of each challenge or barrier across all 5 screening topic areas. P values are for paired t tests used to compare mean percentages.

Practices reported improved office systems, including prompts to conduct screenings (from 50% to 86%; P = .03), documentation of discussion (from 57% to 100%; P = .01), and documentation of referral outcomes (from 36% to 79%; P = .03) (Table 5). On final surveys, more practices reported conducting staff training for maternal depression (from 57% to 100%; P = .01) and SDoH (from 29% to 100%; P = .002). More practices reported having patient educational materials on maternal depression (from 7% to 50%; P = .03) and SDoH (from 0% to 43%; P = .01). Fewer practices reported lack of time (from 56% to 21%; P = .02), referral resources (from 54% to 29%, P = .01), or appropriate screening tool (from 30% to 4%; P < .001) as barriers.

Many practices found that creating protocols to handle identified needs helped to implement and sustain screening. Standardizing electronic templates for well visits was another commonly endorsed change. Other implementation strategies included developing a directory of best referral resources, restructuring staff time to ensure that screening and referral tasks could be managed, and bringing community partners into the clinic for face-to-face introductions to staff. Academic centers involved residents and students in learning sessions, in team participation and mapping of community resources, and in attending conferences aimed at improving education about screening.

Practices reached collaborative aims for development and ASD screening and demonstrated the largest improvements for maternal depression and SDoH. Practices that implemented social-emotional screening also demonstrated large improvements. Referrals for positive screening results improved in all areas, but referral follow-up did not improve consistently. Parents or caregivers also reported screening more frequently after the collaborative, corroborating the improvements observed in chart reviews.

Chart reviews and practices’ self-ratings indicated that they implemented multiple screenings during a relatively short time frame. After making changes such as creating protocols for responding to screenings, clarifying best referral resources, and hosting meetings with community partners, practices described improved office systems and fewer barriers. Practices accomplished these improvements with supports that included in-person and virtual learning opportunities, access to experts and frequent contact with coaches, regular QI data collection and feedback, a stipend to defray expenses, and availability of Maintenance of Certification and Continuing Medical Education credits to enhance engagement of all physicians. Practices anticipated sustaining screening improvements because they implemented structural and system-level changes, such as new protocols, documentation templates, and referral lists, which embedded improved screening into daily practice routines. These findings are corroborated by separate qualitative interviews in which 2 Project I-SCRN practices participated and described strengthening of relationships within clinics, with families, and with community systems.24  Practices were encouraged to continue to monitor screening rates using PediaLink QI18  as an ongoing resource for tracking quality metrics.

Parent or caregiver surveys was one method used to center screening efforts around families. Of note, overall parent satisfaction ratings, a balancing measure, did not change. This suggests that new screenings, with an accompanying increase in written questionnaires, did not adversely affect parents’ perception of care. Obtaining meaningful feedback from parents or caregivers about their screening experiences remains challenging.

To emphasize family centeredness, involvement of parent or caregiver partners was encouraged.25  Most practices did not have a parent or caregiver partner at baseline but successfully involved one during the project; however, many faced typical challenges, including turnover and limited availability. Dedicated sessions at the second learning session provided parents with opportunities to share experiences and provide collective guidance to practices. Practices were given guidance in taking next steps toward greater parent or caregiver involvement such as through parent advisory councils.26 

Consistent with previous studies,13,15,19  follow-up of positive screening results remained challenging. The greatest improvement was seen for development, possibly because baseline screening was high due to previous improvement efforts27  and because referral pathways are clearer than for other screening areas. Multiple barriers remain for closing the loop on referrals. Internal referrals and bidirectional communication through electronic or interpersonal methods facilitated follow-up. For example, one practice with an integrated behavioral health specialist was able to document referral follow-up, and another practice made direct, within-institution referrals to therapists. However, many positive screening results required external referrals to agencies and health professionals with which there were not standardized bidirectional communication systems. Because parents of children with identified problems may experience challenges meeting increased health care needs,28  innovative approaches are needed to improve connections with resources.

Standard EHR packages often lack features for documenting and tracking screenings,29  and this was a persistent barrier to screening improvement. Most practices administered paper screening tools to patients, and staff entered responses, because EHRs frequently did not support electronic questionnaire entry by patients. EHRs often lacked standard forms for documenting screenings; practices reported that building these was time-consuming and costly. These documentation challenges hindered automation of tracking positive results and necessitated manual chart review to identify positive screening results and track referrals. Referral categories such as early intervention, parent support, and early childhood education typically do not exist as structured data fields and therefore are not easily tracked within EHRs. Including these functionalities as a part of standard EHR packages has potential to automate monitoring of quality measures and facilitate sustainability of improved screening and follow-up.

Our findings suggest continued challenges and opportunities for practices in screening for SDoH, including connecting families with resources, given the heterogeneous nature of SDoH and diversity of resources and/or referrals required.10  Several promising models suggest how these challenges can be addressed. North Carolina’s state health leadership is implementing an electronic resource platform with new capability to allow bidirectional communication to community agencies for SDoH.30  Other promising models have been described comprehensively31  and include involvement of navigators32  or parent coaches33  who can facilitate families’ connections with resources.

As with previous studies,29  access to appropriate referral resources was a barrier, such that geographic location, patient language, and payers often limit access. Advocacy at the AAP chapter and state level will be important to ensure payment for screening services, as has been described.34  Similarly, continued state and federal advocacy is important to enact policy and structural changes that support all aspects of screening processes,31,35  including availability of resources for children with identified developmental and social problems.

Limitations are similar to those of other QI initiatives and include internal data collection by practices and limited data quality control. The small number of positive screening results each month reduced the stability of estimates for referral and follow-up measures. Measures were chosen to minimize burden and only reflect one time point; additional measures would be valuable to determine if children receive screenings at all recommended time points. Our data were not sufficient to assess the effect of screening interventions on other preventive services or on cycle time, office revenue, or productivity, which are important areas for future investigation.

Expectations for primary care have broadened to include identifying, addressing, and ameliorating risk factors for children and families.36  Practices may benefit from support and shared learning, as was provided during this collaborative, to identify strategies for managing increased screening. Instead of being an additive burden, screenings can be embedded within practices’ strategies for promoting overall development. Careful attention to integrating screenings in visit flow and emphasizing their potential impact on child health can make implementation possible in multiple screening areas.

We thank the practices, physicians, staff, and families who made Project I-SCRN possible, including the following practices: Alaska Center for Pediatrics, Bozeman Health Pediatrics, Burlington Pediatrics, Carilion Children’s Pediatric Medicine, Child and Adolescent Clinic, Community Pediatric Programs, Duke Children’s Primary Care, Friend Family Health Center, Miyoco Medical Center, Northeast Pediatrics & Adolescent Medicine, Pediatrics on Hudson, Pediatric Partners in Palm Beach Gardens, Texas Children’s, Texas Tech Physicians of El Paso Department of Pediatrics, University of Florida Health Jacksonville Pediatrics, University of Kentucky General Pediatrics at Kentucky Clinic South, University Pediatric and Adolescent Center, and YBPC Pediatrics.

We also thank the expert faculty who contributed to learning sessions and webinars, including, Rich Porcelli, Tessa Pendry, and Tigee Hill of Health Leads; Brenda Liz (Bren) Muñoz of the Georgia Council on Developmental Disabilities; Dr Sara del Campo de Gonzalez of The University of New Mexico Young Children’s Health Center; and Dr Ellen Perrin of Tufts Medical Center. We appreciate statistical review by Victor Silva Ritter and Dr Jason Fine at the University of North Carolina at Chapel Hill. We thank the AAP staff who supported the project, including Sarah Hueneke and Brittany Nagy.

Drs Flower and Earls conceptualized and designed the study, designed the data collection instruments, collected data, conducted the analysis, and reviewed and revised the manuscript; Ms Massie designed the data collection instruments, collected data, conducted the analysis, and reviewed and revised the manuscript; Drs Coker, Zubler, Macias, Gillespie, and Whitaker, Ms Janies, Ms Bassewitz, Ms Steinberg, and Ms DeStigter contributed to the study design, participated in the design of data collection instruments, and reviewed and provided critical input on the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Funded by The JPB Foundation to the American Academy of Pediatrics.

     
  • AAP

    American Academy of Pediatrics

  •  
  • ASD

    autism spectrum disorder

  •  
  • EHR

    electronic health record

  •  
  • I-SCRN

    Improving Screening

  •  
  • Connections with Families

    and Referral Networks

  •  
  • PAC

    project advisory committee

  •  
  • PDSA

    plan-do-study-act

  •  
  • QI

    quality improvement

  •  
  • SDoH

    social determinants of health

  •  
  • TA

    technical assistance

  •  
  • WCV

    well-child visit

1
Lipkin
PH
,
Macias
MM
;
Council on Children With Disabilities
;
Section on Developmental and Behavioral Pediatrics
.
Promoting optimal development: identifying infants and young children with developmental disorders through developmental surveillance and screening
.
Pediatrics
.
2020
;
145
(
1
):
e20193449
2
Sills
MR
,
Shetterly
S
,
Xu
S
,
Magid
D
,
Kempe
A
.
Association between parental depression and children’s health care use
.
Pediatrics
.
2007
;
119
(
4
).
3
Earls
MF
,
Yogman
MW
,
Mattson
G
,
Rafferty
J
;
Committee on Psychosocial Aspects of Child and Family Health
.
Incorporating recognition and management of perinatal depression into pediatric practice
.
Pediatrics
.
2019
;
143
(
1
):
e20183259
4
Palmer
FB
,
Graff
JC
,
Jones
TL
, et al
.
Socio-demographic, maternal, and child indicators of socioemotional problems in 2-year-old children: a cohort study
.
Medicine (Baltimore)
.
2018
;
97
(
28
):
e11468
5
Folger
AT
,
Eismann
EA
,
Stephenson
NB
, et al
.
Parental adverse childhood experiences and offspring development at 2 years of age
.
Pediatrics
.
2018
;
141
(
4
):
e20172826
6
Shonkoff
JP
,
Garner
AS
;
Committee on Psychosocial Aspects of Child and Family Health; Committee on Early Childhood, Adoption, and Dependent Care; Section on Developmental and Behavioral Pediatrics
.
The lifelong effects of early childhood adversity and toxic stress
.
Pediatrics
.
2012
;
129
(
1
).
7
Hyman
SL
,
Levy
SE
,
Myers
SM
;
Council on Children with Disabilities
;
Section on Developmental and Behavioral Pediatrics
.
Identification, evaluation, and management of children with autism spectrum disorder
.
Pediatrics
.
2020
;
145
(
1
):
e20193447
8
Committee on Practice and Ambulatory Medicine
;
Bright Futures Periodicity Schedule Workgroup
.
2019 recommendations for preventive pediatric health care
.
Pediatrics
.
2019
;
143
(
3
):
e20183971
9
Weitzman
C
,
Wegner
L
;
Section on Developmental and Behavioral Pediatrics
;
Committee on Psychosocial Aspects of Child and Family Health
;
Council on Early Childhood
;
Society for Developmental and Behavioral Pediatrics
;
American Academy of Pediatrics
.
Promoting optimal development: screening for behavioral and emotional problems
.
Pediatrics
.
2015
;
135
(
2
):
384
395
10
Chung
EK
,
Siegel
BS
,
Garg
A
, et al
.
Screening for social determinants of health among children and families living in poverty: a guide for clinicians
.
Curr Probl Pediatr Adolesc Health Care
.
2016
;
46
(
5
):
135
153
11
Williams
ME
,
Zamora
I
,
Akinsilo
O
,
Chen
AH
,
Poulsen
MK
.
Broad developmental screening misses young children with social-emotional needs
.
Clin Pediatr (Phila)
.
2018
;
57
(
7
):
844
849
12
Hirai
AH
,
Kogan
MD
,
Kandasamy
V
,
Reuland
C
,
Bethell
C
.
Prevalence and variation of developmental screening and surveillance in early childhood
.
JAMA Pediatr
.
2018
;
172
(
9
):
857
866
13
Carbone
PS
,
Norlin
C
,
Young
PC
.
Improving early identification and ongoing care of children with autism spectrum disorder
.
Pediatrics
.
2016
;
137
(
6
):
e20151850
14
Kerker
BD
,
Storfer-Isser
A
,
Szilagyi
M
, et al
.
Do pediatricians ask about adverse childhood experiences in pediatric primary care?
Acad Pediatr
.
2016
;
16
(
2
):
154
160
15
Marks
KP
,
Page Glascoe
F
,
Macias
MM
.
Enhancing the algorithm for developmental-behavioral surveillance and screening in children 0 to 5 years
.
Clin Pediatr (Phila)
.
2011
;
50
(
9
):
853
868
16
Lannon
CM
,
Flower
K
,
Duncan
P
,
Moore
KS
,
Stuart
J
,
Bassewitz
J
.
The Bright Futures Training Intervention Project: implementing systems to support preventive and developmental services in practice
.
Pediatrics
.
2008
;
122
(
1
).
17
Duncan
PM
,
Pirretti
A
,
Earls
MF
, et al
.
Improving delivery of Bright Futures preventive services at the 9- and 24-month well child visit
.
Pediatrics
.
2015
;
135
(
1
).
18
American Academy of Pediatrics
.
Screening tools. 2017. Available at: https://screeningtime.org/star-center/#/screening-tools#top. Accessed December 7, 2019
19
Van Cleave
J
,
Kuhlthau
KA
,
Bloom
S
, et al
.
Interventions to improve screening and follow-up in primary care: a systematic review of the evidence
.
Acad Pediatr
.
2012
;
12
(
4
):
269
282
20
Institute for Healthcare Improvement
.
The Breakthrough Series: IHI’s Collaborative Model for Achieving Breakthrough Improvement
.
Boston, MA
:
Institute for Healthcare Improvement
;
2003
21
Langley
GJ
,
Moen
RD
,
Nolan
KM
,
Nolan
TW
,
Norman
CL
,
Provost
LP
.
The Improvement Guide: A Practical Approach to Enhancing Organizational Performance
, 2nd ed.
San Francisco, CA
:
Jossey-Bass
;
2009
22
Perla
RJ
,
Provost
LP
,
Murray
SK
.
The run chart: a simple analytical tool for learning from variation in healthcare processes
.
BMJ Qual Saf
.
2011
;
20
(
1
):
46
51
23
Salkind
NJ
, ed.
.
Encyclopedia of Research Design
.
Thousand Oaks, CA
:
SAGE Publications, Inc
;
2010
24
Spain
A
,
Brown
A
,
Sander
A
.
Implementing DULCE and I-SCRN (Research Brief 2)
.
Chicago, IL
:
Chapin Hall at the University of Chicago
;
2019
25
Dillon
AD
.
Parent partners: creative Forces on medical home improvement teams.
2003
. Available at: https://medicalhomeinfo.aap.org/tools-resources/Documents/CMHI-Parent-Partner-Guide.pdf. Accessed December 7, 2019
26
American Academy of Pediatrics, National Resource Center for Patient/Family Centered Medical Home.
For practices. Available at: https://medicalhomeinfo.aap.org/tools-resources/Pages/For-Practices.aspx. Accessed February 9, 2019
27
Earls
MF
,
Hay
SS
.
Setting the stage for success: implementation of developmental and behavioral screening and surveillance in primary care practice–the North Carolina Assuring Better Child Health and Development (ABCD) Project
.
Pediatrics
.
2006
;
118
(
1
).
28
Kogan
MD
,
Vladutiu
CJ
,
Schieve
LA
, et al
.
The prevalence of parent-reported autism spectrum disorder among US children
.
Pediatrics
.
2018
;
142
(
6
):
e20174161
29
Garner
AS
,
Storfer-Isser
A
,
Szilagyi
M
, et al
.
Promoting early brain and child development: perceived barriers and the utilization of resources to address them
.
Acad Pediatr
.
2017
;
17
(
7
):
697
705
30
NCCARE360
.
NCCARE360 quarterly report – January-March 2019. 2019. Available at: https://foundationhli.org/wp-content/uploads/2019/05/NCCARE360-Quarterly-Report-January-March-2019.pdf. Accessed June 14, 2019
31
Beck
AF
,
Tschudy
MM
,
Coker
TR
, et al
.
Determinants of health and pediatric primary care practices
.
Pediatrics
.
2016
;
137
(
3
):
e20153673
32
Gottlieb
LM
,
Hessler
D
,
Long
D
, et al
.
Effects of social needs screening and in-person service navigation on child health: a randomized clinical trial
.
JAMA Pediatr
.
2016
;
170
(
11
):
e162521
33
Coker
TR
,
Chacon
S
,
Elliott
MN
, et al
.
A parent coach model for well-child care among low-income children: a randomized controlled trial
.
Pediatrics
.
2016
;
137
(
3
):
e20153013
34
Coker
TR
,
DuPlessis
HM
,
Davoudpour
R
,
Moreno
C
,
Rodriguez
MA
,
Chung
PJ
.
Well-child care practice redesign for low-income children: the perspectives of health plans, medical groups, and state agencies
.
Acad Pediatr
.
2012
;
12
(
1
):
43
52
35
Bethell
CD
,
Solloway
MR
,
Guinosso
S
, et al
.
Prioritizing possibilities for child and family health: an agenda to address adverse childhood experiences and foster the social and emotional roots of well-being in pediatrics
.
Acad Pediatr
.
2017
;
17
(
suppl 7
):
S36
S50
36
Fierman
AH
,
Beck
AF
,
Chung
EK
, et al
.
Redesigning health care practices to address childhood poverty
.
Acad Pediatr
.
2016
;
16
(
suppl 3
):
S136
S146

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data