Improving Autism Screening and Referrals With Electronic Support and Evaluations in Primary Care

We studied autism screening at 2 pediatric clinics at The University of Utah (intervention clinics) and 27 community clinics in the same health care system. The intervention clinics are staffed by resident and attending pediatricians, serve numerous children with special health care needs, and have a culture that promotes training and improvement. The community clinics are located in 12 multispecialty health centers in 8 cities, including inner-city, suburban, and semirural locations, serving patients with a range of sociodemographic characteristics.

Before intervention, all clinics were using the Modified Checklist for Autism in Toddlers, Revised (M-CHAT-R) for autism screening.¹⁶  The follow-up interview, an additional step in screening to decrease false-positives, was rarely being used, similar to previous reports.^17,18  The intervention and community clinics used an electronic health record (EHR) (Epic; Epic Systems Corporation, Verona, WI) with shared infrastructure and management. In Utah, early intervention services do not include autism evaluation and treatment. To access autism services, children must first be referred to a developmental pediatrician or psychologist for autism diagnosis. After children have a diagnosis of autism, Medicaid insurance covers autism therapy. In Utah, health insurance plans that are not covered by the Employee Retirement Income Security Act are mandated to cover autism treatment, contingent on diagnosis.¹⁹  Therefore, referral for a diagnostic evaluation is crucial for entry into treatment.

Our QI champions included a pediatric resident, 3 faculty attending pediatricians who worked at the intervention sites, and a faculty pediatrician with expertise in the care of children with autism. The faculty champions had experience in QI methodology. Groups of 1 to 3 pediatric residents joined the improvement team each month during their developmental and behavioral pediatrics rotation and were assigned a task related to improvement activities.

We analyzed screening practices before intervention via process mapping and data analysis. We identified several barriers to successful screening and referral and planned process improvements to address these issues (Table 1). In an analysis of data from the previous 3 years at the intervention clinics, we found that 51% of children were screened for autism and only 10 of 2001 children (0.5%) were diagnosed with autism. The prevalence of autism diagnosed in children aged 4 years in 2016 was threefold greater at other US sites (1.6%) than our clinics.²⁰  We determined that the process of screening in our clinics was not adequately identifying children with autism.

We sought to improve autism screening and referral by modifying screening and its supporting processes in the EHR and by better integrating referrals into screening processes. The process improvements were executed in 3 phases (Table 1). We first addressed the sensitivity of screening. Studies conducted in large health care systems in Utah and Pennsylvania revealed that the M-CHAT-R has a sensitivity of 33% to 39% in realistic conditions without research support.^18,21  Therefore, we changed our screening instrument to the Parent’s Observation of Social Interaction (POSI), which is embedded in a broadband developmental screen, is shorter than the M-CHAT-R, and includes questions about the consistency of the child’s behavior (ie, sometimes or always).²²  The POSI has greater sensitivity than the M-CHAT-R (POSI, 75%–94%; M-CHAT-R, 50%–78%) and similar, although somewhat lower, specificity (POSI, 41%–74%; M-CHAT-R, 54%–84%).^22,23 

For intervention phase 1, we implemented screening with the POSI at intervention clinics. We prepared the physicians and staff to use the POSI and interpret the results with in-person training and e-mailed descriptions. We programmed an electronic version of the POSI with automatic scoring into the EHR before implementation. For both the M-CHAT-R and the POSI, all clinics used a paper version, which was then entered by staff into an electronic version with automatic scoring. Thus, the difference between the clinics was the change in instrument and additional training for administering and scoring the POSI. We changed our note templates for the visits at ages 18, 24, and 30 months to include a sentence prompting clinicians to refer a child for autism evaluation when the autism screening result suggested autism risk. The note template was required at intervention clinics and available, but not required, at the community clinics.

In intervention phase 2, to help staff remember to screen every child at the appropriate ages, we added an automatic reminder for autism screening to the EHR health maintenance screen (Fig 1). The health maintenance screen reminds physicians and clinic staff about preventive care, such as vaccinations. We received approval from the local clinical decision support committee to implement the reminder system-wide.²⁴  Therefore, intervention and community clinics were equally affected by the automatic reminder.

In intervention phase 3, we added a referral option that clinicians could use for rapid access to autism-specific evaluation. Using an online tutorial, we trained 3 clinicians in the intervention clinics to administer an observational assessment known as the Screening Tool for Autism in Toddlers (STAT) (http://stat.vueinnovations.com/about/stat-training).²⁵  The STAT requires a 30-minute patient visit and has an estimated sensitivity of 78% to 100% and specificity of 57% to 78% for autism.^25–27  We offered the STAT as a referral option for children who had a POSI result suggestive of autism and for whom the clinician had sufficient concerns about autism that would indicate the need for referral for autism evaluation. Children who had a STAT result suggestive of autism were referred for expedited autism diagnostic evaluation, which was performed by a multidisciplinary team in our university-based developmental assessment clinic. Children who had an STAT result that did not suggest autism did not receive further autism evaluations unless the clinician felt they still needed further evaluation at the developmental clinic. Most children with a negative STAT result were instead referred for other clinically indicated services, such as speech therapy, hearing screening, or early intervention.

We studied well-child visits for children aged 16 to 30 months in a 2-year study from July 2017 to June 2019. We extracted EHR data containing completed screening forms, screening results, referrals placed from eligible visits, and demographic information about subjects.

The primary outcome measure was the proportion of visits with screening performed. The secondary outcome measure was the proportion of visits with a referral for autism evaluation, either with an autism evaluation in primary care by using the STAT or with a diagnostic evaluation at the developmental assessment clinic (both were options for referrals). The proportion of visits with a positive screen result was added as a post hoc descriptive measure to further compare the performances of the M-CHAT-R and the POSI.

We analyzed patient demographics using a χ² test. We allowed for multiple visits for the same patient because the outcome measures were focused on the clinician’s actions at the visits. We compared the results for intervention versus community clinics with a χ² test and interrupted time series (ITS) to quantify the rate of change in outcomes over time in the intervention versus community clinics and the intervention phases versus baseline.²⁸  The ITS regression line (trendline) from the start to the end of each phase was quantified and reported as the slope (95% confidence interval [CI]; P). The trendline interruption associated with an event that defined the start of a new study period was the difference between the y-axis values of the trendlines of the new period and previous period, determined at the time of the first measurement after the event.

To analyze the change in referrals associated with intervention, we analyzed the rate ratio of referrals. We applied a mixed-effects Poisson regression, with count of referrals as the outcome, phase of intervention as the main predictor, and nesting by physician to control for referral patterns between physicians. Covariates included the interaction between clinic group and intervention phase, age (<24 vs ≥24 months, including screening at the 30-month visit), race, language, sex, insurance status, and physician training level (resident versus attending). We used backward elimination to discard nonsignificant predictors from the final model. The interaction between phase of intervention and clinic group was significant, so the results were stratified by clinic group and the adjusted rate ratios from the stratified models were reported.

Statistical software was used for the χ² test and Poisson regression (R version 3.5.2; R Foundation for Statistical Computing, Vienna, Austria) and the ITS (Stata IC 15.1; Stata Corp, College Station, TX).^29,30 

We received an exemption for QI from The University of Utah Institutional Review Board.

We studied 12 233 visits (intervention clinics, 4155 visits; community clinics, 8078 visits) over 2 years, including 4889 visits (40%) that were from patients who had >1 visit during the study period. The frequencies of visits from patients of each demographic characteristic are given in Table 2. There were slightly more visits from white, Spanish-speaking, and privately insured children at the intervention clinics than the community clinics (Table 2).

From baseline to the intervention periods (phases 1–3 combined), the proportion of visits with screening increased by 51% in intervention clinics (58.6%–88.8%) and by 21% in community clinics (43.4%–52.4%) (Table 3). The proportion of referrals increased 1.5-fold in intervention clinics (1.3%–3.3%) but not in community clinics (1.2% vs 1.3%) (Table 3). After changing to the more sensitive POSI in the intervention clinics, the number of visits with a positive screen result increased from 97 (4.7%) to 281 (13.5%). A few providers at the community clinics started using the POSI. However, the POSI was used in community clinics in only 55 of 4046 visits (1.4%), with an average use of 1.7 times per week.

The ITS revealed that screening was neither increasing nor decreasing during the baseline period in either group (Table 4). During phase 1, screening frequency increased markedly (positive slope) versus baseline in the intervention clinics but not in the community clinics (Fig 2). Screening frequency also increased during phase 3 in the intervention clinics and during phases 2 and 3 in the community clinics. The increase in screening frequency in the intervention group observed in phase 1 was sustained in phases 2 and 3. The trendline interruptions were not significant for intervention or community clinics at any phase break except for a decrease at phase 2 in the intervention clinics. The slope was greater for the intervention clinics than the community clinics in phase 1 but similar in phases 2 and 3.

In the intervention clinics, referrals were 3.4 times more frequent for visits in phase 3 than in the baseline period (95% CI: 2.0 to 5.8), providers referred boys more frequently than girls, and providers referred less frequently when children had private insurance (Fig 3A). In the community clinics, providers referred boys and older children more frequently, but there was no effect of phase of intervention on referrals (Fig 3B).

This QI study increased the proportion of visits with autism screening and referrals for autism-specific evaluation. Effective interventions included training the staff to administer a more sensitive screening instrument, prompting referral for scores suggestive of autism, adding reminders to the EHR, and adding autism evaluation in intervention clinics. Screening increased to almost universal in the intervention clinics, and community clinics continued to have an upward trend at the end of the study. The study may have improved the identification of autism by increasing the frequency of referrals for diagnostic evaluation.

A strength of this study was the comparison between intervention and community clinics within the same health care system, which enabled us to draw conclusions about the changes in screening that were attributable to the intervention and the effectiveness of a remotely administered intervention. Previous studies revealed increased screening with decision support and automatic scoring, but few had comparison sites or implemented the interventions at a comparison site.^12,13  In our study, the community clinics had an increase in screening frequency with only automatic reminders, which may be disseminated remotely or with minor educational intervention.

With each study phase, we observed increased referrals for autism-specific evaluations. The change to a more sensitive screening instrument increased the frequency of screening results suggestive of autism and informed our improvement team of the need to implement autism evaluation in primary care to avoid overwhelming our referral system. A novel aspect of our study was the creation of a referral option for autism-specific evaluation within primary care. Most previous studies of autism screening and referral used research-funded evaluations rather than expanding clinical capacity for autism evaluation within primary care.^14,16,27,31  One study that used community providers to administer the STAT increased eligibility for autism treatment.³²  As practices increase screening frequency and sensitivity, there may be a greater need for primary care or community-based autism evaluations. Increasing screening without the full intervention described here, including the STAT, would likely increase wait times for autism diagnostic testing.

It is unclear why children with private insurance were less likely to be referred for autism evaluation. One possibility is that families with high-deductible insurance plans, similar to those who are uninsured, may be unable to pay for evaluation. There may be a demographic association with parental refusal of provider-offered autism evaluation, similar to associations noted with vaccine refusal.³³  Qualitative studies of provider referral decisions and parental reaction to autism screening could help clarify referral bias or parental refusal of services.

Limitations of the current study include the implementation of multiple changes simultaneously and limited understanding of the acceptability of the intervention. We implemented several changes in phase 1 simultaneously, which does not enable us to discern whether one of these changes or the combination stimulated increased screening. We did not evaluate whether the change in screening was acceptable to physicians and families. Other studies have revealed increased self-efficacy of physicians with interventions that included more education.^10,11  In future studies, researchers should assess whether physicians felt they understood the screening instrument and judged it effective in screening and referral for diagnosis of autism because this could affect the long-term sustainability of a changed screening process. Furthermore, it may be useful to add discrete EHR fields to enable providers to document the reasons for referral decisions because this information may inform future educational and QI interventions.

In addition, no long-term outcome data are available about our patients. Some children in our study have been diagnosed with autism, but others have not yet been evaluated. Future studies may be used to determine if the increase in screening and referrals improves the time to diagnosis and treatment and long-term functional abilities of children diagnosed with autism. In addition, further study is needed with diagnostic outcomes to fully understand the performance and acceptability of the STAT as a referral option to expedite autism-specific evaluation or rule out autism. In addition, some clinics may be limited in their ability to use the STAT in primary care because of the barriers of training providers and purchasing materials. However, the time-based billing for lengthier appointments and billing for developmental testing help to cover cost.

A shorter and more sensitive screening instrument, staff training, clinical decision support, EHR automatic reminders, and primary care integration of autism evaluations were associated with increased autism screening and referrals.

We thank the pediatric residents at The University of Utah, Dr Kensaku Kawamoto for assistance with EHR clinical decision support, Dr Gregory Stoddard for statistical advice, and Dr Molly Conroy for editing and advice.

CI

confidence interval

EHR

electronic health record

ITS

interrupted time series

M-CHAT-R

Modified Checklist for Autism in Toddlers, Revised

POSI

Parent’s Observation of Social Interaction

QI

quality improvement

STAT

Screening Tool for Autism in Toddlers