Diagnostic Accuracy of Primary Care Clinicians Across a Statewide System of Autism Evaluation

This study took place within the Early Autism Evaluation (EAE) Hub system, a statewide network of community PCPs trained to provide streamlined diagnostic evaluations for young children, aged 14 to 48 months, at risk for ASD.³⁹  As outlined in McNally Keehn et al,³⁹  EAE Hub clinician training involved didactics in ASD evaluation of young children, as well as a clinical practicum with practice-based coaching until mastery of all components of the standard clinical diagnostic evaluation was achieved. After initial training, all PCPs participated in a virtual longitudinal learning collaborative (which meets every month) with content on challenging case presentations, updated ASD diagnosis and care management practice standards, billing and coding guidance, and information on statewide ASD resources. EAE Hubs receive referrals from regional PCPs for evaluation of children determined to be at risk for ASD on the basis of surveillance and/or developmental screening, and then follow a standard clinical evaluation protocol including administration/review of standard developmental and autism screening tools (ie, Ages and Stages Questionnaire-3 and Modified Checklist for Autism in Toddlers–Revised With Follow-Up), a developmental history and Diagnostic and Statistical Manual of Mental Disorders, Fifth Edition, focused ASD clinical interview, physical examination, and administration of an observational assessment of ASD (Screening Tool for Autism in Toddlers⁴¹ ). The EAE Hub clinician then issues a best-estimate ASD diagnosis and report with clinical recommendations, including information on community and statewide interventions and resources for children with ASD and developmental disabilities.

Standards for Reporting Diagnostic Accuracy guidelines were followed in the design and conduct of this prospective diagnostic study. Seven EAE Hubs set within primary care practices (ie, including 6 large health system group practices and 1 private practice) referred children, aged 14 to 48 months evaluated for ASD in the community primary care setting, to the study between June 2019 and August 2022. To be included, children were aged 14 to 48 months at time of EAE Hub evaluation and had an English-speaking primary caregiver/guardian. This study was approved by the Indiana University School of Medicine institutional review board, and written informed consent was obtained from legal guardians of all participants.

EAE Hubs were recruited into the study in a nonrandom, staggered manner during the study period; site startup was impacted by coronavirus disease 2019 institutional research and patient care regulations. Each site referred a prospective, consecutive sample of children who received an EAE Hub evaluation after site startup until ∼20 children from each site were enrolled (note: Site 1 recruited a greater number of participants because they served as a pilot and study site; Site 4 recruited fewer participants because of relocation of the EAE Hub clinician during the study period). This recruitment procedure allowed the study team to maintain diagnostic blindness and assess children with both ASD and non-ASD outcomes without referral bias. During EAE Hub evaluations, clinicians (or a member of the EAE Hub clinical team; eg, nurse or medical assistant) provided caregivers of children evaluated with a study brochure, brief verbal description of the study, and obtained signed consent to share contact information with the study team for recruitment and enrollment. Once enrolled, an electronic caregiver-report survey (ie, caregiver-reported demographic data on child race/ethnicity and caregiver/family income and education level) and EAE Hub evaluation data (ie, index ASD diagnosis and clinician diagnostic certainty) were collected by a member of the study team (Fig 1). The study team, consisting of a licensed clinical psychologist (R.M.K., B.E., or T.R.) with expertise in evaluation of ASD in toddlers and young children and clinical research technician (advanced graduate student or postdoctoral fellow: G.K., L.H., or A.M.M.), traveled to the EAE Hub to conduct a follow-up gold-standard ASD diagnostic assessment within 16 weeks of the initial EAE Hub evaluation. Figure 1 and Supplemental Information detail the outcome measures, including child, caregiver, and clinician measures, administered to obtain a best-estimate ASD diagnosis (reference standard diagnosis). Participants were compensated with a gift card in the amount of $25 per hour of completed research evaluation (up to a maximum of $75).

Data analyses were performed using SPSS (IBM SPSS Statistics, Version 28, Armonk, NY: IBM Corp) and JMP (JMP, Version 13.0, Cary, NC: SAS Institute Inc.). Continuous variables are reported as means and SDs, and categorical variables are reported as absolute frequencies and percentages. Diagnostic accuracy, the primary outcome of interest, was calculated by comparing percentage agreement between the EAE Hub diagnosis and ASD specialist on categorical ASD diagnosis (ASD; non-ASD). Chance-corrected agreement (κ) and accuracy indices of sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and the 95% confidence interval (CI) are reported. There were no missing data or indeterminate results regarding index or reference diagnoses. Variability in diagnostic accuracy by EAE Hub site and clinician was examined via a series of χ² and Fisher’s exact tests. Exploratory post-hoc analyses were conducted to examine diagnostic, demographic, and phenotypic differences between true positive (TP) and false negative (FN) outcome groups to understand differences between children who were correctly diagnosed with ASD and those who were missed (or unable to be definitively diagnosed) by nonspecialist clinicians. χ² and 2-sided t tests were used to compare categorical and continuous variables, respectively. Finally, an additional subset analysis of accuracy indices for cases with EAE Hub clinician diagnostic certainty ratings of Highly or Completely Certain was conducted to examine whether agreement improved on the basis of clinician perceptions of diagnostic confidence. A sample size of N = 126 provided an upper/lower limit 95% CI width of 0.067 for overall diagnostic agreement, assuming 82% agreement (ie, as observed in this study).

Participant flow through the study is detailed in Fig 2. Of 182 referred children, 131 enrolled, and index and reference standard diagnosis evaluations were included in the final analysis for 126 children across 6 EAE Hubs. Ten clinicians conducted a mean of 12.6 (SD = 7.4) evaluations (Supplemental Table 4 for EAE Hub clinician demographics and learning collaborative participation). Mean age of children was 2.6 years; 77% (n = 97) were male, 14% (n = 18) were Hispanic/Latinx, and 66% (n = 84) were non-Latinx white (Table 1). Across all children evaluated, scores on measures of developmental and adaptive skills fell well below the average range and ASD symptom severity was in the moderate range.⁴²  Seventy-five percent (n = 94) of children had a reference diagnosis of ASD; 10% had global developmental delay (GDD) (n = 13), 10% had language delay (n = 13), and 5% (n = 6) had another emotional, behavioral, or medical concern (n = 6).

Of 126 children evaluated, ASD diagnosis was consistent between the EAE Hub evaluation (index diagnosis) and expert research evaluation (reference diagnosis) for 82% (n = 103) of cases (Table 2). Chance-corrected diagnostic agreement was moderate, κ = 0.580 (95% CI, 0.429–0.731). Sensitivity, or correct classification of ASD diagnosis, was 81.5% (95% CI, 72.4–88.1), whereas specificity, or correct classification of non-ASD diagnosis, was 82.4% (95% CI, 66.5–91.7). PPV was 92.6% (95% CI, 84.8–96.6) and NPV was 62.2% (95% CI, 47.6–74.9). Overall, 60% (n =75) of cases were TP (index = ASD; reference = ASD), 5% (n = 6) were false positive (FP: index = ASD; reference = non-ASD), 22% (n = 28) were true negative (TN: index = non-ASD; reference = non-ASD), and 14% (n = 17) were FN (index = non-ASD; reference = ASD) Table 2).

There was no difference between EAE Hub sites in overall accuracy (ie, accurate versus not; TP + TN versus FP + FN; P = .89) or proportion of FN (compared with TP) cases (P = .67) (Supplemental Table 5). Similarly, there was no difference in overall accuracy (P = .24) or proportion of FN cases (P = .09) by EAE Hub clinician for those submitting data for ≥5 children (n = 8).

Descriptive, clinical, and phenotypic data by diagnostic group can be found in Supplemental Table 6. To address the question of what diagnostic, demographic, and/or phenotypic factors may be associated with FN diagnoses made by trained PCPs, we conducted an exploratory analysis of differences between TP and FN cases (Table 3). There was a significant difference in dichotomized (ie, Highly or Completely Certain versus all other ratings) index clinician diagnostic certainty ratings between TP and FN groups, P = .002, with a higher proportion of Highly–Completely Certain ratings for the TP (95%; 71 of 75), as compared with the FN (65%; 11 of 17) group. Similarly, index clinicians flagged a significantly higher proportion of FN cases (69% of FN; 17% TP) for specialty follow-up evaluation, P ≤ .001. There were no demographic differences by age or sex (Ps > .20). Across measures of developmental and adaptive skills (ie, Mullen Scales of Early Learning Verbal Developmental Quotient, Nonverbal Developmental Quotient, and Early Learning Composite; Vineland-3 Adaptive Behavior Composite), there were significant differences between TP and FN cases (all Ps < .001), with the TP group evidencing significantly greater impairment as compared with the FN group. There was no significant difference between Autism Diagnostic Observation Schedule, Second Edition, Calibrated Severity Scores between the TP and FN groups (P = .28), suggesting no meaningful differences in ASD symptom severity between groups. To address factors that may be associated with FP diagnoses, we examined differences between TN and FP cases (Supplemental Table 7). Although these results must be interpreted with substantial caution given the small sample size, it appears that there may be a trend toward older age in the FP group.

Among the subset of cases in which an EAE Hub clinician rated diagnostic certainty to be Highly or Completely Certain (N = 105), sensitivity was 84.5% (95% CI, 75.3–90.7) and specificity was 90.5% (95% CI, 71.1–97.3). PPV was 97.3% (95% CI, 90.5–99.2) and NPV was 59.4% (95% CI, 42.3–74.5).

In this prospective diagnostic study, we found 82% agreement between trained primary care clinicians and blinded expert research evaluation on categorical ASD diagnosis of children aged 14 to 48 months. Accuracy indices of sensitivity, specificity, and PPV were high (ie, 81.5%, 82.4%, and 92.6%, respectively), whereas NPV was substantially lower (ie, 62.2%). There were no statistically significant differences in accuracy by EAE Hub site or clinician. Diagnostic disagreements were predominately FN cases in which EAE Hub clinicians had difficulty differentiating ASD and GDD. Clinicians flagged most of these cases for follow-up specialty evaluation. FN (as compared with TP) cases were associated with a differential diagnostic and phenotypic, but not demographic, profile. When analysis was restricted to cases in which EAE Hub clinicians rated their diagnostic certainty high, measures of sensitivity, specificity, and PPV improved. To our knowledge, this is the largest study to date that evaluates diagnostic accuracy of a coordinated system of diagnosis in the primary care setting. Notable strengths of this study include the diversity of included primary care index clinicians (ie, from large health system group practices, federally qualified health centers, and private practices) and children evaluated (ie, from diverse socioeconomic and family education backgrounds), large sample size, and rigorous methodology (ie, including blinded reference standard evaluations).

Although existing reports of primary care-based models of ASD diagnosis show promising evidence for improved service access and acceptable accuracy, studies have been limited by small sample size and reduced methodological rigor, or have not used a standard approach for training and diagnostic evaluation.³⁰  Nonetheless, our 82% diagnostic agreement between index and reference diagnosis is comparable to that of others who have reported rates of agreement between PCP and expert evaluation between 71% and 85%.^34,43,44  Importantly, across EAE Hub sites and clinicians, including PCPs and nurse practitioners, there was no difference in overall accuracy or rate of FN cases. Given the small number of NPs in our study, future examination of accuracy with a larger sample of diverse clinicians is needed.

Our study is the first to report on ASD accuracy metrics between nonspecialist clinicians and expert diagnosis when a standard training and clinical pathway is followed. Findings suggest that PCPs who receive specialty training are highly reliable when they confirm an ASD diagnosis, as evidenced by our very low rate of FP cases (6%) and high PPV (92.6%). Clinicians were unable to make a definitive diagnosis or missed ASD in 14% of cases, resulting in low NPV (62.2%). Similar to the findings of Penner et al,³⁴  FN cases evidenced higher verbal and nonverbal developmental level and adaptive skills, though most in our study still met criteria for GDD. Notably, there was no difference in ASD symptom severity between TP and FN cases, suggesting that index clinicians may place more emphasis on developmental impairment than ASD-specific symptoms when making diagnostic decisions. FN cases were associated with lower index clinician diagnostic certainty and higher rates of referral (69% of cases) for specialty evaluation, suggesting that clinicians recognized that these children demonstrated a more complex profile, making differential diagnosis between GDD and ASD challenging. When analysis of diagnostic agreement was restricted to only cases for which index diagnostic certainty was high, sensitivity, specificity, and PPV increased, suggesting that primary care clinicians perceive their ability to render a correct ASD diagnosis with high accuracy. Future research should evaluate whether triaging cases for specialty evaluation on the basis of the child’s overall developmental level (ie, those with higher developmental skills) and/or low clinician diagnostic certainty may mitigate the rate of FN diagnosis in the primary care setting.

A primary limitation of the current study is the high proportion (75%) of reference ASD diagnosis in the sample, resulting in low sample size for comparisons by accuracy subgroup. Because of small sample size, we did not use modeling approaches to adjust for potential correlated outcomes of patients clustered within EAE Hub sites or clinicians; however, we did examine site and clinician differences on overall accuracy and proportion of FN cases, and results were not significant. We also did not collect data on those children evaluated in the EAE Hub system who did not consent to participate, and thus we cannot rule out unmeasured bias in our findings or confirm the generalizability of our findings to all young children who require ASD evaluation. Inclusion of only children with English-speaking caregivers limits the generalizability of our findings, and further solutions to ensure equitable access to diagnostic evaluations are necessary. Although we asked clinicians to flag children that required specialty follow-up evaluation, we designed the study to force index clinicians to make a binary (ASD present/absent) choice about ASD outcome, perhaps artificially deflating accuracy indices because of caution against overdiagnosis. Finally, although an initial ASD diagnosis is needed to access specific services, longitudinal developmental evaluation is important for individualized intervention and prognosis planning.⁴⁵  As such, tiered diagnostic approaches represent an important and promising solution to one component of the larger ASD service bottleneck problem.

Tiered diagnostic approaches, including primary care-based models such as the EAE Hub system, are now being tested as a solution to address the need for increased ASD diagnostic evaluation capacity.^18,31  The EAE Hub system was developed with the primary goal of lowering the age of ASD diagnosis through providing streamlined access to localized diagnostic evaluation within the primary care setting. We have previously shown that the EAE Hub model, which involves intensive training for PCPs in ASD diagnosis and ongoing participation in a longitudinal learning collaborative, is feasible,³⁹  with >4000 children evaluated since 2012. In the present prospective diagnostic study, we extend the empirical support for this model by demonstrating a high level of diagnostic accuracy (ie, 82%) in a sample of diverse community PCPs and at-risk children. Additional research is needed to understand implementation promotors and barriers to broad scale-up of community-based ASD models, as well as replication and comparative effectiveness studies that allow for determination of the key components of training and model implementation necessary for success. Testing strategies aimed to mitigate FN cases will be an essential next step in ensuring the accuracy and quality of streamlined, community-based ASD evaluations. Further, it will be critical to develop and test adaptations of tiered diagnostic approaches for non-English speaking children, as well as to determine whether these types of models lead to earlier intervention enrollment, both efforts currently underway in the EAE Hub system. Collectively, the study of innovative diagnostic models has important implications for how future population health solutions that address the ASD diagnosis crisis are designed and implemented. ASD experts, self-advocates and families, and other health service stakeholders (ie, insurers, service providers) must work together to construct and put into action flexible, evolving, evidence-driven health policies that account for scientific innovation and advancements in ASD diagnosis.^17,46 

We thank Mary Delaney for her work as a practice liaison for the Indiana University School of Medicine EAE Hub leadership team, as well as Yvonne Purcell for her support to the team during the data collection phase of the study. We also thank the EAE Hub clinicians for their commitment to the children and families of Indiana, our learning collaborative network, and their participation in this research. Finally, and most importantly, we thank the children and caregivers who participated in this study.

ASD

autism spectrum disorder

CI

confidence interval

EAE

Early Autism Evaluation

FN

false negative

FP

false positive

GDD

global developmental delay

NPV

negative predictive value

PCP

primary care provider

PPV

positive predictive value

TN

true negative

TP

true positive