Video Abstract

Video Abstract

Close modal
BACKGROUND:

Persistent disparities exist in early identification of autism spectrum disorder (ASD) among children from low-income families who are racial and/or ethnic minorities and where English is not the primary language. Parental literacy and level of maternal education may contribute to disparities. The Developmental Check-In (DCI) is a visually based ASD screening tool created to reduce literacy demands and to be easily administered and scored across settings. In a previous study, the DCI showed acceptable discriminative ability between ASD versus non-ASD in a young, underserved sample at high-risk for ASD. In this study, we tested the DCI among an unselected, general sample of young underserved children.

METHODS:

Six hundred twenty-four children ages 24 to 60 months were recruited through Head Start and Early Head Start. Parents completed the DCI, Modified Checklist for Autism in Toddlers, Revised with Follow-Up, and Social Communication Questionnaire. Children scoring positive on any measure received evaluation for ASD. Those screening negative on both Modified Checklist for Autism in Toddlers, Revised with Follow-Up and Social Communication Questionnaire were considered non-ASD.

RESULTS:

Parents were primarily Hispanic, reported high school education or less, and had public or no insurance. The DCI demonstrated good discriminative power (area under the curve = 0.80), performing well across all age groups, genders, levels of maternal education, primary language, and included ethnic and racial groups. Item-level analyses indicated that 24 of 26 DCI items discriminated ASD from non-ASD.

CONCLUSIONS:

The DCI is a promising ASD screening tool for young, underserved children and may be of particular value in screening for ASD for those with low literacy levels or with limited English proficiency.

What’s Known on This Subject:

Early detection of autism spectrum disorder (ASD) helps link children to intervention that improves functional outcomes. Existing ASD screening tools demonstrate questionable accuracy among young children from low-income, ethnic and racial minority families, which may contribute to disparities in service access.

The Developmental Check-In, a primarily pictorial ASD screening tool, demonstrates promise in identifying ASD risk among young underserved children in the general population. Screening with the Developmental Check-In may potentially reduce disparities in identification of ASD related to ethnicity and racial background.

Early detection and diagnosis of autism spectrum disorder (ASD) is needed to link children to early intervention, which can lead to improved outcomes.1,2  Children with ASD from poor, racial, or ethnic minority groups are identified later, on average, than other children37  and often are misdiagnosed.7,8  When they receive diagnoses at young ages, they have more severe symptoms than white children, suggesting that those less severely impaired often are missed.911  These disparities are compounded among Latino individuals with limited English proficiency.12

Although the American Academy of Pediatrics recommends using a standardized ASD screening tool at 18 and 24 months,13,14  existing tools may not accurately identify autism in underserved populations.15  For example, the Modified Checklist for Autism in Toddlers, Revised (M-CHAT-R)16  overidentifies children of low-income, racial minority parents or parents with less education.1719  Even with the recommended follow-up interview designed to reduce false positives, positive predictive value (PPV) is lower in children of color and those from lower-income households.20  Although Dai et al21  found no racial differences in PPV for the Modified Checklist for Autism in Toddlers, Revised, with Follow-Up (M-CHAT-R/F) interview, PPV was only 0.40 to 0.42. Another common ASD screening tool, the Social Communication Questionnaire (SCQ),22  has decreased sensitivity among children with Black or multiracial mothers or those with lower income or education.23,24

Literacy demands of ASD screening tools may influence their accuracy. Parent response on these tools often is inconsistent with parent descriptions given during follow-up interviews, particularly for those with limited English proficiency or formal education, who express confusion about the meaning of written items.15  Pictorial tools may reduce literacy demands, although few have been created, with most designed for use among non–North American populations25,26  or focusing only on select features of ASD.27

Although the average age of autism diagnosis in the United States is 4 years,28  many screening tools are validated on samples of children <3. Among minority and poor children, in whom symptoms often are missed or misinterpreted, having screening tools that are effective for preschool-aged children may be particularly important.

In response to these concerns, we created the Developmental Check-In (DCI), a visually based ASD screening tool. We conducted a preliminary validation study with 376 children ages 24 to 60 months from racial or ethnic minority, low-income, or limited English proficiency families.15  Of the sample, 288 were considered at-risk because they had been referred for ASD evaluation. An additional 88 children from community day care centers were also included. The DCI demonstrated utility in discriminating ASD from non-ASD (area under the curve [AUC] = 0.75), and 26 of the 28 DCI items predicted ASD status. The current study extends validation of the DCI in an unselected sample of young underserved children.

The Rutgers Biomedical Health Sciences Institutional Review Board approved this study.

We recruited children ages 24 to 60 months from Early Head Start and Head Start programs in 4 low-income communities in New Jersey. Staff distributed invitation packets to all parents, consisting of the DCI, M-CHAT-R, SCQ, short demographic form, and consent form. Both the M-CHAT-R and SCQ were included because of their common use in clinical care and research. Parents reviewed and signed the consent form to participate. If parents consented, Head Start staff distributed the DCI, M-CHAT-R, and SCQ to teachers. Parents who completed the screening tools were entered into a drawing for a $100 gift card. Research staff collected completed packets from sites weekly, scored screeners, and contacted parents to complete missing data. If a child screened at medium risk on the M-CHAT-R (completed by parent or teacher), research staff called (repeatedly) to conduct the follow-up interview in the parent’s primary language (English or Spanish). Those who continued to screen positive after the interview were considered positive. Clinical evaluations were offered to any child who screened positive on the parent- or teacher-completed M-CHAT-R/F, SCQ, or DCI. This allowed for evaluation of children who would have not been considered at-risk based only on results of the parent-completed M-CHAT-R/F and SCQ. Teacher screening results were used to increase the pool of children offered clinical evaluations but were not otherwise used in analyses. Although the DCI was scored and used to identify children eligible for clinical evaluation, we did not use it in any way to determine ASD status. DCI Development of the DCI is described elsewhere.15 Created in English and Spanish, it consists of 26 photographs grouped by domain: communication (4 items), play (6 items), social (8 items), and behavior (8 items). Each photograph has a brief descriptor. Cutoff scores based on age of child suggested by earlier work15 were used: cutoff was ≥7 for children 24 to 35 months, ≥4 for children 36 to 47 months, and ≥3 for children 48 to 60 months. See Fig 1 for a sample item. The same cutoff scores were used with parent and teacher-completed versions. FIGURE 1 Sample DCI item. FIGURE 1 Sample DCI item. Close modal M-CHAT-R/F The M-CHAT-R16,29 is a 20-item, parent-report ASD screening tool used to examine children’s development. It has been validated for children ages 16 to 48 months.30 A score of 0 to 2 is considered low risk, 3 to 7 is considered medium risk, and ≥8 is considered high risk. A follow-up interview is given for scores between 3 and 7. In a low-risk sample of toddlers, the MCHAT-R/F AUC was 0.907, and PPV was 0.45 to 0.51.29 SCQ The SCQ22 is a 40-item parent report ASD screening measure, originally designed for children ≥48 months, that has demonstrated acceptable rates of sensitivity and specificity.22 Researchers in previous studies extended the age range down to 24 months, with AUCs ranging from 0.62 to 0.90.23,24,3034 In this study, we used the current form of the SCQ and a cutoff score of 15, as recommended by the SCQ manual.22 Screening packets were distributed for 1648 children; 747 parents returned a packet. After excluding subjects with missing data, 637 subjects remained in the sample. In Figure 2, we depict participant inclusion. FIGURE 2 Procedure flow. FIGURE 2 Procedure flow. Close modal Developmental Evaluation Measures The Autism Diagnostic Observation Schedule-Second Edition (ADOS-2),35 Mullen Scales of Early Learning (MSEL),36 and Adaptive Behavior Assessment System, Second Edition37 were administered during clinical evaluation and considered in formulating diagnoses, but scores were not analyzed in this study. ADOS-2 The ADOS-2, is a semistructured, play-based assessment for ASD with standardized administration and scoring schema. Sensitivity for modules 1 and 2 ranges from 0.77 to 0.98, and specificity ranges from 0.19 to 0.94 depending on language level, age, and diagnosis.35 The measure of interest used in analyses was overall algorithm classification (autism and autism spectrum were combined as “ASD”). This algorithm classification also helped inform clinical diagnosis. MSEL MSEL is a comprehensive assessment of development normed for children ages birth to 5 years, 8 months that was administered to measure cognitive and language functioning. For this study, standard scores for all scales except gross motor were the measures of interest used to inform clinical diagnosis. We offered clinical evaluations at Head Start locations for children screening positive on any parent or teacher MCHAT-R/F, SCQ, or DCI. Decisions about ASD status were based on results of clinical evaluation (ASD or non-ASD) or on low-risk results on both the M-CHAT-R/F and SCQ (presumptive non-ASD), as described below. ASD status was determined on the basis of the child’s medical history, behavioral observations, and ADOS-2, which informed the clinical judgment of the experienced, licensed evaluator who was research reliable on the ADOS-2 and blind to screening results. Those completing the evaluation received a$50 gift card. Within 2 to 3 weeks of the research appointment, a full evaluation report with recommendations for follow-up intervention and/or further testing was provided regardless of ASD status. Children who did not meet ASD criteria, regardless of other diagnoses, were assigned a status of non-ASD.

To reduce risk of false-positives, we took a conservative approach to assigning ASD status and excluded from analysis children who screened positive on multiple screening tools but whose parents declined clinical evaluation (n = 13).

Children who qualified for evaluation by screening positive on the DCI or any teacher-completed screening tool, but who screened negative on the parent-completed MCHAT-R/F and SCQ, were assigned presumptive non-ASD status if their parents declined clinical evaluation (n = 43). This strategy is more data-driven than other studies in which researchers assigned ASD status on the basis of presence or absence of diagnosis in a health chart26  or parent-reported ASD diagnosis38,39  and follows other validation procedures in which multiple methods were used to reduce risk of false-negatives.29  This resulted in a sample of 624 children for data analysis.

We assessed the performance of the DCI in discriminating between children with ASD and without ASD using the same methods described in our previous DCI validation study.15  Total DCI score and subscores for speech, play, social engagement, and behavior were compared between groups by using the Mann–Whitney U test. Given the skewed distribution of scores, we used nonparametric tests. Using logistic regression, we performed item analysis to examine discriminant ability of each item. We estimated the odds of scoring at-risk on each item for children with ASD compared with children without ASD.

We performed a receiver operator characteristic (ROC) AUC analysis to examine discriminant ability of the DCI in distinguishing ASD from non-ASD. We estimated optimal cutoff points for the DCI using the Youden and Liu indices.4042  We report specificity and sensitivity of the DCI at the optimal cutoff points. An AUC of 0.5 indicates no discrimination above chance, and an AUC of 1.0 indicates perfect discrimination. In general, an AUC of 0.9 to 1.0 indicates excellent, AUC of 0.8 to 0.9 indicates good, AUC of 0.7 to 0.8 indicates fair, and AUC of 0.6 to 0.7 indicates poor discriminative ability. Discrimination is assumed to be useful if AUC ≥0.75.43,44

Pepe et al proposed guidelines4547  on the ROC-regression methods to account for confounders and effect modifiers. Following these guidelines, we introduced covariate adjustment by regression model in ROC analysis to account for potential confounding.46,48,49  We explored associations between all measured covariates, diagnostic status, and DCI total score. If any significant associations were observed, we adjusted estimated ROC curves for these covariates. We reported the AUC for both unadjusted and adjusted ROC curves. To determine if covariate-specific ROC analysis was needed, we examined the effect of the covariates on the estimated ROC curve.4547  For covariates identified as potential confounders, we estimated covariate-specific cutoff points.50,51  We also compared discriminative ability of the DCI among different demographic groups.

All procedures performed in this study involving human participants were in accordance with the ethical standards of the institution and/or national research committee and with the 1964 Helsinki declaration and its later amendments or comparable ethical standards.

In Table 1, we depict demographic characteristics of the sample. Average age was 49 months. Most children (66%) were Hispanic and from families in which English was not the primary language spoken at home (65%). Most parents reported high school or less education (65%). Most families (86%) were Medicaid enrolled or had no insurance.

TABLE 1

Sample Characteristics by ASD Status

ASD (n = 52)Non-ASD (n = 572)P
Child
Mean age in mo (SD) 46 (6.71) 49 (7.28) .001
Age range in mo
24–35 .40
36–47 50 38 .10
48–60 44 58 .05
Sex
Male 87 48 <.001
Race and ethnicity
White, not Hispanic .81
Black, not Hispanic 13 24 .08
Hispanic 73 66 .28
Other race, not Hispanic .16
Parent
Primary language
Not English 67 65 .80
Education level
High school or less education 67 65 .73
Insurance coverage
Medicaid 88 86 .65
No health insurance .88
Private insurance .59
Respondent
Mother 94 86 .11
ASD (n = 52)Non-ASD (n = 572)P
Child
Mean age in mo (SD) 46 (6.71) 49 (7.28) .001
Age range in mo
24–35 .40
36–47 50 38 .10
48–60 44 58 .05
Sex
Male 87 48 <.001
Race and ethnicity
White, not Hispanic .81
Black, not Hispanic 13 24 .08
Hispanic 73 66 .28
Other race, not Hispanic .16
Parent
Primary language
Not English 67 65 .80
Education level
High school or less education 67 65 .73
Insurance coverage
Medicaid 88 86 .65
No health insurance .88
Private insurance .59
Respondent
Mother 94 86 .11

Sample mean and SD for age in months, percentage in each demographic category, and P values for the test of difference in variables between ASD and non-ASD groups.

As shown in Table 1, children with ASD were younger (46 months versus 49 months) and more likely to be male than children without ASD. There were no statistically significant differences between ASD and non-ASD children in race and ethnicity, primary language at home, parent education, health insurance, or survey respondent.

Children with ASD had significantly higher scores, on average, in each category of the DCI than children without ASD (eg, total DCI score 6.67 for ASD versus 1.94 for non-ASD, P < .001) (Table 2). Discriminative performance of each DCI item was compared by ASD status (Table 3). Parents of children subsequently diagnosed with ASD were significantly more likely to endorse all but 2 of the 26 DCI items. “Says 1 word” and “responds to name” did not discriminate ASD from non-ASD.

TABLE 2

Comparison of DCI Scores by ASD Status

ASD (n = 52)Non-ASD (n = 572)Pa
Total DCI score (total 28 items) (SD) 6.67 (4.95) 1.94 (2.32) <.001
DCI, speech score (toal 4 items) (SD) 0.62 (0.97) 0.19 (0.57) <.001
DCI, play score (total 6 items) (SD) 1.73 (1.92) 0.40 (0.99) <.001
DCI, social engagement score (total 8 items) (SD) 1.23 (1.79) 0.17 (0.52) <.001
DCI, behavior score (total 8 items) (SD) 3.09 (1.99) 1.19 (1.51) <.001
ASD (n = 52)Non-ASD (n = 572)Pa
Total DCI score (total 28 items) (SD) 6.67 (4.95) 1.94 (2.32) <.001
DCI, speech score (toal 4 items) (SD) 0.62 (0.97) 0.19 (0.57) <.001
DCI, play score (total 6 items) (SD) 1.73 (1.92) 0.40 (0.99) <.001
DCI, social engagement score (total 8 items) (SD) 1.23 (1.79) 0.17 (0.52) <.001
DCI, behavior score (total 8 items) (SD) 3.09 (1.99) 1.19 (1.51) <.001
a

For Mann–Whitney U Test statistic for comparison of instrument scores among ASD and non-ASD children.

TABLE 3

Percentage of Children in Each Group Who Scored Positive for Each DCI Item

ASD (n = 52), %Non-ASD (n = 572), %Odds RatioP
Asks for help 0.15 0.03 7.24 <.001
Says 1 word 0.09 0.07 1.53 .39
Says 2 words 0.13 0.06 2.62 .003
Says 3 words 0.23 0.04 6.85 <.001
Plays peek-a-boo 0.25 0.04 8.33 <.001
Feeds toy 0.33 0.12 3.60 <.001
Imitates 0.23 0.03 11.95 <.001
Plays pretend 0.31 0.08 4.96 <.001
Plays make believe 0.25 0.06 6.62 <.001
Plays pretend with children 0.37 0.08 6.74 <.001
Smiles back 0.10 0.002 60.74 <.001
Makes eye contact 0.15 0.01 14.67 <.001
Points 0.06 0.01 4.31 .04
Shows toys 0.08 0.00 1.00
Comforts others 0.13 0.02 8.74 <.001
Shares interest 0.21 0.02 16.78 <.001
Shares toy 0.31 0.09 4.44 <.001
Plays with children 0.19 0.02 12.14 <.001
Walks on toes often 0.37 0.19 2.40 .01
Bothered by noise 0.44 0.20 3.22 <.001
Tantrums often 0.65 0.27 5.08 <.001
Avoids other children 0.35 0.09 5.40 <.001
Avoids eye contact 0.35 0.08 6.35 <.001
Spins wheels over and over 0.50 0.20 3.97 <.001
Flaps hands a lot 0.40 0.12 5.02 <.001
Responds to name 0.04 0.04 0.95 .90
ASD (n = 52), %Non-ASD (n = 572), %Odds RatioP
Asks for help 0.15 0.03 7.24 <.001
Says 1 word 0.09 0.07 1.53 .39
Says 2 words 0.13 0.06 2.62 .003
Says 3 words 0.23 0.04 6.85 <.001
Plays peek-a-boo 0.25 0.04 8.33 <.001
Feeds toy 0.33 0.12 3.60 <.001
Imitates 0.23 0.03 11.95 <.001
Plays pretend 0.31 0.08 4.96 <.001
Plays make believe 0.25 0.06 6.62 <.001
Plays pretend with children 0.37 0.08 6.74 <.001
Smiles back 0.10 0.002 60.74 <.001
Makes eye contact 0.15 0.01 14.67 <.001
Points 0.06 0.01 4.31 .04
Shows toys 0.08 0.00 1.00
Comforts others 0.13 0.02 8.74 <.001
Shares interest 0.21 0.02 16.78 <.001
Shares toy 0.31 0.09 4.44 <.001
Plays with children 0.19 0.02 12.14 <.001
Walks on toes often 0.37 0.19 2.40 .01
Bothered by noise 0.44 0.20 3.22 <.001
Tantrums often 0.65 0.27 5.08 <.001
Avoids other children 0.35 0.09 5.40 <.001
Avoids eye contact 0.35 0.08 6.35 <.001
Spins wheels over and over 0.50 0.20 3.97 <.001
Flaps hands a lot 0.40 0.12 5.02 <.001
Responds to name 0.04 0.04 0.95 .90

Stacks blocks, lines up toys removed in phase 2.

Unadjusted ROC analysis of the discriminative power of the DCI yielded an AUC of 0.84 (95% confidence interval [CI]: 0.78–0.89). In Table 4, we provide a detailed report of sensitivity and specificity of the DCI at each potential cutoff point. The optimal cutoff point across all ages, as estimated by Liu and Youden indices, was 3.5.

TABLE 4

ROC Estimation: Detailed Report of Sensitivity and Specificity

CutpointSensitivity, %Specificity, %Correctly Classified, %LR+LR−
(≥0) 100.0 0.0 8.3 —
(≥1) 98.1 32.9 38.3 1.46 0.06
(≥2) 88.5 52.5 55.5 1.86 0.22
(≥3) 80.8 73.1 73.7 3.00 0.26
(≥4) 71.2 83.4 82.4 4.28 0.35
(≥5) 57.7 86.9 84.5 4.40 0.49
(≥6) 48.1 91.1 87.5 5.39 0.57
(≥7) 44.2 94.4 90.2 7.91 0.59
(≥8) 32.7 97.4 92.0 12.47 0.69
(≥9) 26.9 98.1 92.2 14.00 0.75
(≥10) 23.1 99.1 92.8 26.40 0.78
(≥11) 19.2 99.3 92.6 27.50 0.81
(≥12) 15.4 99.3 92.3 22.00 0.85
(≥13) 11.5 99.5 92.2 22.00 0.89
(≥15) 11.5 99.7 92.3 33.00 0.89
(≥16) 9.6 99.8 92.3 55.00 0.91
(≥18) 7.7 99.8 92.2 44.00 0.92
(≥19) 1.9 100.0 91.8 — 0.98
(>19) 0.0 100.0 91.7 —
CutpointSensitivity, %Specificity, %Correctly Classified, %LR+LR−
(≥0) 100.0 0.0 8.3 —
(≥1) 98.1 32.9 38.3 1.46 0.06
(≥2) 88.5 52.5 55.5 1.86 0.22
(≥3) 80.8 73.1 73.7 3.00 0.26
(≥4) 71.2 83.4 82.4 4.28 0.35
(≥5) 57.7 86.9 84.5 4.40 0.49
(≥6) 48.1 91.1 87.5 5.39 0.57
(≥7) 44.2 94.4 90.2 7.91 0.59
(≥8) 32.7 97.4 92.0 12.47 0.69
(≥9) 26.9 98.1 92.2 14.00 0.75
(≥10) 23.1 99.1 92.8 26.40 0.78
(≥11) 19.2 99.3 92.6 27.50 0.81
(≥12) 15.4 99.3 92.3 22.00 0.85
(≥13) 11.5 99.5 92.2 22.00 0.89
(≥15) 11.5 99.7 92.3 33.00 0.89
(≥16) 9.6 99.8 92.3 55.00 0.91
(≥18) 7.7 99.8 92.2 44.00 0.92
(≥19) 1.9 100.0 91.8 — 0.98
(>19) 0.0 100.0 91.7 —

Number of observations: 624, ROC area: 0.84, SE: 0.029, 95% CI: 0.78–0.89, optimal cutoff point: 3.5. LR +, positive likelihood ratio; LR−, negative likelihood ratio; —, not applicable.

In Table 5, we report ROC estimation results adjusted for covariates, and in Fig 3, we depict the corresponding ROC curve. The estimated AUC is 0.80 (95% CI: 0.71–0.88), slightly lower than the unadjusted AUC. The significant coefficients for sex and parental education suggest that boys and children with low parental education have higher DCI scores, controlling for other covariates. Estimated cutoff points were girls: 2.5, boys: 3.5, low parent education: 3.5, and higher parent education: 2.5.

TABLE 5

Total DCI scoreCoefficientP
Child
Age −0.02 .16
Male 0.71 <.001
Race and ethnicity
Black, not Hispanic 0.78 .15
Hispanic 0.28 .56
Other race, not Hispanic 0.87 .25
Parent
Primary language
English not primary language 0.52 .13
Education Level
High school or less education 0.66 .01
Insurance coverage
Medicaid coverage 0.05 .92
No health insurance coverage 1.05 .09
Respondent
Mother −0.16 .66
Total DCI scoreCoefficientP
Child
Age −0.02 .16
Male 0.71 <.001
Race and ethnicity
Black, not Hispanic 0.78 .15
Hispanic 0.28 .56
Other race, not Hispanic 0.87 .25
Parent
Primary language
English not primary language 0.52 .13
Education Level
High school or less education 0.66 .01
Insurance coverage
Medicaid coverage 0.05 .92
No health insurance coverage 1.05 .09
Respondent
Mother −0.16 .66

Site indicators are included in the estimating equation. Reference categories are white-non-Hispanic, English primary language at home, parent with post–high school education, private insurance coverage, respondent is not mother. Observed coefficient: 0.80; bootstrap SE; 0.0422, 95% CI: 0.714–0.879.

FIGURE 3

ROC curve: ASD versus Non-ASD. Area under ROC curve = 0.80

FIGURE 3

ROC curve: ASD versus Non-ASD. Area under ROC curve = 0.80

Close modal

The DCI discriminated ASD from non-ASD better among children with private insurance (AUC = 1.00) than among those with Medicaid (AUC = 0.83, 95% CI: 0.77–0.89) or with no insurance (AUC = 0.83, 95% CI: 0.69–0.97) (Supplemental Table 6). This result should be interpreted cautiously because the sample with private insurance was small (n = 39). There were no statistically significant differences in the discriminative power of the DCI based on ethnicity, parent education, or language spoken at home.

The DCI demonstrates good ability (AUC = 0.80) to discriminate ASD from non-ASD across all age groups in this unselected sample of young children from families who were primarily Hispanic, spoke Spanish as a primary language, were insured through Medicaid or not insured, and had a maternal education of high school or below. Although the gap in diagnoses of autism has been narrowing between Black and white young children, Hispanic children are still diagnosed with ASD at a lower rate.3  The DCI holds promise for addressing disparity in age of diagnosis among Hispanic children.7  The DCI has good discriminative power for children between 36 and 47 months of age, a group for which most common autism screening tools are not validated. It is noteworthy that in this sample, the proportion identified with ASD was 8%, much higher than the estimated ASD prevalence by the Centers for Disease Control and Prevention.28

The DCI had an unadjusted AUC of 0.84. This is lower than the AUC reported in the M-CHAT-R/F validation study29  (AUC = 0.90) and higher than most AUC values reported in SCQ validation studies (ranges from 0.62 to 0.90).34  However, M-CHAT-R/F and SCQ validation studies typically report lower tool performance among low socioeconomic status, parent education, and minority race and ethnicity groups. This study was solely composed of an underserved sample, suggesting that the DCI holds promise for identifying ASD in groups in which other screening tools may be less predictive. Direct comparison of performance between the DCI, MCHAT-R/F, and SCQ is not possible in this full sample because clinical evaluation and subsequent diagnosis were not obtained for all subjects. Comparison of the 3 tools in the subsample who received clinical evaluation is beyond the focus of the current study.

It was not feasible to conduct full evaluations of every screened negative case to confirm non-ASD status. To reduce false-negatives, we required subjects to screen negative on 2 different validated ASD screening tools (MCHAT-R/F and SCQ) to be presumed non-ASD. To increase potential to find true-positives, at-risk score on any of the 3 screening tools (M-CHAT-R/F, SCQ, or DCI) resulted in offering a clinical evaluation. The design of this study provided multiple pathways for case finding, which is a more-robust strategy than in other ASD screening tool validation studies.18,21,29

All but 2 DCI items discriminated ASD from non-ASD: “says 1 word” and “responds to name.” It is surprising that “responds to name” does not discriminate because it is often considered a key indicator of social communication. Informal query with parents suggests that many endorse this item if their child ever responds to name, rather than basing their response on the child’s typical performance. Providers may want to assess this skill directly rather than relying on parent report.

The DCI performed at least as well in this unselected sample as with a high-risk sample.15  Similarities and differences between the at-risk and unselected samples are noted. In both samples, “smiles back” and “shares interest” most discriminated ASD from non-ASD. However, in the unselected sample, “imitates,” “makes eye contact,” and “plays with children” also discriminated well. “Points” was the strongest discriminator item in the high-risk sample (odds ratio = 31.7, P < .001) but not in this unselected sample (odds ratio = 4.31, P = .04). These findings underscore the importance of assessing multiple categories of behavior and social communication when screening, even when demographic characteristics are similar.

Researchers in future studies should clinically evaluate all children, regardless of screening status, a design that would allow direct comparison of the DCI to the MCHAT-R/F and SCQ. Researchers should assess parental literacy, increase sampling of children <24 months, and test DCI performance in other samples. It would also be worthwhile to identify optimal methods of administering the DCI (eg, electronically, paper, sent home, in waiting room).

Although we presented empirical cutoff scores that maximize sensitivity and specificity in our sample, selection of cutoffs in practice should be informed by clinical aim. Thus, lower cutoffs may be chosen to ensure that no children at-risk for ASD are missed, even if specificity is reduced. Different cutoffs may also be important among different demographic groups.

The DCI is intended to be a broader ASD screener than other pictorial screeners.27  As a visually based ASD screener, the DCI may be of particular value in screening for ASD among low literacy or limited English proficiency groups in the United States. It is not known how the DCI would perform outside the United States.

This study has several limitations. Although the DCI has low literacy demands, parental literacy was not assessed. Because participation was voluntary, there may be some selection bias. It is not known if any of the children in this study were related to each other or had a family member already diagnosed with ASD, which might have had implications for a child’s ASD risk status or a parent’s knowledge of ASD and screening responses. It is also possible that this sample does not fully represent all underserved young children. Although presumptive non-ASD status was assigned on the basis of low-risk screening results on both the M-CHAT-R/F and SCQ, a full clinical evaluation was not conducted for this group. Some children who screened negative on both measures may have had ASD. A larger-scale study may have yielded sufficient power to more fully determine impact of ethnicity, parent education, or language spoken at home on DCI performance.

Despite these limitations, there are important implications related to these findings. Results of initial testing of the DCI are promising. With the DCI, we offer an ASD screening tool that appears to work well among underserved families and addresses limitations of other, validated autism screeners. As such, the DCI holds potential to help reduce disparities in identification of ASD.

We thank Marilyn Lopez and Natalia Gonzalez for their involvement in data collection and review of the study design. Thank you also to Dr Rhiannon Luyster for assistance in conceptualizing the study design.

Dr Harris conceptualized and designed the study, collected data, and drafted the initial manuscript; Dr Coffield conceptualized and designed the study, supervised data collection, and drafted the initial manuscript; Dr Janvier conceptualized and designed the study, conceived of and designed the instrument, and collected data; Dr Mandell conceptualized and designed the study and reviewed data analysis; Dr Cidav conducted the data analyses and drafted the data analysis section of the manuscript; and all authors reviewed and revised the manuscript and approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Funded in part by a grant from the New Jersey Department of Health, Governor’s Council for Medical Research and Treatment of Autism, CAUT13APS025.

COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2020-033803.

Autism Diagnostic Observation Schedule-Second Edition

ASD

autism spectrum disorder

AUC

area under the curve

CI

confidence interval

DCI

Developmental Check-In

M-CHAT-R

Modified Checklist for Autism in Toddlers, Revised

M-CHAT-R/F

Modified Checklist for Autism in Toddlers, Revised with Follow-Up

MSEL

Mullen Scales of Early Learning

PPV

positive predictive value

ROC

SCQ

Social Communication Questionnaire

1
Dawson
G
,
Rogers
S
,
Munson
J
, et al
.
Randomized, controlled trial of an intervention for toddlers with autism: the Early Start Denver Model
.
Pediatrics
.
2010
;
125
(
1
).
2
Reichow
B
.
Overview of meta-analyses on early intensive behavioral intervention for young children with autism spectrum disorders
.
J Autism Dev Disord
.
2012
;
42
(
4
):
512
520
3
Christensen
DL
,
Baio
J
,
Braun
KV
, et al
.
Prevalence and characteristics of autism spectrum disorder among children aged 8 years--autism and developmental disabilities monitoring network, 11 sites, United States, 2012
.
MMWR Surveill Summ
.
2016
;
65
(
3
):
1
23
4
Durkin
MS
,
Maenner
MJ
,
Meaney
FJ
, et al
.
Socioeconomic inequality in the prevalence of autism spectrum disorder: evidence from a US cross-sectional study
.
PLoS One
.
2010
;
5
(
7
):
e11551
5
Magaña
S
,
Lopez
K
,
Aguinaga
A
,
Morton
H
.
Access to diagnosis and treatment services among Latino children with autism spectrum disorders
.
Intellect Dev Disabil
.
2013
;
51
(
3
):
141
153
6
Mandell
DS
,
Listerud
J
,
Levy
SE
,
Pinto-Martin
JA
.
Race differences in the age at diagnosis among Medicaid-eligible children with autism
.
.
2002
;
41
(
12
):
1447
1453
7
Mandell
DS
,
Wiggins
LD
,
Carpenter
LA
, et al
.
Racial/ethnic disparities in the identification of children with autism spectrum disorders
.
Am J Public Health
.
2009
;
99
(
3
):
493
498
8
Mandell
DS
,
Ittenbach
RF
,
Levy
SE
,
Pinto-Martin
JA
.
Disparities in diagnoses received prior to a diagnosis of autism spectrum disorder
.
J Autism Dev Disord
.
2007
;
37
(
9
):
1795
1802
9
Liptak
GS
,
Benzoni
LB
,
Mruzek
DW
, et al
.
Disparities in diagnosis and access to health services for children with autism: data from the National Survey of Children’s Health
.
J Dev Behav Pediatr
.
2008
;
29
(
3
):
152
160
10
Palmer
RF
,
Walker
T
,
Mandell
D
,
Bayles
B
,
Miller
CS
.
Explaining low rates of autism among Hispanic schoolchildren in Texas
.
Am J Public Health
.
2010
;
100
(
2
):
270
272
11
Tek
S
,
Landa
RJ
.
Differences in autism symptoms between minority and non-minority toddlers
.
J Autism Dev Disord
.
2012
;
42
(
9
):
1967
1973
12
Zuckerman
KE
,
Lindly
OJ
,
Reyes
NM
, et al
.
Disparities in diagnosis and treatment of autism in Latino and non-Latino white families
.
Pediatrics
.
2017
;
139
(
5
):
e20163010
13
Johnson
CP
,
Myers
SM
;
American Academy of Pediatrics Council on Children With Disabilities
.
Identification and evaluation of children with autism spectrum disorders
.
Pediatrics
.
2007
;
120
(
5
):
1183
1215
14
Zwaigenbaum
L
,
Bauman
ML
,
Fein
D
, et al
.
Early screening of autism spectrum disorder: recommendations for practice and research
.
Pediatrics
.
2015
;
136
(
suppl 1
):
S41
S59
15
Janvier
YM
,
Coffield
CN
,
Harris
JF
,
Mandell
DS
,
Cidav
Z
.
The Developmental Check-In: development and initial testing of an autism screening tool targeting young children from underserved communities
.
Autism
.
2019
;
23
(
3
):
689
698
16
Robins
DL
,
Fein
D
,
Barton
ML
,
Green
JA
.
The Modified Checklist for Autism in Toddlers: an initial study investigating the early detection of autism and pervasive developmental disorders
.
J Autism Dev Disord
.
2001
;
31
(
2
):
131
144
17
Khowaja
MK
,
Hazzard
AP
,
Robins
DL
.
Sociodemographic barriers to early detection of autism: screening and evaluation using the M-CHAT, M-CHAT-R, and follow-up
.
J Autism Dev Disord
.
2015
;
45
(
6
):
1797
1808
18
Scarpa
A
,
Reyes
NM
,
Patriquin
MA
, et al
.
The modified checklist for autism in toddlers: reliability in a diverse rural American sample
.
J Autism Dev Disord
.
2013
;
43
(
10
):
2269
2279
19
Windham
GC
,
Smith
KS
,
Rosen
N
, et al
.
Autism and developmental screening in a public, primary care setting primarily serving Hispanics: challenges and results
.
J Autism Dev Disord
.
2014
;
44
(
7
):
1621
1632
20
Guthrie
W
,
Wallis
K
,
Bennett
A
, et al
.
Accuracy of autism screening in a large pediatric network
.
Pediatrics
.
2019
;
144
(
4
):
e20183963
21
Dai
YG
,
Porto
KS
,
Skapek
M
, et al
.
Comparison of the modified checklist for Autism in toddlers, revised with follow-up (M-CHAT-R/F) positive predictive value by race [published online ahead of print March 3, 2020]
.
J Autis Dev Disord
. doi:
22
Rutter
M
,
Bailey
A
,
Lord
C
.
Social Communication Questionnaire
.
Los Angeles, CA
:
Western Psychological Services
;
2003
23
Moody
EJ
,
Reyes
N
,
Ledbetter
C
, et al
.
Screening for autism with the SRS and SCQ: variations across demographic, developmental and behavioral factors in preschool children
.
J Autism Dev Disord
.
2017
;
47
(
11
):
3550
3561
24
Rosenberg
SA
,
Moody
EJ
,
Lee
L-C
, et al
.
Influence of family demographic factors on social communication questionnaire scores
.
Autism Res
.
2018
;
11
(
5
):
695
706
25
Lejarraga
H
,
Kelmansky
D
,
Lejarraga
C
, et al
.
Validation of a questionnaire for the detection of children at risk of developmental disorders: the PRUNAPE Pre-screening Test
.
Arch Argent Pediatr
.
2013
;
111
(
6
):
476
483
26
Perera
H
,
Jeewandara
KC
,
Seneviratne
S
,
Guruge
C
.
Culturally adapted pictorial screening tool for autism spectrum disorder: a new approach
.
World J Clin Pediatr
.
2017
;
6
(
1
):
45
51
27
Ghilain
CS
,
MV
,
McBee
MT
, et al
.
Validation of the Pictorial Infant Communication Scale for preschool-aged children with autism spectrum disorder
.
Autism
.
2017
;
21
(
2
):
203
216
28
Baio
J
,
Wiggins
L
,
Christensen
DL
, et al
.
Prevalence of autism spectrum disorder among children aged 8 years-Autism and Developmental Disabilities Monitoring Network, 11 sites, United States, 2014
.
MMWR Surveill Summ
.
2018
;
67
(
6
):
1
23
29
Robins
DL
,
Casagrande
K
,
Barton
M
,
Chen
CM
,
Dumont-Mathieu
T
,
Fein
D
.
Validation of the modified checklist for Autism in toddlers, revised with follow-up (M-CHAT-R/F)
.
Pediatrics
.
2014
;
133
(
1
):
37
45
30
Snow
AV
,
Lecavalier
L
.
Sensitivity and specificity of the Modified Checklist for Autism in Toddlers and the Social Communication Questionnaire in preschoolers suspected of having pervasive developmental disorders
.
Autism
.
2008
;
12
(
6
):
627
644
31
Allen
CW
,
Silove
N
,
Williams
K
,
Hutchins
P
.
Validity of the social communication questionnaire in assessing risk of autism in preschool children with developmental problems
.
J Autism Dev Disord
.
2007
;
37
(
7
):
1272
1278
32
Corsello
CM
,
Akshoomoff
N
,
Stahmer
AC
.
Diagnosis of autism spectrum disorders in 2-year-olds: a study of community practice
.
J Child Psychol Psychiatry
.
2013
;
54
(
2
):
178
185
33
Corsello
C
,
Hus
V
,
Pickles
A
, et al
.
Between a ROC and a hard place: decision making and making decisions about using the SCQ
.
J Child Psychol Psychiatry
.
2007
;
48
(
9
):
932
940
34
Marvin
AR
,
Marvin
DJ
,
Lipkin
PH
,
Law
JK
.
Analysis of Social Communication Questionnaire (SCQ) screening for children less than age 4
.
Curr Dev Disord Rep
.
2017
;
4
(
4
):
137
144
35
Lord
C
,
Rutter
M
,
DiLavore
PC
, et al
.
Autism Diagnostic Observation Schedule, Second Edition (ADOS-2)
.
Torrence, CA
:
Western Psychological Services
;
2012
36
Mullen
EM
.
Mullen Scales of Early Learning (AGS Ed.)
.
Circle Pines, MN
:
American Guidance Service Inc
;
1995
37
Harrison
P
,
Oakland
T
.
, 2nd ed.
San Antonio, TX
:
Harcourt Assessment
;
2003
38
Smith
NJ
,
Sheldrick
C
,
Perrin
ER
.
An abbreviated screening instrument for autism spectrum disorders
.
Infant Ment Health J
.
2013
;
34
:
149
155
39
Volker
MA
,
Lopata
C
,
Smerbeck
AM
, et al
.
BASC-2 PRS profiles for students with high-functioning autism spectrum disorders
.
J Autism Dev Disord
.
2010
;
40
(
2
):
188
199
40
Fluss
R
,
Faraggi
D
,
Reiser
B
.
Estimation of the Youden Index and its associated cutoff point
.
Biom J
.
2005
;
47
(
4
):
458
472
41
Liu
X
.
Classification accuracy and cut point selection
.
Stat Med
.
2012
;
31
(
23
):
2676
2686
42
Perkins
NJ
,
Schisterman
EF
.
The inconsistency of “optimal” cutpoints obtained using two criteria based on the receiver operating characteristic curve
.
Am J Epidemiol
.
2006
;
163
(
7
):
670
675
43
Hosmer
DW
,
Lemeshow
S
.
Applied Logistic Regression
, 2nd ed.
New York, NY
:
John Wiley & Sons
;
2000
44
Fan
J
,
S
,
Worster
A
.
Understanding receiver operating characteristic (ROC) curves
.
CJEM
.
2006
;
8
(
1
):
19
20
45
Janes
H
,
Pepe
MS
.
.
Biometrika
.
2009
;
96
(
2
):
371
382
46
Janes
H
,
Pepe
MS
.
Adjusting for covariates in studies of diagnostic, screening, or prognostic markers: an old concept in a new setting
.
Am J Epidemiol
.
2008
;
168
(
1
):
89
97
47
Janes
HG
,
Longton
M
,
Pepe
MS
.
Accommodating covariates in ROC analysis
.
Stata J
.
2009
;
9
(
1
):
17
39
48
Hajian-Tilaki
K
.
Receiver operating characteristic (ROC) curve analysis for medical diagnostic test evaluation
.
Caspian J Intern Med
.
2013
;
4
(
2
):
627
635
49
Pepe
MS
.
Three approaches to regression analysis of receiver operating characteristic curves for continuous test results
.
Biometrics
.
1998
;
54
(
1
):
124
135
50
Tosteson
AN
,
Begg
CB
.
A general regression methodology for ROC curve estimation
.
Med Decis Making
.
1988
;
8
(
3
):
204
215
51
Pepe
MS
.
The Statistical Evaluation of Medical Tests for Classification and Prediction
.
New York
:
Oxford University Press
;
2003

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.