OBJECTIVES

To evaluate International Classification of Diseases, 10th Revision (ICD-10) coding strategies for the identification of patients with a brief resolved unexplained event (BRUE).

METHODS

Multicenter retrospective cohort study, including patients aged <1 year with an emergency department (ED) visit between October 1, 2015, and September 30, 2018, and an ICD-10 code for the following: (1) BRUE; (2) characteristics of BRUE; (3) serious underlying diagnoses presenting as a BRUE; and (4) nonserious diagnoses presenting as a BRUE. Sixteen algorithms were developed by using various combinations of these 4 groups of ICD-10 codes. Manual chart review was used to assess the performance of these ICD-10 algorithms for the identification of (1) patients presenting to an ED who met the American Academy of Pediatrics clinical definition for a BRUE and (2) the subset of these patients discharged from the ED or hospital without an explanation for the BRUE.

RESULTS

Of 4512 records reviewed, 1646 (36.5%) of these patients met the American Academy of Pediatrics criteria for BRUE on ED presentation, 1016 (61.7%) were hospitalized, and 959 (58.3%) had no explanation on discharge. Among ED discharges, the BRUE ICD-10 code alone was optimal for case ascertainment (sensitivity: 89.8% to 92.8%; positive predictive value: 51.7% to 72.0%). For hospitalized patients, ICD-10 codes related to the clinical characteristics of BRUE are preferred (specificity 93.2%, positive predictive value 32.7% to 46.3%).

CONCLUSIONS

The BRUE ICD-10 code and/or the diagnostic codes for the characteristics of BRUE are recommended, but the choice between approaches depends on the investigative purpose and the specific BRUE population and setting of interest.

The term “brief resolved unexplained event” (BRUE) was introduced by the American Academy of Pediatrics (AAP) in 2016 to replace the older term “apparent life-threatening event” (ALTE).1  It is not yet known whether the use of the more precise definition, risk-based classification, and evidence-based management recommendations are leading to better clinical outcomes.1,2  Accurate identification of patients with a BRUE from administrative databases is a crucial first step for improving the care of these infants and young children.

The use of administrative data that rely on the International Classification of Diseases, 10th Revision (ICD-10) to identify patient populations is a convenient and cost-effective approach to quality improvement and research, particularly when a large multicenter sample is needed.2,3  However, discharge codes used for billing purposes may not accurately reflect the clinical situation.4  Consequently, systematic underclassification and overclassification are concerns for patients initially diagnosed with a BRUE by history and physical examination in the emergency department (ED). For example, among patients subsequently identified as having a serious underlying cause for the event, the BRUE ICD-10 code may not be applied on discharge. This error would result in a systematic underclassification of serious explanations for a BRUE, such as abusive head injury or seizure requiring antiepileptic therapy. Alternatively, the BRUE ICD-10 code could be applied to patients presenting with a concerning event who do not meet AAP criteria because of symptoms such as fever. This type of error could lead to systematic overclassification.

Accurate and validated methods for the identification of BRUE patient populations using ICD-10 codes and administrative data could facilitate quality improvement efforts focused on improving the evaluation and management of BRUE and research efforts evaluating clinical outcomes after discharge. Thus, the primary objective of this study was to evaluate the performance of various ICD-10 coding strategies for the identification of (1) patients presenting to an ED who met the AAP clinical definition for a BRUE and (2) the subset of these patients discharged from the ED or hospital without an explanation for the BRUE.

This multicenter retrospective cohort study included 11 tertiary care freestanding children’s hospitals in the United States participating in the BRUE Research and Quality Improvement Network. The institutional review board at each hospital approved the study.

Data sources for this study included the Pediatric Health Information System (PHIS; Children’s Hospital Association, Lenexa, KS) database and medical record review. The PHIS database is an administrative database containing demographic, clinical, and billing data for 50 tertiary care children’s hospitals, with data quality ensured by both the participating institutions and the Children’s Hospital Association.5  As reported previously, trained investigators from each site reviewed the institutional medical records of patients identified through the PHIS database using a standardized data collection tool to confirm eligibility and determine event characteristics, risk factors, and final diagnoses.4  Investigator training included practice cases, regular site-specific meetings, and biweekly multicenter conference calls over a 3-month period until an interrater reliability of 0.80 was achieved. Data were managed by using REDCap, a centralized secure Web-based data collection system.6 

The PHIS database was queried for the index visit for patients aged <1 year with an ED visit between October 1, 2015, and September 30, 2018, and an ICD-10 code grouped into the following cohorts: (1) BRUE (R6813); (2) characteristics associated with BRUE (23 codes; eg, apnea [R0681]); (3) serious underlying diagnoses presenting as a BRUE (445 codes; eg, seizures [G4089]); and (4) nonserious diagnoses presenting as a BRUE (145 codes; eg, gastroesophageal reflux disease [K219]).4  A complete listing of all diagnostic codes by cohort is provided in Supplemental Table 3. The study time period was selected to (1) provide 3 years of data since the introduction of ICD-10 and (2) account for the addition of BRUE to the ICD-10 code for ALTE (R6813).7,8  We excluded patients with diagnostic codes indicating extreme prematurity (P0701, P0702, P0721, P0722, P0723, P0724, P0725) or interhospital transfer. As reported previously, a weighted convenience sample of each aforementioned cohort was performed (40%, 36%, 12%, and 12% for cohorts 1–4, respectively) to maximize sensitivity and specificity on the basis of the perceived probability of a BRUE.4  On the basis of previous ALTE and BRUE literature, cohort 1 (“BRUE”) and cohort 2 (“BRUE characteristics”) were oversampled, given their specificity and the higher likelihood of qualifying patients.4 

On the basis of medical record review, a BRUE was defined according to the AAP guidelines as a sudden, unexplained, brief, and resolved episode occurring in an infant aged <1 year characterized by cyanosis or pallor; absent, decreased, or irregular breathing; marked change in tone; and/or altered level of responsiveness.1  Per AAP guidelines, patients with (1) recent objective symptoms, such as fever, (2) abnormal vital signs or physical examination, or (3) a comorbid contributory condition, such as complex congenital heart disease, were not considered to have a BRUE and were excluded from the study. We assessed the performance of 16 ICD-10 algorithms for the identification of (1) patients presenting to the ED with a BRUE and (2) the subset of these patients who did not have a probable or definite explanation on ED or hospital discharge. We examined these 2 populations because the former population is most appropriate for monitoring ED practice patterns and the latter is most appropriate for monitoring clinical outcomes. To ensure a comprehensive approach, the 16 algorithms included ICD-10 codes from the 4 aforementioned cohorts either alone or in combination, with the codes in either the first or any diagnostic position (Supplemental Table 4). The 16 algorithms were developed on the basis of a previously reported approach to examining coding accuracy9,10  and identification strategies used in the ALTE and BRUE literature.2,1117 

To characterize the study population, we used proportions for categorical variables and median and interquartile range for continuous variables. We calculated the sensitivity, specificity, positive predictive value (PPV), and negative predictive value (NPV) of each identification algorithm for (1) all patients presenting to the ED with a BRUE and (2) the subset of those patients without an explanation on discharge. The subset of patients presenting to an ED who met the AAP clinical definition for a BRUE but were discharged from the ED or hospital with an explanation for the event were excluded from the analysis. We stratified the analyses by disposition as either “discharged from the ED” or “hospitalized and discharged” because these 2 groups vary clinically. We graphically present the sensitivity and 1-specificity by disposition groups to demonstrate the relationships across algorithms. We did not include cohorts 3 (“serious underlying diagnoses”) and 4 (“nonserious diagnoses”) in the figures because they did not alter the performance characteristics of the algorithms. We also examined variability in coding across hospitals using algorithm F (BRUE code or any one of the 23 codes for the characteristics associated with BRUE in any diagnostic position). All statistical analyses were performed by using SAS version 9.4 (SAS Institute, Inc, Cary, NC) and P values <.05 were considered statistically significant.

We reviewed 4512 patient records from 227c760 identified encounters (Fig 1). Overall, 1646 (36.5%) records met the AAP clinical criteria for a BRUE on ED presentation, and, of these, 1016 (61.7%) were hospitalized (Table 1). Of the patients presenting with a BRUE, 959 (58.3%) were discharged from either the ED or the hospital without an explanatory diagnosis.

FIGURE 1

Sampling strategy to identify BRUE patients by using ICD-10 codes. a Patients were excluded from detailed chart review if (1) the AAP BRUE criteria were not met, (2) they had a potentially explanatory comorbid condition, (3) they were transferred from another ED, or (4) the presenting concern was distinct from BRUE.

FIGURE 1

Sampling strategy to identify BRUE patients by using ICD-10 codes. a Patients were excluded from detailed chart review if (1) the AAP BRUE criteria were not met, (2) they had a potentially explanatory comorbid condition, (3) they were transferred from another ED, or (4) the presenting concern was distinct from BRUE.

Close modal
TABLE 1

Characteristics of the BRUE Study Population

BRUEa, N=1646
Patient type, n (%)  
 Inpatient 1016 (61.7) 
 ED 630 (38.3) 
Patient characteristics  
 Median age, d (IQR) 49 (19, 109) 
 Sex, female, n (%) 857 (52.1) 
 Race and ethnicity, n (%)  
  Non-Hispanic white 671 (40.8) 
  Non-Hispanic Black 415 (25.2) 
  Hispanic 409 (24.8) 
  Other 151 (9.2) 
 Primary language, English, n (%) 1350 (83.1) 
 Government Insurance, n (%) 1002 (60.9) 
Patient risk factors, n (%)  
 Gestational age  
  Term 1061 (64.5) 
  35–37 wk 274 (16.6) 
  30–34 wk 122 (7.4) 
  <30 wk 28 (1.7) 
  Unknown 161 (9.8) 
 Premature and corrected ≤45 wk 305 (20.5) 
 Age <60 d 928 (56.4) 
 History of similar event 580 (35.2) 
 History of multiple events or event clusters 459 (27.9) 
 Event duration <1 min 844 (51.3) 
 Family history concerning for serious condition 145 (8.8) 
 Social history concerning for abuse 61 (3.7) 
 Abnormal medical history 467 (28.4) 
BRUE characteristics, n (%)  
 Color change 902 (54.8) 
 Breathing absent, decreased, or irregular 1192 (72.4) 
 Tone change 712 (43.3) 
 Altered responsiveness 573 (34.8) 
Outcomes, n (%)  
 Recurrent event or BRUE in ED or hospital 194 (11.8) 
 Readmission related to index BRUE before 1 y of age 138 (8.4) 
 Serious underlying condition 68 (4.1) 
BRUEa, N=1646
Patient type, n (%)  
 Inpatient 1016 (61.7) 
 ED 630 (38.3) 
Patient characteristics  
 Median age, d (IQR) 49 (19, 109) 
 Sex, female, n (%) 857 (52.1) 
 Race and ethnicity, n (%)  
  Non-Hispanic white 671 (40.8) 
  Non-Hispanic Black 415 (25.2) 
  Hispanic 409 (24.8) 
  Other 151 (9.2) 
 Primary language, English, n (%) 1350 (83.1) 
 Government Insurance, n (%) 1002 (60.9) 
Patient risk factors, n (%)  
 Gestational age  
  Term 1061 (64.5) 
  35–37 wk 274 (16.6) 
  30–34 wk 122 (7.4) 
  <30 wk 28 (1.7) 
  Unknown 161 (9.8) 
 Premature and corrected ≤45 wk 305 (20.5) 
 Age <60 d 928 (56.4) 
 History of similar event 580 (35.2) 
 History of multiple events or event clusters 459 (27.9) 
 Event duration <1 min 844 (51.3) 
 Family history concerning for serious condition 145 (8.8) 
 Social history concerning for abuse 61 (3.7) 
 Abnormal medical history 467 (28.4) 
BRUE characteristics, n (%)  
 Color change 902 (54.8) 
 Breathing absent, decreased, or irregular 1192 (72.4) 
 Tone change 712 (43.3) 
 Altered responsiveness 573 (34.8) 
Outcomes, n (%)  
 Recurrent event or BRUE in ED or hospital 194 (11.8) 
 Readmission related to index BRUE before 1 y of age 138 (8.4) 
 Serious underlying condition 68 (4.1) 

IQR, interquartile range.

a

Patients presenting with a qualifying event that met the AAP BRUE criteria.

The majority of patients with a BRUE on ED presentation were identified with the ICD-10 codes for BRUE (cohort 1, n = 1196; 66.1%) and the BRUE characteristics (cohort 2, n = 431; 26.9%). The proportion of patients without an explanation for the BRUE on discharge was highest in the BRUE cohort (cohort 1, n = 746; 41.2%). Few patients with an ICD-10 code for a serious diagnosis (cohort 3, n = 8; 1.5%) or a nonserious diagnosis (cohort 4, n = 11; 2.0%) met AAP clinical criteria for a BRUE on ED presentation. Only 3 of 8 patients with an ICD-10 code for a serious diagnosis (cohort 3) and none of 11 with an ICD-10 code for a nonserious diagnosis (cohort 4) were discharged without an etiologic diagnosis for the BRUE presentation (Fig 1).

The characteristics of each algorithm by diagnosis and disposition are provided in Table 2. Among patients discharged from the hospital from the ED, the sensitivity of the various algorithms for the identification of a BRUE on ED presentation ranged from 1.4% to 100% (NPV: 0% to 98.9%), whereas the specificity ranged from 0% to 92.8% (PPV: 3.5% to 72.0%). For patients discharged from the ED without an explanation for the BRUE, sensitivity ranged from 0.9% to 100% (NPV: 0% to 100%) and specificity from 0% to 89.8% (PPV: 1.2% to 51.7%). For hospitalized patients presenting to the ED with a BRUE, sensitivity ranged from 2.6% to 100% (NPV: 0% to 97.8%) and specificity ranged from 0% to 93.2% (PPV: 9.7% to 69.7%). For hospitalized patients discharged with a BRUE without an explanation, sensitivity ranged from 1.8% to 100% (NPV: 0% to 100%) and specificity ranged from 0% to 93.2% (PPV 4.1% to 47.8%).

TABLE 2

Sensitivity, Specificity, and Predictive Value of 16 ICD-10 Identification Algorithms for BRUE

ED OnlyInpatient
BRUE Without Explanation on DischargeBRUE on ED PresentationBRUE Without Explanation on DischargeBRUE on ED Presentation
Algorithm LabelaSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPV
48.1 89.8 51.7 88.4 37.1 92.8 72.0 74.6 53.9 82.1 47.8 85.5 47.2 87.0 69.7 72.3 
69.1 80.9 45.0 92.0 57.5 86.2 67.7 80.2 82.8 61.8 39.6 92.2 82.1 72.6 65.4 86.5 
22.6 74.4 16.7 80.9 31.0 77.9 41.2 69.2 11.0 93.2 32.7 77.6 9.4 93.2 46.3 62.0 
30.4 54.8 13.2 77.6 41.4 57.0 32.6 66.0 17.0 65.4 13.0 72.3 16.7 60.8 21.2 53.6 
78.5 44.6 24.3 90.1 78.6 49.8 44.0 82.2 71.0 47.6 29.1 84.4 64 47.8 43.6 67.8 
99.4 35.7 26.0 99.6 98.9 43.2 46.6 98.7 99.8 27.2 29.3 99.8 98.8 33.4 48.4 97.8 
0.9 83.6 1.2 78.8 1.4 80.4 3.5 61.9 1.8 87.2 4.1 74.6 2.6 84.9 9.7 58.0 
2.6 77.2 2.5 77.7 3.2 73.0 5.6 60.1 13.1 61.8 9.4 70.1 13.7 55.8 16.3 50.6 
7.2 71.3 5.4 77.2 11.4 68.6 15.5 60.7 14.6 64.6 11.1 71.4 23.5 65.0 29.8 57.4 
17.2 53.8 7.8 74.1 25.2 51.4 20.6 57.8 41.5 29.7 15.1 62.6 51.1 28.4 31.1 48.0 
50.4 67.3 25.9 85.7 39.8 65.9 37.0 68.6 62.6 46.4 26.1 80.4 56.9 45 39.5 62.3 
58.7 46.1 19.8 83.1 55.4 45.5 33.8 67.1 78.7 19.7 22.9 75.4 81.5 21.1 39.5 64.4 
74.8 25.0 18.5 81.4 68.7 22.0 30.6 58.4 91.5 5.7 22.7 68.9 92.8 5.8 38.3 56.3 
100.0 0.0 18.5 0.0 100.0 0.0 33.4 0.0 100.0 0.0 23.2 0.0 100.0 0.0 38.7 0.0 
100.0 17.9 21.7 100.0 99.5 21.7 38.9 98.9 100.0 13.6 25.9 100.0 99.2 16.6 42.9 97.1 
99.7 14.4 20.9 99.6 99.5 17.5 37.7 98.7 99.8 5.2 24.2 99.1 99.7 6.4 40.2 97.2 
ED OnlyInpatient
BRUE Without Explanation on DischargeBRUE on ED PresentationBRUE Without Explanation on DischargeBRUE on ED Presentation
Algorithm LabelaSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPVSensitivitySpecificityPPVNPV
48.1 89.8 51.7 88.4 37.1 92.8 72.0 74.6 53.9 82.1 47.8 85.5 47.2 87.0 69.7 72.3 
69.1 80.9 45.0 92.0 57.5 86.2 67.7 80.2 82.8 61.8 39.6 92.2 82.1 72.6 65.4 86.5 
22.6 74.4 16.7 80.9 31.0 77.9 41.2 69.2 11.0 93.2 32.7 77.6 9.4 93.2 46.3 62.0 
30.4 54.8 13.2 77.6 41.4 57.0 32.6 66.0 17.0 65.4 13.0 72.3 16.7 60.8 21.2 53.6 
78.5 44.6 24.3 90.1 78.6 49.8 44.0 82.2 71.0 47.6 29.1 84.4 64 47.8 43.6 67.8 
99.4 35.7 26.0 99.6 98.9 43.2 46.6 98.7 99.8 27.2 29.3 99.8 98.8 33.4 48.4 97.8 
0.9 83.6 1.2 78.8 1.4 80.4 3.5 61.9 1.8 87.2 4.1 74.6 2.6 84.9 9.7 58.0 
2.6 77.2 2.5 77.7 3.2 73.0 5.6 60.1 13.1 61.8 9.4 70.1 13.7 55.8 16.3 50.6 
7.2 71.3 5.4 77.2 11.4 68.6 15.5 60.7 14.6 64.6 11.1 71.4 23.5 65.0 29.8 57.4 
17.2 53.8 7.8 74.1 25.2 51.4 20.6 57.8 41.5 29.7 15.1 62.6 51.1 28.4 31.1 48.0 
50.4 67.3 25.9 85.7 39.8 65.9 37.0 68.6 62.6 46.4 26.1 80.4 56.9 45 39.5 62.3 
58.7 46.1 19.8 83.1 55.4 45.5 33.8 67.1 78.7 19.7 22.9 75.4 81.5 21.1 39.5 64.4 
74.8 25.0 18.5 81.4 68.7 22.0 30.6 58.4 91.5 5.7 22.7 68.9 92.8 5.8 38.3 56.3 
100.0 0.0 18.5 0.0 100.0 0.0 33.4 0.0 100.0 0.0 23.2 0.0 100.0 0.0 38.7 0.0 
100.0 17.9 21.7 100.0 99.5 21.7 38.9 98.9 100.0 13.6 25.9 100.0 99.2 16.6 42.9 97.1 
99.7 14.4 20.9 99.6 99.5 17.5 37.7 98.7 99.8 5.2 24.2 99.1 99.7 6.4 40.2 97.2 
a

Algorithms defined in Supplemental Table 4.

Among the 6 algorithms that used only the ICD-10 code for BRUE or the codes for the BRUE characteristics, the combination of codes of BRUE and its characteristics in any diagnostic position (algorithm F) had a sensitivity of 98.9% (NPV: 98.7%) for patients with a BRUE on ED presentation and 99.4% (NPV: 99.6%) for patients with a BRUE discharged from the ED without an explanation (Table 2). This combination (BRUE code plus codes for the characteristics of BRUE) had similar sensitivities among hospitalized patients for both BRUE on ED presentation (98.8%; NPV: 97.8%) and BRUE without an explanation on discharge (99.8%; NPV: 99.8%) (Fig 2). Conversely, use of the BRUE code alone in the first diagnostic position (algorithm A) had better specificity for BRUE on ED presentation (92.8%, PPV: 72.0%) and BRUE without an explanation (89.8%; PPV: 51.7%) discharged from the ED. Among hospitalized patients, a code for the characteristics of BRUE in the first diagnostic position (algorithm C) had the best specificity for BRUE on ED presentation (93.2%; PPV: 46.3%) and BRUE without an explanation on discharge (93.2%; PPV: 32.7%). The combination of codes of BRUE and its characteristics in any diagnostic position (algorithm F) had the highest sensitivity across the ED and inpatient settings for both BRUE on ED presentation and BRUE without an explanation on discharge, although the PPV across institutions revealed wide variability (Fig 3).

FIGURE 2

Sensitivity and specificity of 6 ICD-10 identification algorithms for BRUE on ED presentation BRUE without an explanation on discharge.

FIGURE 2

Sensitivity and specificity of 6 ICD-10 identification algorithms for BRUE on ED presentation BRUE without an explanation on discharge.

Close modal
FIGURE 3

Variability in the PPV of the ICD-10 identification algorithm F across sites for BRUE on ED presentation and BRUE without an explanation on discharge. ICD-10 algorithm F indicates BRUE code in any position or code for features of BRUE in any position.

FIGURE 3

Variability in the PPV of the ICD-10 identification algorithm F across sites for BRUE on ED presentation and BRUE without an explanation on discharge. ICD-10 algorithm F indicates BRUE code in any position or code for features of BRUE in any position.

Close modal

In this large multicenter study, we examined the performance of 16 different ICD-10 coding algorithms for the identification of BRUE among infants presenting to the ED. Our results reveal that administrative data can be used to identify patients with BRUE in both the ED and inpatient settings. Among ED discharges, the BRUE ICD-10 code alone was optimal for case ascertainment, whereas, for hospitalized patients, ICD-10 codes related to the clinical characteristics of BRUE are preferred. However, because each approach has a trade-off, the intended purpose and population of interest must be carefully considered because this will inform which coding algorithm is ultimately preferable for case ascertainment.

Case ascertainment is challenging in BRUE because patients may present for evaluation of an event that is not explained at the time of discharge. Alternatively, a reason for the event might be identified in the ED or during hospitalization and patients may be assigned a new diagnostic code that explains the etiology of the event. Diagnostic evaluation and management approaches will also differ between the ED and inpatient settings, particularly when additional insight is gained through inpatient evaluation and/or observation. Awareness of these complexities is essential for definition of a study population. Furthermore, the PPV across all algorithms examined was poor, highlighting a role for medical record review for diagnostic confirmation.9,10 

Here, we demonstrate the trade-offs of different strategies. For clinical outcomes research evaluating rare outcomes that cause a BRUE-like presentation, the most comprehensive algorithm that uses the BRUE code, BRUE characteristics, and associated serious and nonserious diagnoses had the best sensitivity, but with the trade-off of poor specificity. This approach likely captures all patients presenting to the ED with a BRUE but would require considerable resources for medical record review to identify the rare case in which a serious (cohort 3) and nonserious (cohort 4) condition is associated with a BRUE presentation. Alternatively, reliance on the BRUE code and BRUE characteristic codes (algorithm F), offers comparable sensitivity at the expense of missing a small number of patients with a serious explanation for the BRUE-like event.

For quality improvement activities focusing on clinical management, algorithms using the BRUE code alone (algorithm A) or the codes for the features of BRUE (algorithm C) offer reasonable accuracy. For instance, for quality improvement initiatives aimed at improving adherence to AAP recommendations for the evaluation and management of patients discharged from the hospital from the ED after a lower-risk BRUE,1  investigators can optimize specificity and PPV through use of a principal diagnosis of BRUE for case ascertainment. In contrast, in the inpatient setting in which hospitalized patients are more likely to be higher-risk, for whom evidence-based recommendations currently do not exist,18  investigators interested in maximizing specificity may use codes for the characteristics of BRUE to identify patients presenting with a BRUE who may or may not retain this diagnosis on hospital discharge. When resources allow, however, and optimal sensitivity is preferred, investigators may elect to use the combination of codes for BRUE and its characteristics (algorithm F). Until a unique ICD-10 code for BRUE with qualifiers for risk stratification is available, use of the examined algorithms will enable more accurate case ascertainment and facilitate comparative effectiveness research aimed at developing optimal diagnostic and management approaches for patients meeting BRUE criteria.

The results of this study should be considered in light of several limitations. First, this study was conducted at tertiary care institutions where the billing and coding practices may differ from other pediatric and community-based hospitals, limiting the generalizability of our findings to these settings. Second, administrative coding practices varied markedly across institutions and can vary over time, potentially contributing to an underestimation of patients diagnosed with a BRUE. This was minimized by assessing systematically developed cohorts of billing codes associated with the BRUE code (R6813). However, it is still possible that we were not able to capture all the serious underlying problems that initially presented with a BRUE. Additional variability in coding may have occurred because the ICD-10 code for ALTE is synonymous with BRUE,8  and coding for BRUE may have improved over the course of study period.

We found that, depending on the location of service, different approaches are needed to accurately identify BRUE patients using administrative billing codes. Investigators should consider tailoring the ICD-10 coding strategy to the population and setting of interest to balance accuracy and chart review resources.

We thank all the members of the BRUE Research and Quality Improvement Network for their invaluable contributions.

FUNDING: No external funding.

Dr DeLaroche conceptualized and designed the study, designed the collection instruments, collected data, conducted the initial analyses, drafted the initial manuscript, and reviewed and revised the manuscript; Drs Mittal, Neuman, Stephans, Wilkins, Cohen, Kaplan, Shastri, and Tieder conceptualized and designed the study, designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Ms Sullivan and Dr Hall conceptualized the study, coordinated and supervised data collection, managed data and data quality, critically reviewed the manuscript, and performed the analysis; the Brief Resolved Unexplained Event Research and Quality Improvement Network authors designed the study, evaluated the data collection instruments, collected data, reviewed the analysis, and critically reviewed the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

1.
Tieder
JS
,
Bonkowsky
JL
,
Etzel
RA
, et al
;
SUBCOMMITTEE ON APPARENT LIFE-THREATENING EVENTS
.
Brief resolved unexplained events (formerly apparent life-threatening events) and evaluation of lower-risk infants
.
Pediatrics
.
2016
;
137
(
5
):
e20160590
2.
DeLaroche
AM
,
Haddad
R
,
Farooqi
A
,
Sapién
RE
,
Tieder
JS
.
Outcome prediction of higher-risk brief resolved unexplained events
.
Hosp Pediatr
.
2020
;
10
(
4
):
303
310
3.
Kaplan
RM
,
Chambers
DA
,
Glasgow
RE
.
Big data and large sample size: a cautionary note on the potential for bias
.
Clin Transl Sci
.
2014
;
7
(
4
):
342
346
4.
Tieder
JS
,
Sullivan
E
,
Stephans
A
, et al
.
Risk factors and outcomes after a brief resolved unexplained event: a multicenter study
.
Pediatrics
.
2020
;
147
(
6
):
2020036095
5.
Mongelluzzo
J
,
Mohamad
Z
,
Ten Have
TR
,
Shah
SS
.
Corticosteroids and mortality in children with bacterial meningitis
.
JAMA
.
2008
;
299
(
17
):
2048
2055
6.
Harris
PA
,
Taylor
R
,
Thielke
R
,
Payne
J
,
Gonzalez
N
,
Conde
JG
.
Research Electronic Data Capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support
.
J Biomed Inform
.
2009
;
42
(
2
):
377
381
7.
US Centers for Disease Control and Prevention
;
National Center for Health Statistics
.
International Classification of Diseases (ICD-10-CM/PCS) transition—background: 2017
.
Available at: https://www.cdc.gov/nchs/icd/icd10cm_pcs_background.htm. Accessed January 5, 2021
8.
AAP Division of Healthcare Finance
.
New ICD-10-CM codes take effect on Oct. 1
.
Available at: https://www.aappublications.org/news/2017/08/30/Coding083017. Accessed January 5, 2021
9.
Williams
DJ
,
Shah
SS
,
Myers
A
, et al
.
Identifying pediatric community-acquired pneumonia hospitalizations: Accuracy of administrative billing codes
.
JAMA Pediatr
.
2013
;
167
(
9
):
851
858
10.
Tieder
JS
,
Hall
M
,
Auger
KA
, et al
.
Accuracy of administrative billing codes to detect urinary tract infection hospitalizations
.
Pediatrics
.
2011
;
128
(
2
):
323
330
11.
Tieder
JS
,
Cowan
CA
,
Garrison
MM
,
Christakis
DA
.
Variation in inpatient resource utilization and management of apparent life-threatening events
.
J Pediatr
.
2008
;
152
(
5
):
629
635
,
635.e1
635.e2
12.
Bonkowsky
JL
,
Guenther
E
,
Filloux
FM
,
Srivastava
R
.
Death, child abuse, and adverse neurological outcome of infants after an apparent life-threatening event
.
Pediatrics
.
2008
;
122
(
1
):
125
131
13.
Percelay
J
,
Linzer
J
 Sr
.
Apparent life-threatening event (ALTE) ICD-9-CM code proposal
.
Available at: https://www.cdc.gov/nchs/ppt/icd9/att6percelaysep08.pdf. Accessed March 3, 2017
14.
Meyer
JS
,
Stensland
EG
,
Murzycki
J
,
Gulen
CR
,
Evindar
A
,
Cardoso
MZ
.
Retrospective application of BRUE criteria to patients presenting with ALTE
.
Hosp Pediatr
.
2018
;
8
(
12
):
740
745
15.
Colombo
M
,
Katz
ES
,
Bosco
A
,
Melzi
ML
,
Nosetti
L
.
Brief resolved unexplained events: Retrospective validation of diagnostic criteria and risk stratification
.
Pediatr Pulmonol
.
2019
;
54
(
1
):
61
65
16.
Ramgopal
S
,
Noorbakhsh
KA
,
Callaway
CW
,
Wilson
PM
,
Pitetti
RD
.
Changes in the management of children with brief resolved unexplained events (BRUEs)
.
Pediatrics
.
2019
;
144
(
4
):
e20190375
17.
Duncan
DR
,
Growdon
AS
,
Liu
E
, et al
.
The impact of the American Academy of Pediatrics brief resolved unexplained event guidelines on gastrointestinal testing and prescribing practices
.
J Pediatr
.
2019
;
211
:
112
119.e4
18.
Merritt
JL
 II
,
Quinonez
RA
,
Bonkowsky
JL
, et al
.
A framework for evaluation of the higher-risk infant after a brief resolved unexplained event
.
Pediatrics
.
2019
;
144
(
2
):
e20184101

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data