Learning From a National Quality Improvement Collaborative for Brief Resolved Unexplained Events

The Brief Resolved Unexplained Event Research and Quality Improvement Network is a collaborative of 15 pediatric tertiary care hospitals in the United States. Hospital characteristics and contextual factors were assessed via a Web-based survey. Characteristics differed based on geographic location, annual ED volume, and type of children’s hospital (free-standing vs hospital within a larger hospital; Table 1). All participating hospitals are affiliated with a medical school. Designated physician site leads (pediatric emergency medicine or hospitalist medicine) participated in bimonthly collaborative virtual meetings, managed local data collection, and led improvement teams.

Of the 15 collaborative hospitals, 2 engaged existing BRUE QI teams and 13 formed new QI teams. Teams included stakeholders from the ED and IP settings and included attending physicians, trainees, nurses, child life, social workers, neurologists, and speech or language pathologists. Starting in January 2019 (“planning period”), the collaborative convened, and site leaders began to collect and submit data. Starting in April 2019 (“QI collaborative period”), site leaders engaged local team members in the evaluation of quality measures with peer benchmarking, improvement efforts, and intervention implementation. All teams participated together in 4 mandatory webinars with the goal of identifying and mitigating barriers to improving key quality measures. During each webinar, sites reviewed and interpreted benchmarked data against the collaborative mean (examples provided in Supplemental Fig 5), shared barriers, and together identified change ideas. Collaborative data collection continued through June 2020, and site leads continued bimonthly meetings until June 2021. Additionally, the Seattle Children’s Maintenance of Certification Program provided hospitals with expert coaching on leading QI.

During the first webinar, a BRUE change package and tool kit were provided to sites containing a summary of the collaborative, aim statement, operational definitions, key driver diagrams (collaborative and site-specific examples), and QI learning resources. Suggested interventions included 4 key primary drivers for sites to use as a starting point to adapt to local needs (Fig 1). Suggested tools included electronic medical record (EMR) documentation templates, order sets, and communication scripts for nurses and physicians. Sharing of improvement ideas, tools (eg, key driver diagram), pathways, and documents (along with iterative improvements) were “crowdsourced” on a password-protected platform throughout the collaborative period.

Interventions focused on (1) education, (2) enhanced documentation, (3) clinical practice pathway development and use, (4) EMR integration, and (5) CPR training at or after discharge.

1. Education: Educational sessions with hospital staff, physicians, and trainees were used to kick off the collaborative locally and updated throughout the QI collaborative period as needed.

2. Documentation: Sites worked with their institution’s EMR teams to standardize BRUE-specific documentation templates according to CPG recommendations. Templates prompt providers to document BRUE characteristics, risk factors, family and social history, and risk classification. Plan-do-study-act cycles focused on incorporating feedback. Sites used educational sessions to share these newly developed or adapted tools.

3. BRUE clinical practice pathway and (4) EMR integration: All sites were encouraged to develop and implement a BRUE clinical practice pathway because the use of these pathways has been shown to improve and standardize care.^5,6  Pathway building engaged teams and clinicians across disciplines and supported BRUE-related education. Nine sites shared pathways on their hospitals’ webpages and 4 were able to have it EMR-integrated.

4. CPR training: Although not directly part of the collaborative aims, site teams tried to improve CPR training for caregivers of infants at or after discharge (as recommended by the AAP).

The primary aims and outcomes were based on recommendations from the AAP: (1) the proportion of lower-risk BRUE infants with recommended testing defined as no laboratory testing except for pertussis or electrocardiogram, and (2) the proportion of lower-risk BRUE infants admitted to the IP setting. The denominator was defined as all infants <1 year of age meeting lower-risk BRUE criteria, as defined in the AAP CPG. All measures were stratified by discharge status from the ED versus IP admission. Patients were excluded if they had an explanation for the event after the initial ED evaluation. This was defined as a “definite” or “probable” diagnosis on the basis of the ED documentation. A definite diagnosis was defined as only 1 discharge diagnosis mentioned in the discharge documentation. A probable diagnosis was defined when 1 cause, among other causes listed, was mentioned as the most likely in the discharge documentation. These exclusion criteria were applied to all primary and secondary outcome and process measures.

The secondary aims and processes were (1) the proportion of infants with the correct use of the AAP BRUE criteria in which the denominator was defined as all infants meeting BRUE criteria (irrespective of correct diagnosis), and (2) the proportion of infants with the correct use of the risk classification (lower- vs higher-risk) in which the denominator was defined as all infants meeting BRUE criteria who were correctly diagnosed. Findings were stratified by discharge status from the ED versus IP admission when patient volumes allowed helping sites identify and mitigate barriers to improvement based on clinical setting.

Two additional process measures evaluated the improvement in documentation quality. These included documentation of the presence or absence of 3 of 4 BRUE characteristics and 5 of 7 BRUE risk factors. We did not require the documentation of all 4 characteristics and 7 risk factors because the collaborative considered “partial” documentation to be an indicator of reasonable compliance, a proxy for provider intent, and a starting point for assessing and designing interventions to further improve documentation. The denominator was defined as all infants meeting BRUE criteria. Balancing measures to monitor the unintended consequences of improvement interventions included 7- and 30-day readmissions to the ED and IP settings and mortality.

The Pediatric Health Information System was used to identify infants from 11 children’s hospitals. The 4 non-Pediatric Health Information System data hospitals performed identical queries of their institutional EMR to identify cases. The method for identifying BRUE patients is described in previous publications.^7,8  Briefly, infants with possible BRUE diagnoses were initially identified by using the International Classification of Diseases, Tenth Revision diagnosis codes using weighted convenience sampling^7,8  of 4 groups of discharge codes: ALTE/BRUE (R68.13), codes indicating a BRUE characteristic such as “altered consciousness” or “apnea,” codes indicating serious conditions such as seizures or child abuse, and codes indicating less serious discharge codes, such as gastroesophageal reflux. Infants with codes indicating transfer from another hospital or with extreme prematurity were excluded.

We identified 3 time periods: (1) pre-BRUE CPG, (2) post-BRUE CPG, and (3) QI collaborative. The pre-BRUE CPG period (October 2015 to June 2016) included the time before publication of the BRUE CPG. The post-BRUE period (July 2016 to September 2018) included the time after the publication of the BRUE CPG and 6 months before starting the collaborative. These data were used for the collaborative and sites to understand baseline performance and design interventions accordingly. Sites did not review medical records leading up to the start of the collaborative (October 2018 to March 2019) to focus limited resources and site leader time on planning and intervention development (planning period). The QI collaborative began in April 2019 and finished in June 2020. A final meeting occurred in November 2020 to reflect on the impact of improvement efforts and learning during the collaborative period and consider ways to sustain improvements. The time interval was chosen to allow the collaborative sufficient time to collect and display summary data.

QI data were aggregated on a quarterly basis. Outcomes for each site were displayed by using run charts benchmarked against the collaborative’s performance (data not shown). We used statistical process control charts (p-charts) for all collaborative level charts and some site-specific charts with monthly or bimonthly data intervals because of larger patient numbers. For sites with lower patient volumes, we used run charts and grouped patients by 5. Center lines represent the overall average proportion or mean in control charts and the median in run charts. A pre-BRUE CPG center line was calculated for primary outcome measures only (all other measures were contingent on the BRUE definition). The post-BRUE CPG period center line was used as the baseline for the QI collaborative. Shifts in center lines occur when a minimum of 6 or more consecutive data points is observed above or below the center line for run charts and 8 or more for control charts. Standard special cause rules were applied to the interpretation of control charts.³  QI Macros© was used for data analysis and the display of graphs.

Demographic characteristics were compared between 3 time periods (pre-BRUE CPG, post-BRUE CPG, QI collaborative) using χ-squared tests for categorical variables and a Kruskal-Wallis test for age (continuous); P values are reported, and an α of .05 is used. Race and ethnicity were included to evaluate for disparities over time.

Institutional review board approval was obtained from each site.

Table 1 reveals participating hospital characteristics and QI interventions. Sex, race and ethnicity, and gestational age proportions were stable over time (Table 2). A total of 1756 infants met the final BRUE criteria (lower- and higher-risk combined) over the 5-year period; patients with a definite or probable diagnosis after initial presentation to the ED were excluded. Of these, 219 presented during the pre-BRUE CPG period, 894 during the post-BRUE CPG period, and 643 during the QI collaborative period. The proportion of patients with government insurance and infants aged <60 days decreased, and the median age increased.

Comparing the pre-BRUE to the post-BRUE CPG period, recommended testing adherence for lower-risk BRUE infants in the ED revealed special cause with a center-line shift from 56% to 64% within 14 months of CPG publication on the p-chart (Fig 2A). Comparing the post-BRUE CPG to the QI collaborative period, there was no change in adherence to recommended testing (Fig 2A). Comparing the pre-BRUE to the post-BRUE CPG period, lower-risk infant hospitalization rates decreased from 49% to 27% rapidly after CPG publication (Fig 2B). At the site level, 2 hospitals showed additional improvement, but none of the hospitals met a special cause for either primary measure because of low numbers of lower-risk infants (not shown).

To evaluate the implementation of the new BRUE definition and risk stratifications recommended in the CPG, we evaluated the baseline data stratified by discharge status from the ED or by admission to the IP setting. When compared with the baseline post-CPG period, the proportion of infants who received the correct use of diagnostic BRUE criteria remained unchanged during the QI collaborative period (ED 65%, IP 63%; Figs 3A & 3B); one site with improvement in this measure is shown in Supplemental Fig 5A. The correct BRUE risk classification in the ED remained unchanged (ED 26%; Fig 3C). However, special cause was met for correct risk classification in the ED for the subset of infants hospitalized in the IP setting with a center-line shift from 26% to 49% (goal 50%; Fig 3D). Additionally, correct risk classification in the IP setting for hospitalized patients increased from 15% to 33% (goal 30%; Fig 3E). Only 1 hospital demonstrated special cause improvement in this measure (not shown).

Comparing the baseline post-BRUE CPG to the collaborative period, the proportion of infants having at least 3 of 4 BRUE characteristics in the ED documentation remained unchanged (ED 67% IP 63%; Fig 4A & 4B). However, 4 hospitals demonstrated special cause improvement in the use of the 3 of 4 BRUE characteristics in the ED or IP setting (Supplemental Fig 5B). The proportion of infants with at least 5 of 7 risk factors in ED documentation for the subset of infants hospitalized in the IP setting, improved with a center-line shift from 84% to 91% after the collaborative’s first quarter (Fig 4D). Two hospitals met a special cause for this measure with a shift in center lines (Supplemental Figs 5C and 5D).

Across all 3 time periods, mean hospital readmissions to the ED and IP settings remained stable. During the pre-BRUE CPG, post-BRUE CPG, and collaborative periods, respectively, 5 patients (2.3%), 18 patients (2.0%), and 14 patients (2.2%) were readmitted within 7 days, and 13 patients (5.9%), 41 patients (4.6%), and 28 patients (4.4%) were readmitted within 30 days. There were no deaths during any of the initial presentations to the ED or IP setting. One patient had cardiopulmonary failure at home after discharge and died after readmission to the ICU. An autopsy did not reveal a cause of death.

A 15-hospital national BRUE Research and QI Collaborative demonstrated that CPG adherence and BRUE-related outcomes and processes can be enhanced through QI work. The collaborative’s goal of further reducing hospitalization rates and recommended testing in lower-risk infants was not achieved despite the use of high-reliability interventions, such as EMR-integrated clinical pathways, note templates, and order sets. Nonetheless, the collaborative was able to improve adherence to the AAP recommendations for classifying risk and considering BRUE characteristics and risk factors. Many sites were also able to make notable improvements.

Using detailed chart review, this study expands on findings from 2 recent studies using administrative data that revealed the strong impact of the publication of the AAP BRUE CPG on decreasing testing and hospitalization.^2,9  The stark improvement with publication likely indicates a wide adoption of the CPG recommendations and a more specific definition by providers. The lack of continued improvement may indicate a “ceiling effect,” possibly due to limits in the risk tolerance for families and providers when it comes to making decisions around testing and admission. For example, in the ED, caregivers and providers may rely on testing to help address diagnostic uncertainty and facilitate discharge from the ED. Whereas, in the IP setting, caregivers and providers may have different risk tolerance and are more likely to avoid unnecessary testing. Future improvement work should explore the risk tolerance of families and providers in the ED and IP setting through a qualitative approach and the use of shared decision-making tools to understand the risks and benefits of testing and hospitalization.¹⁰  Additionally, another reason for the limited improvement is the perceived inaccuracy in the AAP risk prediction and exclusion of higher-risk patients from this work. Recent studies reveal that the vast majority of BRUE patients presenting to the ED are “higher-risk” but are, in fact, not at an elevated risk of a serious underlying diagnosis.^7,8  It is possible that physicians perceived that the risk categorization proposed by the AAP was inaccurate and stopped applying it clinically, leading to stagnation in improvement in the secondary measures. The incorporation of higher-risk patients into the CPG recommendations and more refined risk prediction models are needed.^7,8  In addition, although we attempted to address the barrier to improved documentation, a component of many of our improvement measures, sites likely reached their QI capacity to improve overall provider documentation, which is typically an organizational-wide challenge.

Although the collaborative was unable to demonstrate improvement for the primary measures beyond what was already accomplished with CPG publication alone, several hospitals showed improvement across a number of collaborative goals. Many contextual factors likely contributed to their success, such as QI and EMR infrastructure. For instance, 3 larger sites (>100 000 ED visits annually) demonstrated the early adoption of the BRUE CPG before the start of the QI collaborative, as well as early standardization or BRUE QI work at a site level. All of these early adoption hospitals used institutional EMR and data support for clinical practice pathways and order set integration into their EMRs. As a result, secondary process measures for these hospitals improved early during the post-BRUE CPG period. Although QI coaching and peer learning were used to try to support hospitals with less QI capacity, several hospitals were unable to make further advances.^4,11  Like with many QI collaboratives, success partially depends on the resources and QI capacity to implement high-reliability interventions, such as EMR-integrated pathways and documentation templates. In addition, EMR systems and provider workflow must align to support the just-in-time application of the recommendations. For example, in the future, an EMR could require the appropriate documentation of BRUE risk factors in the history and physical examination that could then be automated to provide a risk score and clinical recommendations for shared decision-making.¹²  This design could better support families and providers to make decisions on true, rather than perceived, risk.

There were many benefits to the QI collaborative approach. First, we used a Web-based platform to collect and manage quality data. QI teams and collaborative leaders were able to troubleshoot data quality issues in real time and assure interrater reliability. Next, each site was able to identify areas for improvement by benchmarking their data against peers. Finally, the sharing of ideas and adapted materials, particularly from the larger, higher-performing hospitals, helped the lower-performing hospitals in identifying improvement ideas. Three sites, for example, had existing BRUE clinical practice pathways and EMR note templates that served as a model for others; the implementation of note templates for the ED and IP settings was crucial for all sites because documentation compliance was directly enhanced through forced or prompted functions and linked to the improvement of secondary outcome and process measures.

Our QI work has several limitations based on the retrospective nature of data collection. First, chart reviewers relied on the documentation and interpretation of medical records, which may have contained omissions or errors. Second, QI teams received quarterly run charts and might have had better success with more time-sensitive opportunities. Third, we do not know the data trends in the 6-month gap between the baseline period and the start of the collaborative and it is possible that they were higher or lower, although they had been stable for a year previously and we are not aware of any reason for them to have changed. Fourth, the age of the study population increased proportionally during the collaborative period. We speculate that this was because sites improved the application of the AAP’s more specific definition of a BRUE that excludes a large number of young infants that were likely previously classified as an ALTE or BRUE (eg, feeding difficulties or gastroesophageal reflux). Fifth, new research reveals that most patients do not meet lower-risk criteria (86% of patients were considered higher-risk).^7,8  Low numbers of lower-risk patients limited the ability to fully assess the impact and generalize findings for smaller hospitals. However, the documentation of BRUE characteristics and risk factors improved for several of these hospitals. Sixth, new research reveals that many of the recommendations should be applied to higher-risk patients because they are not at an increased risk of worse outcomes. Future work should evaluate recommendations for all patients, regardless of higher-risk status. Seventh, caregiver representation on QI teams could have helped identify and mitigate barriers to QI. Caregivers should be included in future improvement efforts. Next, although race and ethnicity data were collected, the impact on CPG adherence was not studied. The BRUE characteristic “color change” for different skin tones needs to be examined critically. Next, although there was no discernable impact on QI engagement, the second half of our QI collaborative period coincided with the onset of the coronavirus disease 2019 pandemic, creating unprecedented challenges for clinical care. Finally, our collaborative exists of tertiary care centers only, stressing the need for a more inclusive approach that represents community hospital settings in future BRUE-related research.

In conclusion, a large QI collaborative improved AAP recommendations for BRUE definition, risk classification, and documentation. Despite the use of high-reliability interventions, it failed to make further gains after CPG publication on hospitalizations and CPG testing adherence, indicating that, perhaps, a ceiling effect may have been reached. Future QI work on the “de-implementation” of unnecessary hospitalizations and testing, particularly in patients currently classified by the AAP as higher-risk, would benefit from caregiver perspectives and the use of shared decision-making tools.

Riley Hospital for Children, Indiana University Health (Indianapolis, Indiana): Roxanna Lefort, MD, MPH; Seattle Children’s and School of Medicine, University of Washington (Seattle, Washington): Ron L. Kaplan, MD; Baylor College of Medicine and Section of Hospital Medicine, Texas Children’s Hospital (Houston, Texas): Adam Cohen, MD, Hannah C. Neubauer, MD, Teena Hadvani, MD; School of Medicine, University of Utah (Salt Lake City, Utah): Bruce E. Herman, MD, Joshua L. Bonkowsky, MD, PhD, Caleb Porter, MD; Ann and Robert H. Lurie Children’s Hospital of Chicago (Chicago, Illinois): Kathryn Westphal, MD, Yiannis Katsogridakis, MD; Nicklaus Children’s Hospital (Miami, Florida): Melissa Clemente, MD, Kathleen Murphy, DO; Carilion Clinic (Roanoke, Virginia): Lisa Uherick; Boston Children’s Hospital (Boston, Massachusetts): Beth D. Harper, MD, Atima C. Delaney, MD; Rainbow Babies and Children’s Hospital: Allayne Stevens, MD; University of Utah and Primary Children’s Hospital (Salt Lake City, Utah): Victoria Wilkins, MD, MPH; Children’s Hospital of Philadelphia (Philadelphia, Pennsylvania): Manoj Mittal, MD; Children’s Mercy Hospital (Kansas City, Kansas): Nirav Shastri, MD; State University of New York Downstate Health Sciences University and New York Health and Hospitals/Kings County (Brooklyn, New York): Risa Bochner, MD.

AAP

American Academy of Pediatrics

ALTE

apparent life-threatening event

BRUE

brief resolved unexplained event

CPG

clinical practice guideline

CPR

cardiopulmonary resuscitation

ED

emergency department

EMR

electronic medical record

IP

inpatient

QI

quality improvement