Association Between Rapid Response Algorithms and Clinical Outcomes of Hospitalized Children

To determine the impact of the rapid response algorithms, we conducted a retrospective cohort study using historical controls at a 289-bed freestanding children’s hospital. The efferent limb of the RRS at the study institution consists of a RRT made up of 3 clinicians, (1) a pediatric critical care medicine (PCCM) fellow, PCCM attending physician, or pediatric critical care nurse practitioner, (2) a pediatric critical care charge nurse, and (3) a respiratory therapist. The RRT can be activated by anyone with any clinical concern, including the parents of a patient. After RRT intervention, the patient is either transferred to the PICU or cardiac ICU (CICU) or remains on the floor. Patients can also be transferred from the floor to an ICU via a physician-initiated transfer, independent of RRT activation.

We defined the study population as patients admitted to the PICU as unplanned transfers from general medical or surgical floors, given this is a target population for the RRS. We did not limit inclusion criteria to just those patients who triggered a rapid response event to decrease selection bias. We included only PICU transfers and not all floor patients because of inaccessibility of floor patient data related to a change in the study institution’s electronic health record (EHR) during the study period. We defined the preimplementation period as July 1, 2017, to March 31, 2019, and the postimplementation period as April 1, 2019, to January 31, 2020. We chose April 2019 (study month 23) as the start of the implementation period because this was the first month we observed consistent and meaningful change in algorithm usage. We excluded patients transferred to the PICU from the emergency department, operating room, imaging suite, postanesthesia care unit, or outside hospital.

A PCCM attending physician, pediatric nurse practitioner, and 2 pediatric registered nurses developed the rapid response algorithms. The team designed the algorithms to help guide the RRT’s and floor team’s management steps when initially evaluating a deteriorating patient. Simulation sessions with nurses, physicians, and respiratory therapists were used to refine the algorithms. The team organized the algorithms into categories on the basis of 8 pediatric clinical syndromes: respiratory distress, sepsis and shock, overdose, anaphylaxis, altered mental status, seizure, neurologic deficit, and suspected heart disease (Fig 1; Supplemental Fig 4).

The algorithms were uploaded to the hospital’s EHR and linked to clinical syndrome-specific EHR order-sets on January 22, 2019. Physical copies were placed on code carts and in charge nurses’ offices. Before implementation, health care provider education was provided via virtual and in-person presentations, descriptive e-mails, and computer-based training sessions. After implementation, nurses and resident physicians participated in additional computer-based training and simulation sessions to reinforce concepts and provide hands-on practice with the algorithms and order-sets.

We defined the primary outcome as the rate of critical deterioration events (CDEs) per 1000 non-ICU patient days. CDEs are associated with increased in-hospital mortality and increased hospitalization costs and have been used to evaluate the effectiveness of an RRS.^20–22  We defined a CDE as an event resulting in noninvasive positive pressure ventilation (NPPV), intubation, or vasopressor infusion within 12 hours of transfer to the PICU.²⁰ 

We defined secondary outcomes as the rate of death before PICU discharge, the rate of intubation and mechanical ventilation within 1 hour of PICU admission, PICU length of stay [LOS], and the rate of cardiopulmonary arrests (CPAs) before transfer to the PICU. We defined each event rate as the number of events per 1000 non-ICU patient days. CPAs included respiratory arrests, defined as emergent intubation on the floor, and cardiac arrests, defined as events with documented pulselessness or a pulse with inadequate perfusion requiring chest compressions, defibrillation, or both. Additionally, we included the rates of each component of a CDE as secondary outcomes and the percent usage of the algorithm-based order-sets as a process measure.

We used the Virtual Pediatric Systems (VPS) database for the study institution to identify unplanned PICU admissions from the general medical and surgical floors. The VPS is a Web-based PICU data repository that stores patient-level data from >100 PICUs in the United States and abroad (Virtual Pediatric Systems LLC, Los Angeles, CA).²³  From this data set, we extracted patient age, sex, illness severity (as measured by the Pediatric Risk of Mortality, Version 3, [PRISM 3] score²⁴ ), date and time of PICU admission as well as several patient outcomes (death before PICU discharge, intubation within 1 hour of PICU admission, CPA before PICU admission, and PICU LOS).

Using the unique patient encounter identifiers obtained from VPS, we queried the study institution’s electronic data warehouse and determined if NPPV, intubation, or vasopressors were initiated within 12 hours of PICU admission. We used International Classification of Diseases, 10th Revision, diagnostic codes obtained from the electronic data warehouse to determine the number of complex chronic conditions (CCCs) associated with each patient.²⁵ 

We queried the study institution’s rapid response database, using the patient identifiers from VPS, to determine if the RRT was activated before PICU transfer and whether the algorithm-based order-sets were used during each rapid response event in the postimplementation group. The study institution’s institutional review board determined this study to be exempt human subjects research per §45 CFR 46.104, category 4.

We described continuous patient characteristic data with medians and interquartile ranges (IQRs) because of nonnormal distribution and categorical patient characteristic data with frequencies and percentages. We stratified patient demographics and clinical characteristics by preimplementation and postimplementation status and compared them by using χ² and Wilcoxon rank tests where appropriate. We calculated incident rate ratios for unadjusted outcome rates and tested for significance using the mid-P exact test for person-time data.²⁶  Because of nonnormal distribution, we used the Wilcoxon rank test to compare PICU LOS in the preimplementation and postimplementation groups. For the adjusted analysis, we used a multivariable Poisson regression model with individual subject data to adjust for the covariates of age, sex, PICU admission after a rapid response, number of CCCs, PRISM 3 score, and a rapid response–PRISM 3 interaction term. Covariates were determined a priori on the basis of the clinical and biological relevance to the exposure and outcomes. We summed adjusted patient-level outcomes within each study month to create an adjusted time series data set.

We then performed an interrupted time series analysis to measure 2 possible types of change associated with implementing the rapid response algorithms: an immediate implementation effect and an outcome trajectory over time.²⁷  We fit an ordinary least squares time series model, using the study month as the time unit, with the itsa routine,²⁸  using sine and cosine terms to adjust for seasonality,²⁹  and Newey-West standard errors to adjust for autocorrelation.³⁰  We checked and adjusted the number of lags using the Cumby–Huizinga general test for autocorrelation.³¹  Because of seasonal variation in PICU admission rates, we performed a sensitivity analysis evaluating the impact of censoring 9 months immediately before and 3 months after implementation on the primary outcome. An overview of the statistical analysis can be found in Supplemental Fig 5 and Supplemental Information. We used 2-sided hypothesis tests and considered P values <.05 significant. We analyzed the data using Stata version 14 statistical software (Stata Corp, College Station, TX).

The study period consisted of 32 months and 112c429 non-ICU patient days, with 892 patients included in the study population. A total of 615 unplanned PICU transfers were included in the preimplementation group, and 277 were included in the postimplementation group (Table 1). The rate of unplanned PICU transfers revealed seasonal variation over the study time period, with higher rates in the winter months and lower rates in the summer (Fig 2A). We observed significant differences between the preimplementation and postimplementation groups in age, number of CCCs, PRISM 3 score, season of PICU transfer, and rate of PICU transfer after a rapid response (Table 1). The mean monthly usage rate of the algorithm-based order-sets in the postimplementation group, expressed as a percentage of the total number of rapid response events, was 60.2%, with consistent change in the monthly usage rate first observed in study month 23 (Fig 2B).

In the unadjusted analysis (Supplemental Table 3), we observed no significant change in the rate of CDEs per 1000 non-ICU patient days after algorithm implementation (incidence rate ratio [IRR]: 1.07; 95% confidence interval [CI] 0.88 to 1.29; P = .50). There was no significant change in the rates of NPPV within 12 hours of PICU admission, intubation within 12 hours of PICU admission, vasopressor initiation within 12 hours of PICU admission, death before PICU discharge, or CPAs before PICU admission (Supplemental Table 3). The intubation rate within 1 hour of PICU admission per 1000 non-ICU patient days was significantly lower in the postimplementation group (IRR: 0.65; 95% CI 0.42 to 0.98; P = .03), whereas PICU LOS was longer (median LOS: 2.10 days [IQR: 1.10 days to 4.52 days] versus 2.41 days [IQR: 1.26 days to 5.71 days]; P = .02). Because of low rates of occurrence and risk of overfitting, we did not perform an adjusted analysis of rates of death before PICU discharge, CPAs before PICU admission, or vasopressor initiation within 12 hours of PICU admission.

In the adjusted interrupted time series model, there was no significant change in the rate of CDEs immediately after algorithm implementation; however, there was a downward CDE rate trajectory in the postimplementation time period (Table 2; Fig 3A). The postimplementation CDE rate trajectory was also significantly less than the preimplementation trajectory (trajectory difference of −0.28 events per 1000 non-ICU patient days per month; 95% CI −0.40 to −0.16; P < .001).

The adjusted interrupted time series model results comparing the preimplementation and postimplementation rates of NPPV within 12 hours of PICU admission are shown in Table 2 and Fig 3B. Algorithm implementation was not associated with an immediate change in the NPPV rate but was associated with a downward trajectory of the NPPV rate over time. The downward postimplementation trajectory was significantly more negative than the preimplementation trajectory.

In the adjusted interrupted time series model, algorithm implementation was associated with an immediate increase in the rate of intubations within 12 hours of PICU admission (Table 2; Fig 3C). A significant downward trajectory followed this immediate increase, which was significantly more negative than the preimplementation trajectory.

Similar to the rate of intubations within 12 hours of PICU admission, the adjusted interrupted time series model revealed an immediate increase in the rate of intubations within 1 hour of PICU admission, followed by a decreasing rate over time that was significantly more negative than the preimplementation trajectory (Table 2; Fig 3D).

Censoring 9 months immediately before and 3 months after the intervention had a minor impact on the primary outcome. The change in CDE rate immediately after algorithm implementation increased from 0.91 to 1.42 events per 1000 non-ICU patient days (95% CI 0.15 to 2.68; P = .03). The difference between the preimplementation and postimplementation trajectories changed from −0.28 to −0.45 events per 1000 non-ICU patient days per month (95% CI −0.83 to −0.07; P = .02).

We performed a retrospective cohort study with historical controls to determine the impact of pediatric rapid response algorithms, used in the efferent limb of a RRS, on clinical outcomes of hospitalized children. The algorithms were designed to improve care coordination around deteriorating patients on the general medical and surgical floors by creating a shared mental model between health care team members.

Our study revealed that algorithm implementation was associated with a significant downward change in the CDE rate trajectory over time that was driven primarily by a change in the rate of NPPV. The postimplementation CDE rate trajectory was more negative than the preimplementation trajectory, with no significant change immediately after implementation. The combination of these findings led to the actual CDE rate at the end of the study period being lower than the expected rate if the preimplementation trajectory were to continue.

Immediately after algorithm implementation, there was a significant increase in intubation rates, both within 1 hour and within 12 hours of PICU admission. However, these rate trajectories decreased significantly over time, leading to intubation rates at the end of the study period that were lower than the expected rates if the preimplementation trends were to continue.

Similar to previous studies in which researchers investigated the effect of a RRS on mortality and out of ICU CPAs,^4,6,7,22  our study revealed that algorithm implementation was not associated with a significant change in PICU mortality or the number of CPAs before PICU admission but was associated with an increase in PICU LOS.

Our results are also in line with previous studies in which researchers evaluated the impact of a pediatric RRS using an interrupted time series approach.^9,22  Sharek et al⁹  used an interrupted time series analysis with adjustment for season and case mix and found that implementation of a RRS at a single institution was associated with reductions in hospital-wide mortality and code rates outside of the ICU. Bonafide et al²²  also used a time series model with adjustments for case mix, season, and hospital floor and identified that implementation of a pediatric RRS was associated with a decrease in the CDE rate. As was the case in our study, Bonafide et al²²  did not find a significant decrease in the CDE rate in an unadjusted analysis but did find a significant change when interrupted time series modeling was applied.

In contrast to these 2 previous studies, we adjusted for additional confounders, including age and sex, and used different metrics, CCCs, and PRISM 3 scores instead of case mix index, to represent the medical complexity of included patients. The intervention under study also differed because we evaluated the impact of algorithms embedded within the efferent limb of an existing RRS compared with investigating the impact of an entirely novel response system. Algorithmic processes have proven beneficial in the afferent limb of RRSs,^3,6,9,17,18  and the results of this study support the use of these processes within the efferent limb given the reduction in the CDE rate and CDE rate trajectory after algorithm implementation.

Our study has several limitations. First, this is an observational, single-center study with historical controls, and, thus, the results occurred in the setting of multiple hospital safety initiatives and potential unmeasured confounders. An interrupted time series analysis helps to account for preintervention trends that other initiatives may have impacted,^26,32  and a regression model helps to control for measured confounders; however, all hospital safety programs cannot be controlled for, and the groups may have differed in unmeasured factors that could have impacted the results. Second, we defined the study cohort as all unplanned PICU transfers from general medical and surgical floors, not just those patients who triggered a rapid response. Although defining the cohort in this way decreases the influence of selection bias, it also limits the number of patients exposed to the intervention and does not allow us to analyze the impact on patients who remained on the floor after RRT activation. Third, we did not include patients transferred to the CICU, given that CICU patient data are not stored in the VPS. Fourth, it is possible that unmeasured process changes due to education and increased awareness of ICU transfer criteria led to an initial increase in CDEs in study month 20, when the algorithms were first introduced. However, because we did not see a meaningful and consistent change in algorithm usage (our process measure) until study month 23, we included study months 20 through 22 in the preimplementation group. Finally, we did not measure health care team members’ reactions to the rapid response algorithms’ and cannot conclude that the intervention overcomes barriers to an effective RRS.

In conclusion, through the use of an interrupted time series approach, we identified that the implementation of pediatric rapid response algorithms was associated with an improvement in clinical outcomes of hospitalized children admitted to the PICU from the general floor. With this study, we add to the evidence supporting the use of clinical pathways within a RRS and offers support for expanding the role of algorithmic processes into the efferent limb. Moving forward, we plan to investigate the impact of the algorithms on general floor patients and on additional outcomes, such as emergency transfers.³³  We also hope to ascertain from health care providers how the algorithms affect barriers to a successful RRS. Overall, our results serve as a promising starting point for future research into how clinical pathways can enhance a RRS’s efferent limb and improve patient outcomes.