Leveraging EHR Data to Evaluate the Association of Late Recognition of Deterioration With Outcomes

We developed an EHR-based ET metric for rapid response system measurement across our institution. Utilization and timing data for ET interventions were extracted from the EHR by using SQL queries of our enterprise data warehouse storing Epic Clarity Database clinical data. We leveraged this EHR-derived metric to perform a retrospective cohort study of patients requiring escalation of care from a general medical or surgical ward to the PICU between January 2015 and June 2019. We used a case-control design to compare PICU transfer events meeting criteria for ET (cases) with a propensity-score-matched cohort of transfers not meeting criteria (controls). The project was deemed exempt by the Children’s Hospital of Philadelphia institutional review board per 45 CFR 46.104(d).

This study took place at an academic, free-standing children’s hospital with an estimated 500 to 560 licensed inpatient beds, inclusive of a 60- to 70-bed mixed medical-surgical PICU caring for >4000 admissions annually. Escalations to the pediatric ICU come from ∼300 medical and surgical beds divided into separate wards with dedicated nursing staff, nurse managers, and medical directors. Wards are staffed 24 hours per day by frontline clinicians, including advanced practice providers, hospitalists, and resident physicians.¹² 

Of note, the cardiac ICU (CICU) and neonatal ICU had separate escalation and response systems, which impacted transfer decisions and timing of ICU interventions. These ICUs and their unique patient populations were excluded to focus specifically on the population of patients who are escalated to the PICU. All references to the “ICU” refer to the PICU unless otherwise specified.

The hospital utilizes a multifaceted rapid response system, including a high-risk patient identification system that incorporates risk assessment using an early warning score, a medical emergency team of critical care providers (ICU physician, nurse, and respiratory therapist) that responds within 30 minutes, and a code blue team that responds immediately.^2,13  For the majority of the study period (January 2016–June 2019), the hospital used the pediatric Rothman Index¹⁴  as an early warning score and incorporated standardized escalation protocols for patients meeting threshold criteria.

The hospital had an integrated EHR system, Epic (Epic Systems Corporation, Madison, WI), for the duration of the study.

Patients were eligible for inclusion if they were hospitalized on a general medical or surgical ward and required escalation to the ICU between January 2015 and June 2019 because of clinical deterioration. For eligible patients with multiple ward-to-ICU transfer events during an admission, only the index transfer event was included. Because patient mortality was a primary outcome, patients with an active do not resuscitate (DNR) order at the time of transfer were excluded.

We identified eligible unplanned ICU transfers from the institution’s Virtual Pediatric Systems (VPS) database.¹⁵  VPS identifies ICU transfers as unplanned on the basis of the origin of transfer. Patients arriving from the operating room, emergency department, or outside the hospital were excluded. We defined emergency transfers by the presence of 1 or more interventions documented in the EHR within 1 hour before to 1 hour after ICU transfer: intubation, vasopressor initiation, or administration of ≥60 ml/kg intravenous fluids (herein referred to as “interventions”). We compared patients with ET to a propensity-score-matched cohort of patients with nonET to assess association with outcomes.

For secondary analysis evaluating the association of urgency of interventions with outcomes, we grouped all index transfers to the ICU into 3 groups on the basis of the level of urgency of interventions (intubation, vasopressor initiation, or ≥60 ml/kg intravenous fluids) provided. The emergent intervention group was defined by requiring ≥1 intervention within 1 hour of ICU transfer (the criteria for ET classification). The urgent intervention group was defined by requiring ≥1 intervention within 1 to 12 hours after ICU transfer. The remaining patients were the reference nonintervention group, defined by not requiring any interventions in the first 12 hours after transfer. We chose a 12-hour window on the basis of previous literature, using this timeframe to capture acute ICU interventions in patients with deterioration.^6,16 

We extracted patient data from the VPS database and EHR via our institutional data warehouse. We obtained patient demographic information, source of admission, hospital outcome data, and primary ICU diagnosis from VPS. We extracted intervention data (see below), and patient data not available in VPS (DNR orders, complex chronic condition indicator, and service before transfer) from the EHR (Supplemental Table 4). VPS provided the primary ICU diagnosis as well as prespecified VPS diagnosis categories on the basis of the primary diagnosis STAR code name. To facilitate matching patients with similar physiologic deterioration and escalation decisions, we grouped VPS diagnosis categories into one of the following deterioration physiology categories: circulatory, neurologic, respiratory, or other. We assigned VPS diagnosis categories to relevant deterioration categories on the basis of the category title and, if there was uncertainty, the most frequent diagnosis within that VPS diagnosis category. For example, the respiratory and respiratory and/or ear, nose, and throat diagnosis categories were grouped into the respiratory deterioration category. Similarly, the infectious category, which includes sepsis and bacteremia, was grouped into the circulatory deterioration category. VPS primary diagnosis and VPS diagnosis categories included in each deterioration physiology category are listed in Supplemental Table 5.

We extracted utilization and timing data for all emergency interventions from the EHR. Appropriate EHR data elements were identified after monitoring clinician documentation practices and subsequently validated through clinical workflow observations and a limited number of chart reviews. For example, we conducted several small group meetings with ICU nurses regarding documentation practices for vasopressor infusions and fluid administration to determine the best data elements capturing the time when these events occurred. Intubation was defined by an endotracheal tube placement documented by a respiratory therapist in the lines/drains/airway flowsheet, including the time of placement. Vasopressor initiation was defined as the first documentation of an epinephrine, norepinephrine, dopamine, dobutamine, milrinone, or phenylephrine infusion recorded within the medication administration record as administered. Intermittent bolus doses of hypotensive epinephrine (eg, administered doses of 1 mcg/kg) were not included. We defined ≥3 fluid boluses as ≥60 ml/kg fluid intake within a 2-hour window documented in the intake/output flowsheet record. Fluids included normal saline, lactated ringers, albumin, and blood products. The recorded weight in the VPS dataset, the dosing weight at the time of ICU transfer, was used for per-kilogram calculations. After finalizing data elements, we created an automated query of our enterprise data warehouse within a QlikView dashboard (Qlik Tech Inc, Montgomery, PA) that presented patient-level data on ETs and was available across the institution (Supplemental Fig 3).

For analysis of urgency of interventions, intake was evaluated in 2-hour rolling windows starting at 1 hour after transfer through 12 hours after ICU transfer and evaluated for intake ≥60 ml/kg in that window. If a patient received ≥60 ml/kg in a 2-hour window, the end of that window was used as the time of administration.

The primary outcome was in-hospital mortality. Secondary outcomes included ICU length of stay and posttransfer length of stay accounting for the competing risk of death. ICU length of stay was defined as the number of days from index ICU transfer to discharge from the ICU. Posttransfer length of stay was the number of days from index ICU transfer to discharge from the hospital.

We performed descriptive statistics using the Wilcoxon rank test for continuous variables or χ² test for categorical variables to evaluate the distribution of covariates between ET and non-ET patients in the full unmatched cohort and the propensity-score-matched cohort.

We compared patient mortality and length of stay in the cohort of ET and propensity-score-matched non-ET controls. A propensity score was generated on the basis of a priori identified confounders that were hypothesized to be associated with both ET and mortality: age category, service before transfer, presence of complex chronic condition, transfer quarter, pretransfer length of stay, transfer time of day, and the type of clinical deterioration.^11,17–20  Transfer quarter was included to account for seasonal changes in hospital census and disease prevalence. Pretransfer length of stay was categorized into <1 day, 1 to 7 days, 7 to 30 days, and >30 days to separate patients along a spectrum of new deterioration, subacute chronic illness with deterioration, and prolonged hospitalization before deterioration that could influence response to physiologic derangements on the ward.^18,21  Transfer time of day captured differences in staffing resources between daytime hours (Monday through Friday, 7:00 am–7:00 pm) and nights and weekends that impact resources for appropriately escalating patients.^22–24 

Patients with ET were matched with non-ET controls in a ratio of 1:3, nearest neighbor method, and caliper equal to 0.02. Cases and control characteristics were compared postmatch to ensure balance.

Outcomes in the propensity-score-matched cohort were modeled via conditional logistic regression for mortality. For length of stay measures, we used competing risk survival analysis with discharge as event and death as competing risk, censored at 30 days, with effect sizes reported as subdistribution hazard ratios (SHR) for intact discharge.^25,26  A competing risk model was used to account for the high rate of mortality, which could bias toward shorter length of stay due to patient death rather than discharge alive from the ICU or hospital. In a competing risk model, SHR of discharge reflects the hazard of discharge alive from the hospital, thus an SHR <1 implies a lower probability of being discharged alive, as such a longer length of stay.

We performed a secondary analysis of the entire cohort using a multivariable logistic regression model to compare levels of intervention urgency on mortality. We adjusted for age category, service before transfer, type of clinical deterioration, and pretransfer length of stay. We then calculated the average marginal effect of the level of urgency on the predicted probability of mortality with SE calculated using the δ method. This allowed us to estimate the increase in mortality risk across levels of urgency while adjusting for confounders.

Statistical analyses were performed by using SAS, version 9.4 (SAS institute) and R (R Core Team, 2014).

As a result of probing documentation practices for each intervention in our ICU, the interventions and timing of interventions identified via EHR data aligned with clinical workflows and delivered care per select chart reviews. The metric was successfully incorporated into a QlikView dashboard and integrated into the review process at monthly rapid response system committee meetings.

Between January 2015 and June 2019, 2066 index ICU transfer events met all inclusion criteria. Of these, 29 (1.4%) were excluded because of an active DNR order at the time of transfer. Of the 2037 remaining index transfers, 129 (6.3%) met the criteria for ET (Fig 1).

In the unmatched cohort, subjects in the ET group were older (P < .01) and more likely to have a complex chronic condition (P < .01). They were more likely to have transferred from the oncology service (P < .01) and had a greater incidence of neurologic or circulatory deterioration (P < .01). They also had a longer pretransfer length of stay with a higher frequency of deterioration 7 to 30 days after admission compared with non-ET (P = .02) (Table 1).

Of 129 transfers meeting ET criteria, 127 were matched to 374 non-ET controls. After matching, all standard mean differences were <0.1 (Table 1). In the propensity-score-matched cohort, patients with ET had higher in-hospital mortality compared with matched controls (13% vs 6.1%; odds ratio 2.47; 95% confidence interval [95% CI], 1.24–4.9, P = .01). They also had a longer ICU length of stay (SHR of ICU discharge 0.74; 95% CI, 0.61–0.91, P < .01); and posttransfer length of stay (SHR of hospital discharge 0.71; 95% CI, 0.56–0.90, P <.01) (Table 2).

Within 1 hour of transfer, 129 patients required an emergent intervention, 157 required an urgent intervention 1 to 12 hours after transfer, and 1751 received no intervention. The majority of interventions occurred in the first hour after transfer rather than the first hour before transfer (76%) in the emergent intervention group and in the first 1 to 6 hours after transfer in the urgent intervention group (76%) (Supplemental Fig 4). Patients in the urgent intervention group were more likely to meet the criteria because of intubation than patients in the emergent intervention group (70% vs 58% P < .01) (Table 3). The predicted probability of mortality increased as intervention urgency increased, from 4.1% for nonintervention, to 6.4% for urgent intervention, and to 10.0% for emergent intervention (Fig 2). Only the predicted probability of mortality for emergent intervention compared with nonintervention was statistically significant (P = .02).

In this observational study of hospitalized patients requiring escalation of care to the ICU, patients with transfers meeting EHR-based ET criteria had higher rates of mortality and longer length of stay compared with propensity-score-matched controls. Additionally, comparing patients across varying levels or urgency of intervention, we found a dose response in the relationship between the urgency of intervention and risk of mortality. Although it may seem self-evident that sicker patients requiring interventions will have worse outcomes, it is notable that patients with identical interventions only a few hours later had an improved risk of mortality compared with ETs, suggesting that patient transfer to the ICU earlier in the course of deterioration may contribute to improved outcomes. Finally, we found that ETs more frequently involved circulatory or neurologic deterioration compared with controls and identified new group-level differences to inform future investigation.

Our findings align with previous literature evaluating this metric through manual chart review. In seminal work by Hussain et al,¹¹  ETs were associated with an increased mortality rate (22% vs 9% in controls) and increased length of stay. Similarly, Aoki et al²⁷ demonstrated increased mortality and length of stay in a single institution in Japan. Although our analysis found a lower mortality rate in the controls and ETs, this is most likely attributable to the lack of cardiac patients escalated to the CICU in our analysis. Our work provided more robust evidence that ET is a predictor of outcomes resulting from clinical deterioration with late recognition by incorporating measures of medical complexity and type of physiologic deterioration as confounders in a propensity-score-matched analysis.

In addition to a more robust analysis, our work reveals an easily implementable method of replicating this metric using common EHR data elements. With an EHR-based metric, we could retrospectively determine our baseline ET rate to inform current improvement work. As our institution has shifted away from infrequent cardiopulmonary arrests, we have used the ET metric to measure the impact of detection, early response, and postevent debriefing programs, including a novel critical care outreach team.^6,12  Adopting a similar EHR-based metric across institutions, modified for local clinical workflows, can support generalizability and multisystem collaboratives.

The stepwise increase in mortality risk with increasing urgency of interventions further supports the hypothesis that the late recognition and escalation of care for patients with deterioration may contribute to worse outcomes. Although there is limited literature evaluating the association between timing of interventions and mortality, the distribution of interventions over the first 12 hours of ICU transfer aligns with recent work comparing ETs to clinical deterioration events (CDE) involving intubation, vasopressor initiation, or noninvasive ventilation within 12 hours.¹⁶  CDEs are more frequent than ETs and have a lower mortality rate, in part because CDEs capture patients who may have had a timely escalation to the ICU and rescue from further decline. Similarly, the urgent intervention group, incorporating a comparable timeframe to CDEs, may have had a decreased mortality risk because of safer rescue opportunities over a longer observation in the ICU. Additionally, our results align with the time-sensitive nature of other interventions such as antibiotics for septic shock and antiepileptics for status epilepticus.^28–32 

We also identified key characteristics of ETs that can guide future improvements in situational awareness and proactive recognition of deterioration. Previous work highlighted at-risk cohorts, such as patients with oncologic disease, but did not address the types of physiologic derangements that lead to deterioration.¹¹  Interestingly, the majority of ETs occurred after neurologic or circulatory deterioration rather than respiratory. Although speculative, this finding may be related to the limited tools for closely monitoring and rescuing neurologic or circulatory physiology on the floor. In contrast, providers have multiple modalities for monitoring patients with respiratory pathology and tools for temporary rescue, including noninvasive ventilation or high-flow nasal cannula to delay further deterioration. Alternatively, pediatric care systems may be less effective at detecting and responding to circulatory or neurologic changes relative to respiratory pathophysiology, which is far more prevalent in hospitalized children.³³  Our findings warrant further exploration across different institutions and highlight the need for rapid response monitoring systems that are tailored to each patient’s risk for specific physiologic changes.

Finally, we found that patients with ET tended to have a longer length of stay before deterioration. Although many factors, including preexisting medical complexity, contribute to prolonged length of stay, it could also suggest that rapid response systems are less effective at responding to deterioration in those with prolonged illness. Future interventions to improve the management of in-hospital deterioration could benefit from focused attention to the unique features of patients requiring emergent interventions.

Capturing events from the EHR is subject to misclassification errors secondary to inappropriate documentation. Typically, documentation in acute situations occurs after stabilization biasing toward the null by recording interventions later than they occurred. Coding errors, inherent to any database study, could also lead to misclassification errors in inferring deterioration categories from recorded diagnosis categories. Fortunately, VPS data entry is completed by clinically trained providers and implements quality-check measures to reduce this bias. Like previous work, we could not account for all factors that influence ICU decision making. Because it is not feasible to randomize the timing of rescue interventions, we incorporated measures of medical complexity, deterioration types, and transfer time that influence such decisions. Similarly, we lacked physiologic or early warning score data to identify the start of deterioration relative to the timing of interventions. However, by anchoring on ICU admission and finding a dose response, our analysis still suggests earlier recognition could improve outcomes. Finally, this was a single-center study of patients escalated to the PICU, limiting generalizability to other patient populations. Patients escalated to the CICU and neonatal ICU can have unique reasons for escalation and may have different indications for ET interventions which could require different data from the EHR. Previous work suggests that excluding the CICU likely underestimated the association of ETs and mortality. Automating this metric within the most widespread EHR and using common flowsheet variables should facilitate further dissemination and validation after adaptation to local workflows.³⁴ 

An EHR-based ET metric is associated with an increased risk of mortality and length of stay in hospitalized pediatric patients. Tracking this metric with routine EHR elements supports future efforts at broader dissemination and validation. The urgency of interventions was also associated with a stepwise increase in mortality risk, suggesting that reducing the late recognition of deterioration could improve outcomes. Future efforts should take advantage of identified patient-level risk factors to improve the early recognition and mitigation of clinical deterioration outside the ICU.

The authors thank the following people for their support of this work and preventing deterioration outside the ICU at the Children’s Hospital of Philadelphia: Ana R Altmann, DM, Marlana R Bevan, MSN, Lynn Boyle, MSN, Andrea Colfer, MSN, Kristin Granahan, BSN, Jennifer Jacob-Freese, BSN, Anne Marie Mcgrath, MSN, Kristin Neiswender, MSN, Sagine Simon, MSN, Kelly O’Shea, BS, Ellen Tracy, MSN, Lisa Tyler, MHA, Brianna Weeks, MSN, and Drs Geoffrey Bird, Daniela Davis, Evan Fieldston, Dan Hyman, Ron Keren, Naveen Muthu, Ursula Nawab, and Margaret Priestley.

The authors thank the staff of the CHOP Critical Care Center for Evidence and Outcomes for their efforts in abstracting and coding the CHOP VPS data used to prepare this report. VPS data were provided by Virtual Pediatric Systems, LLC.