Development and Temporal Validation of a Machine Learning Model to Predict Clinical Deterioration Free

This single center retrospective study was performed at a 190-bed children’s hospital-within-a-hospital at a quaternary, academic healthcare system in the southeastern United States. The study was approved by our Institutional Review Board, which waived the requirement for informed consent for the use of identifiable data (Pro00102839). This study followed the Transparent Reporting of a multivariable prediction model for Individual Prognosis or Diagnosis reporting guideline for prognostic studies.¹⁸ 

Data from pediatric inpatient encounters were extracted from the EHR. Pediatric encounters were defined as all patients younger than 18 years of age as well as patients younger than 25 years of age who were admitted to an inpatient pediatric hospital service rather than an adult service, which commonly occurs for young adults still cared for primarily by pediatric subspecialists. Patients were excluded if their admission was limited to the labor and delivery or neonatal wards. We did not explicitly exclude patients with Do Not Attempt Resuscitation orders as mortality occurs almost exclusively in the ICU for these patients at our institution. The development cohort included all encounters between October 1, 2014 and August 31, 2018, which was divided into a train-test split with 90% of data used for training and 10% for testing, each containing mutually exclusive patients. Encounters between January 1, 2019 and December 31, 2019 were used for temporal validation to allow for assessment of model performance on a distinct cohort of patients from the train-test cohort.

A deterioration event was defined as an unplanned transfer to the ICU or inpatient mortality. We performed a sensitivity analysis using the outcome of critical deterioration, defined as use of either mechanical ventilation or vasoactive medication or death within 12 hours of ICU transfer.¹⁹  Admissions to the ICU from the emergency department or operating room and direct transfers from labor and delivery to the neonatal ICU were not considered deterioration events. All deaths occurring between the time of hospital admission and 72 hours after time of discharge were considered inpatient mortality events. Encounters could experience more than 1 transfer to the ICU. Patients could be involved in multiple inpatient encounters during the study timeframe if they had multiple admissions. No clustering was performed at the patient level, although prior ICU admission was a model input.

A total of 227 elements were selected for inclusion in the model development process based on a combination of expert opinion, literature review, and univariate analysis identifying data elements associated with deterioration events (See Supplemental Information for full details). We did not include race and ethnicity as model inputs, but assessed outcomes and model performance based on race and ethnicity given previous work has demonstrated disparate patient outcomes based on the social construct of race.²⁰  These were defined in the EHR according to each individual’s parent or guardian or self-report.

Data elements used in model development included demographic information, comorbidities, prior inpatient encounters, vital signs, laboratory results, flowsheet measurements, orders, and medication administrations during the encounter. For vital signs and laboratory measurements, 6 features were used: the observed value, a flag indicating a new measurement event, change in value since the last hour’s measurement, and 24-hour rolling mean, minimum, and maximum values. We then trained 2 different machine learning model algorithms on the development cohort using the Python scikit-learn library: light-gradient boosting machine (LGBM) and random forest (RF).²¹  These models differ slightly in methods of model development and training, but both allow for interpretation of feature importance in the trained model, which provides insight into the patient factors that contribute most to model output. Random forest models build many independent classification trees with the final result representing an aggregation of the individual observations, whereas gradient boosted machine models build classification trees sequentially, with each tree learning from previous iterations. This difference in development can affect the model's ability to fit the initial dataset as well as model performance on classifying new datasets.

Data are summarized with counts and percentages for categorical variables and median and interquartile values for continues variables. Statistical significance was assessed with Wilcoxon rank sum test and χ-squared statistics. We considered P < .05 to be significant. We did not adjust for multiple comparisons. We evaluated the hourly performance of the machine learning models as well as I-PEWS to predict the primary outcome of unplanned ICU transfer within 24 hours or inpatient mortality within 48 hours for both the internal validation and 2019 temporal validation cohorts. We used the area under the receiver operating characteristic curve (AUROC) to evaluate overall ability to differentiate between true positive and true negative events. However, as AUROC is not affected by overall event incidence, we also used the area under the precision recall curve (AUPRC) to evaluate overall precision (positive predictive value) of the model, which balances model sensitivity with its false positive rate, an important consideration given the overall low incidence of patient deterioration. We calculated the 95% confidence interval for AUROC and AUPRC using bootstrapping (10 000 iterations with replacement).²²  We further calculated the positive predictive value (PPV), number needed to evaluate (NNE), and lead time before an event at different sensitivity thresholds. Lead-time was determined as the earliest consecutive hour before deterioration that the model would flag positive. We performed a sensitivity analysis of model and PEWS performance on detecting only critical deterioration events.¹⁹ 

The development cohort included 17 630 encounters across 10 388 patients. The median age at admission was 6.6 years (interquartile values: 1.6–13.6) (Table 1). Fifty-four percent were male. Among this cohort, 864 (8.3%) unique patients experienced 1330 unplanned ICU transfer events over the course of 1049 (5.8%) unique inpatient encounters. A total of 312 events (23%) occurred in patients who had previously deteriorated during the same admission. Deterioration occurred a median of 101.1 hours (22.2–535.6) after admission. One hundred nine (1.0%) patients died while admitted to the hospital, 27 of whom did not experience an unplanned ICU transfer during the encounter. Patients who deteriorated were younger (3.2 years [0.6–11.2] vs 6.8 years [1.6–13.6], P < .001) and experienced a longer hospital stay (14.2 days [5.9–39.1] vs 3.1 days [1.8–5.3], P < .001). They additionally were more likely to have been admitted nonelectively (P < .001) and more likely to be nonwhite (P = .017).

A total of 2 370 688 hours were represented in the training dataset, 28 302 (1.2%) of which fell within the prediction window for the combined deterioration outcome. As assessed by AUROC, the LGBM model (AUROC 0.847, 95% confidence interval [0.840–0.854]) outperformed the RF model (0.814 [0.806–0.822]) and I-PEWS (0.690 [0.687–0.693]) in the internal validation cohort (Fig 1). The AUPRC was also higher for the LGBM model (0.082 [0.076–0.090]) compared with the RF model (0.067 [0.061–0.075]) and I-PEWS (0.066 [0.063–0.069]). I-PEWS had high positive predictive value (PPV) at thresholds with low sensitivity and sharply declined in PPV with increasing sensitivity. LGBM maintained higher PPV across sensitivity values.

The 25 most important features in the 2 models are compared in Supplemental Fig 4. Vital sign-based features were most commonly represented (48% of combined top features across the 2 models), with features based on comorbidities (24%), laboratory values (16%), and demographics (10%) also common. The value for the 24-hours minimum pulse oximetry saturation was the most important feature for the RF model and second most important for the LGBM model. Hour of encounter was the most important feature for the LGBM model, which corresponded to the peak in both overall deterioration events and rate of deterioration events within the first 12 hours of admission with notable decline after 24 hours (Supplemental Fig 5). Comorbidities of respiratory failure and cardiac dysrhythmias were prominent features in both models, indicating patients with prior history of these were at higher risk for deterioration.

The 2019 temporal validation cohort included 4534 encounters from 3265 unique patients (Table 2). Fifty-seven events (17%) occurred in patients who had previously deteriorated during the same admission. Overall demographics were similar to the training cohort. Patients who deteriorated were similarly younger (3.0 years [0.5–11.1] vs 6.1 years [1.2–13.3], P < .001), and more likely to be nonwhite (P = .002) and admitted nonelectively (P < .001) than those without events. Again, the LGBM model outperformed other methods as measured by AUROC (0.785 [0.780–0.791]) compared with RF (0.743 [0.738–0.749]), and I-PEWS (0.708 [0.701–0.715]) (Fig 2). LGBM model AUROC performance remained high across racial groups (Supplemental Table 4). The AUPRC for I-PEWS was significantly higher than for LGBM (I-PEWS: 0.068 [0.063–0.073]; LGBM: 0.046 [0.043–0.048]) (Fig 2). Although the LGBM model identified patients earlier compared with I-PEWS (median lead-time before deterioration 11 h^2,19  vs 3 h^1,11 ; P = .004), it conferred lower positive prediction value (6% vs 29%) at a sensitivity corresponding to the I-PEWS threshold for activating a rapid response (9% sensitivity) (Table 3). This lower positive predictive value would correspond to an additional 13 patient evaluations required for each true deterioration event.

In sensitivity analysis evaluating performance for predicting critical deterioration, the LGBM model outperformed I-PEWS by AUROC for both internal validation (0.808 [0.792–0.823] vs 0.716 [0.710–0.723]) and 2019 temporal validation cohorts (0.827 [0.816–0.838] vs 0.753 [0.739–0.767]) as well as lead time (median 17 h^2,11  vs 2 h^1,9 ). However, it had lower AUPRC (0.017 [0.013–0.021] vs 0.02 [0.17–0.025]) and positive predictive value (2% vs 6%) in the 2019 temporal validation cohort (Supplemental Figs 6 and 7, Supplemental Table 7).

We have developed a machine learning model that demonstrated increased AUROC and greater lead-time to predict pediatric clinical deterioration compared with our current standard of care, I-PEWS; however, it also has notable limitations with lower positive predictive value and associated higher number needed to evaluate. Our model aimed to generate hourly predictions of clinical deterioration through incorporating numerous patient features from the EHR, including medication administration records as well as trends in vital sign and label value data. Although such a complex model has potential for improving predictions of deterioration, further work will be needed to enhance the model before it can be a clinically effective tool.

Previous machine learning models for pediatric patients have demonstrated the ability to predict ICU transfer with the fixed time of within 24 hours of admission or within a shorter time horizon of 6 to 8 hours before the event.^14,23–25  More recently, models trained on just vital signs, as well as vital signs combined with laboratory values, have showed improved performance in sensitivity, specificity, and positive predictive value compared with their respective institutional early warning score.^15,16  Compared with these more recent models, we explicitly evaluated all deterioration events during an admission, rather than the first such event. Notably, 22% of deterioration events were by patients with prior deterioration, indicating this is a substantial cohort of patients to evaluate. Additionally, our model involved increased complexity. We included medication administration information, demographic, and comorbidity data. We also incorporated 24 hours trends for vital sign and laboratory values rather than single values. For example, the 24-hour peak temperature and respiration rate and 24-hour lowest platelet count were all among the 10 most important features for the LGBM model. Previous studies have shown the benefits of these additional features in model development.^{10,13,26–28}  Finally, we trained our model on the timeline of 24 hours before an event. Reflecting this, our model predicted deterioration a median of 11 hours before an event and 17 hours before a critical event, potentially providing a clinically meaningful interval to allow for therapy to improve patient trajectory. However, this increased model sensitivity may come with the tradeoff of decreased positive predictive value.

The broader aim of constructing an hourly model is to integrate the technology in a hospital setting and use real-time data on admitted patients to provide hourly predictions of future clinical deterioration. Accordingly, effective early warning systems must be adequately sensitive to detect health status before deterioration while maintaining an adequate specificity to minimize additional workflow caused by evaluation of false positive patients who are not at risk for deterioration. Prior implementation of an early warning score for adult patients did not improve outcomes, likely because of low positive predictive value and alarm fatigue among frontline staff.²⁹  Although our developed LGBM model outperformed I-PEWS by several metrics, it demonstrated an inferior positive predictive value at the currently used sensitivity levels of I-PEWS. This translated to 13 additional patient evaluations to detect 1 deterioration. The time and cost of these additional patient assessments, as well as potentially associated alarm fatigue, was not evaluated but may be substantial. Before model implementation, we will need to ensure an adequate balance of sensitivity, lead-time before an event, and positive predicative value. Additionally, machine learning models must balance number of features with model performance. Simpler models with fewer variables may be more generalizable and easier to implement across health systems.³⁰  The performance of our model may not yet justify its additional complexity.

The standard for comparison in this study was I-PEWS (Supplemental Fig 3). At our institution, I-PEWS is measured as part of the nursing assessment at least every 4 hours. In the current workflow, if a patient scores a 2 in any category or has a total score of 3, the primary team and charge nurse are notified, and a bedside evaluation is typically requested. If a patient scores a 3 in any category or has a total score of 4 or higher, then a pediatric rapid response team is activated to assess the patient. This is similar to the score and escalation of care algorithm used by the (C-CHEWS), which has previously been validated to demonstrate both higher discrimination and longer early warning time performance over the Brighton PEWS.³¹  However, neither I-PEWS nor CHEWS were designed to make predictions of clinical deterioration as far in advance as 24 or 48 hours. The median time of an elevated CHEWS score (≥3) before critical deterioration was 11.1 hours, and for a critical CHEWS score (≥5), the time horizon was only 3.8 hours. Similarly, the median time for I-PEWS was 3 hours in our 2019 temporal validation dataset and only 2 hours before a critical deterioration.³¹  These short warning periods may limit the opportunity for intervention and reflect the need for improved methods of detecting deterioration.

Our study identified that incidence of clinical deterioration differed based on patient race, raising concern that this social construct may affect patient outcomes. This discrepancy may potentially be because of differences in prehospital care, initial patient triage, or care provided in the hospital. Prior studies have identified patient race and ethnicity may impact the prehospital care children receive after out-of-hospital cardiac arrest as well as the length of stay for common pediatric admissions.^20,32  Before employing a machine learning based model, it is critical to ensure it improves patient equity rather than preserving disparities.³³  Our LGBM model had high AUROC across racial groups in our 2019 temporal validation cohort, but we will need to ensure continued high performance when we employ any model prospectively.

This study has several limitations. The model would not accurately capture scenarios in which clinical concern or elevated I-PEWS score triggers a patient evaluation and leads to a necessary intervention but not ultimately to ICU transfer. These events would be labeled “false positives” based on model interpretation even though the patient may have received appropriate timely care. Conversely, in additional to objective measures, I-PEWS also incorporates nursing and parental concern, which are both subjective measures that could trigger rapid response evaluation and ICU transfer even though the patient may not have experienced a true deterioration event. We attempted to control for potentially unnecessary ICU transfers with the critical deterioration outcome, but real time, prospective evaluation will be needed to fully assess model performance.

Model performance in the 2019 temporal validation cohort was notably decreased compared with the internal validation cohort. This may possibly be because of small changes in patient population or clinical practice over time, and this difference highlights the need for prospective validation on an independent cohort before model implementation. Further, our model was trained and evaluated at a single center and additional optimization and external validation would be required for use at a different institution. Additionally, since these models were trained on a general cohort of pediatric patients, they may not be optimally calibrated for specific patient subgroups. For example, decompensation can be difficult to recognize in children with congenital heart disease because of baseline abnormalities in underlying physiology.^17,34,35  Future work could include a similar modeling approach trained in these specific populations to provide patient-specific, real-time predictions of outcomes for these children. Similarly, as our institution’s pediatric rapid response team workflow is targeted for patients on the pediatric intermediate and stepdown units, 4 major patient populations were excluded from the development cohort: healthy neonates, pediatric obstetric patients, and ICU or emergency department patients. The models in this study would not be indicated for use within these populations, each of which would likely require different risk stratification models because of their unique physiology.

In this study, using a wide range of clinical data available from the EHR, we developed a machine learning model that exhibits improved performance by AUROC and lead-time before event in predicting clinical deterioration in pediatric inpatients within 24 to 48 hours compared with our current institutional standard of care, I-PEWS. Although the model had a lower positive predictive value and corresponding higher number needed to evaluate, it demonstrates the powerful potential to improve patient detection and lead-time before a deterioration event, potentially mitigating morbidity and mortality in pediatric patients. In the future, machine learning models should be studied to be efficiently implemented into clinical workflow to improve the outcomes of hospitalized children.

The authors would like to acknowledge Duke Pediatrics Research Scholars.