Machine Learning Approach to Predicting Absence of Serious Bacterial Infection at PICU Admission

We included children 3-months-old to 18-years-old at our institution, from 2011 to 2020, who underwent an infectious work-up within the hospital or network of care (NOC) within 24-hours before or after arrival in the PICU. Infants <3-months-old were excluded, given their unique SBI risk profile as compared with older children. We defined infectious work-up as any bacterial, viral, or fungal culture, or PCR, CRP, PCT, or chest radiograph. Patients were excluded if a full set of vital signs was not documented between 24-hours before PICU arrival until 2-hours after PICU arrival. Our institution is a large pediatric center with >3000 annual, noncardiac PICU admissions and a regional care network. We obtained study approval through our institutional review board.

We stratified the cohort by documented administration of antibiotics during the 48-hours before PICU arrival. Antibiotics administered at non-NOC facilities or at home were not captured. Any enteral or intravenous antibiotic administration qualified a patient as being antibiotic-exposed.

We extracted data on study patients from our institution’s instances of the Epic EHR and Virtual PICU Systems (VPS, LLC; Los Angeles, CA) registry. VPS includes clinical data of PICUs from >100 international sites whereby a trained ICU nurse specialist inputs standardized quality improvement patient data from the site EHR. Given the challenge of extracting VPS-equivalent data elements directly from the EHR, inclusion of VPS-derived variables was conducted as part of a sensitivity analysis only.

Model inputs included vital sign, laboratory, and clinical EHR data available at PICU admission (Supplemental Table 4). Vitals and laboratory values were abstracted between 24-hours-before PICU arrival until 2-hours-after PICU arrival (Supplemental Figure 3) to identify trends occurring before PICU admission, while allowing for an initial evaluation of patients admitted directly to the PICU from outside institutions. Microbiologic results (part of the SBI outcome definition, Table 1) were included if obtained between 24-hours-before PICU arrival until 24-hours-after PICU arrival. For each vital sign, we calculated summary statistics for the 26-hour observation window (Supplemental Table 4). Summary statistics were then normalized based on reported pediatric distributions (Supplemental Information).^37–40 

If >1 value of a laboratory was available, we used the mean value, except for reported SBI biomarkers (WBC, ANC, CRP, and PCT) and lactate, for which maximum values were used (per prior SBI studies).^27,41  As per Lipton et al,⁴²  commonly obtained laboratory variables (those present in ≥10% of study patients) were imputed with a normal value (the middle of the normal range documented in our institution’s EHR). For less common laboratories (values present in <10% of patients) missing values were imputed using the study cohort’s median value for that laboratory (see Supplemental Information for additional imputation strategy discussion). Additionally, as absence of a laboratory test result may itself contain significant predictive value,⁴³  we created flags for all imputed values and included these indicators of imputation as variables in the models. To evaluate for potential noise or bias introduced via our imputation method, we conducted a sensitivity analysis comparing AUROC of models created with, versus without, laboratory value imputation (Supplement Information).

We used International Classification of Diseases, Ninth and Tenth Revision codes from previous encounters and validated mappings and software^44–46  to determine if a patient had a complex chronic condition. In a second sensitivity analysis, we included variables extracted from VPS: ventilatory support, foreign bodies present on PICU admission (eg, central venous catheter), patient origin (eg, emergency department), and type of admission (scheduled versus unscheduled). Additional VPS data including mortality, PRISM III score, and ICU length of stay were used for intergroup comparisons (not as model inputs). Finally, to be as practical as possible for future implementation, patients with multiple PICU admissions during the study period were not excluded, and each admission was treated as a new (and independent) encounter by the models.

The model outcome was any SBI (diagnosed via studies performed between 24-hours before and after PICU arrival) using a definition adapted from Lautz et al (Table 1, itself adapted from the CDC and National Healthcare Safety Network guidelines and Fernandez López et al).^27,47,48  To maximize generalizability, we only included structured data available within the EHR. Specifically, we did not include patient symptoms, and we adapted the pneumonia and culture-negative septic shock components (Supplemental Information).

For continuous variables, we compared medians using 2-sided Wilcoxon rank-sum tests. For binary variables, we compared groups using χ² tests. We identified 148 predictors, 77 of which required some level of imputation. The 180 final variables (predictors and laboratory [or laboratory-panel] imputation flags) were used as inputs for each model (Supplemental Tables 4 and 5). Specifically, we input these variables into random forest (RF), penalized logistic regression (PLR), and support vector machine (SVM) models (hereafter referred to as machine learning models) to predict presence versus absence of SBI at PICU admission. Separate ridge, lasso, and elastic-net PLR models and linear, radial, and polynomial SVM models were built. We relied on the more flexible, nonlinear models (such as RF and radial-SVM) to incorporate variable interactions and did not include interaction terms in the regression models. Additional details on model design, including hyperparameter tuning, are detailed in the Supplemental Information.

We randomly divided each stratum (those without documented antibiotic administration in the 48-hours before PICU admission [antibiotic-unexposed], and those with documented antibiotic administration in the 48-hours before PICU admission [antibiotic-exposed]) into training (80%) and validation (20%) sets. We created machine learning models to predict SBI-negativity at PICU arrival using the training sets and fivefold crossvalidation. We identified model SBI probability cut-points such that the negative predictive value (NPV) in the training sets would be ≥99% in antibiotic-unexposed patients and ≥95% in antibiotic-exposed patients, with the practical assumption that a clinician may prefer a stronger NPV to withhold antibiotics in a critically ill child who has not received antibiotics but feel more comfortable discontinuing antibiotics in a patient who has already received antibiotics. We then evaluated model performance using the validation sets.^49,50 

We calculated model performance metrics, including area under the receiver operator characteristic (AUROC) curve.⁵⁰  We identified the variables of highest importance from each model. We also evaluated the predictive abilities (via AUROC) of 4 SBI biomarkers (WBC, ANC, CRP, and PCT) using the subset of each stratum for whom that laboratory was available (Supplement Information).

To quantify potential effects on antibiotic reduction, we identified the SBI-negative children in the validation sets who were provided antibiotics within 24-hours after PICU arrival. We calculated the proportion each model was able to correctly identify as SBI-negative and determined the median antibiotic treatment-days per patient that could have been spared via each model. To assess the balance between model efficacy and safety, we report the number of SBI-negative children who could have been spared antibiotics, as well as the number of SBI-positive children incorrectly predicted to be SBI-negative by each model. See Supplemental Information for programming and statistical software used.

After excluding 36 of 15 110 (0.2%) PICU encounters without a full set of vitals during the 26-hour observation window, 15 074 encounters (12 170 unique children) met inclusion criteria (Supplemental Figure 4). Pneumonia, bacteremia, and UTI and pyelonephritis were the most common SBIs (Supplemental Table 6). In the 48-hours before PICU admission, 4788 (32%) patients received antibiotics (the antibiotic-exposed stratum). Of those, 2325 (49%) were SBI-positive. The remaining 2463 SBI-negative children were continued on antibiotics for a median 3.4 (interquartile range [IQR] 1.1–7.9) days per patient. Among the 10 286 patients who did not receive antibiotics in the 48-hours before PICU admission (the antibiotic-unexposed stratum), 2356 (23%) were SBI-positive. Of the remaining 7930 SBI-negative children, 3092 (39%) were started on antibiotics and continued for a median 2.6 (IQR 1–6.9) days per patient (Table 2). See Supplemental Table 7 for additional patient characteristics and outcomes.

Among the antibiotic-unexposed stratum, RF had the highest AUROC for SBI status discrimination (0.76, 95% confidence interval [95% CI] 0.74–0.79) with NPV of 0.98 (Fig 1, Supplemental Figure 5 and Supplemental Tables 8 and 9). Among the antibiotic-exposed stratum, the RF (AUROC 0.80, 95% CI 0.78–0.83) and radial-SVM (0.80, 0.77–0.82) models performed best with similar NPVs (0.97 and 0.96, respectively) (Table 3, Supplemental Tables 8, and 10, and Supplemental Figure 5). Key SBI predictors included a variety of EHR data types, including (1) vital sign metrics for Fio₂, Spo₂, SBP, temperature, respiratory rate, and heart rate, (2) laboratory values including lactate, urine leukocytes, and percent bands, and (3) clinical data elements such as pre-ICU length of stay, complex chronic condition status, weight percentile, and age. Top predictors varied by pre-ICU antibiotic-exposure stratum (Supplemental Tables 11 and 12, and Supplemental Figure 6). A sensitivity analysis comparing models created with, versus without, missing laboratory value imputation demonstrated similar AUROC values between each model type pair (P > .1 for all) (Supplemental Tables 13 and 14).

For each of 4 SBI biomarkers (WBC, ANC, CRP, and PCT) in both antibiotic-exposure strata, AUROC for SBI discrimination was lower than the machine learning models (though biomarkers were only available in a subset of patients, limiting direct comparison) (Supplemental Tables 15–17).

We next evaluated the antibiotic-reducing potential of each model. We identified all SBI-negative patients in the validation sets provided antibiotics during the first 24-hours of their PICU admission and calculated the percentage of these children identified by each model (Fig 2). Among antibiotic-unexposed patients, 469 SBI-negative children were started on antibiotics upon PICU admission. The RF model identified 33 of 469 (7%) of these children, produced 4 incorrect SBI-negative predictions out of 163 total SBI-negative predictions (false negative rate [FNR] 2.5%), and could have spared these children a median 1.1 (IQR 0.9–3.7) antibiotic-days per patient.

Among antibiotic-exposed patients, 442 SBI-negative children were continued on antibiotics after PICU admission. The radial-SVM model identified 48 of 442 (11%) of these children, produced 2 incorrect SBI-negative predictions of 56 total SBI-negative predictions (FNR 3.6%), and could have spared a median 3.7 (IQR 0.9–9) antibiotic treatments-days per patient (Fig 2, Table 3, and Supplemental Figure 7). See Supplemental Tables 18 and 19 for cut-points used to achieve goal NPV.

When clinical variables from VPS were included as model inputs, model performance improved. In the antibiotic-unexposed stratum, the RF model (AUROC 0.77, NPV 0.96) identified 58 of 469 (12.4%) SBI-negative patients started on antibiotics in the PICU and produced 10 incorrect SBI-negative predictions of 282 total SBI-negative predictions (FNR 3.5%). In the antibiotic-exposed stratum, the RF model (AUROC 0.81, NPV 0.96) identified 72 of 442 (16%) SBI-negative patients continued on antibiotics in the PICU and produced 3 incorrect SBI-negative predictions of 75 total SBI-negative predictions (FNR 4%) (Supplemental Table 20).

To leverage the large amount of inpatient data available, many machine learning-based predictive models have been developed to predict deterioration from various conditions.^51–53  These include EHR data-driven models aimed at early detection of sepsis in adults^54–56  and children^57,58  with promising results. We hypothesized a similar approach might improve our ability to identify critically ill children at low risk of SBI. To our knowledge, this is the first study using a machine learning approach to identify SBI-negative children at PICU admission. We found that antibiotics were initiated in 39% of SBI-negative patients and continued for several days after PICU admission, further highlighting the need for high-confidence clinical decision support for antibiotic decision-making.

In this study, high-dimensional machine learning predictive models incorporating a variety of EHR-available data elements were able to predict SBI status at PICU admission and identify, with high confidence, a portion of SBI-negative children for whom subsequent antibiotics could have been safely avoided. In the antibiotic-exposed stratum, the radial-SVM model would have facilitated antibiotic discontinuation in 11% of SBI-negative children (19% when VPS inputs were included) with high confidence (NPV of 0.96). In antibiotic-unexposed children, the models did not perform as well, demonstrating lower AUROC values and identifying a lower percentage of SBI-negative children. One possible explanation for this discrepancy is that, in antibiotic-unexposed patients, these children have not yet manifested the severity of illness, generating data needed for the models. Given this performance gap, such models might be best suited to aid in antibiotic discontinuation for children provided antibiotics before PICU admission. Additionally, though the NPV for many models was 1.0 in this retrospective study, the observed NPV will realistically be <1.0 when evaluated prospectively, given variation in patient characteristics over time.

We also evaluated the predictive ability of 4 serum biomarkers (WBC, ANC, CRP, and PCT) for comparison, primarily used clinically and in prior studies^21,26,29,32  to predict pediatric bacterial infection. Biomarker AUROC values were mostly low-to-moderate and similar to prior studies,²⁷  though interpretation is limited by the low proportions of children with such laboratories available.

Within the models, variables of highest importance included a variety of data element types: vitals, laboratory results, laboratory imputation flags, and demographics. For both antibiotic-exposure strata, models used multiple parameters of lung disease (Fio₂, Spo₂, and RR) consistent with the high proportion of pneumonia SBIs in our cohort (a proportion similar to prior PICU SBI studies²⁷ ). The diversity of impactful variable types used by these models emphasizes the importance of this multiple input approach to SBI-negativity prediction.

Ultimately, though providers are often aware of many of these heterogeneous variables, a real-time algorithm generating and displaying SBI risk could be an invaluable clinical tool. By focusing on identification of SBI-negative patients, similar multiple input models might allow a clinician to spare identified children from several days of unnecessary antibiotic therapy that, in turn, might decrease rates of known adverse outcomes associated with antibiotics, including nephrotoxicity, clostridium difficile colitis, and prolonged hospital length of stay. Further work is needed to determine the impact of model-driven predictions on these adverse outcomes and to evaluate their potential benefit when compared with current standards of care.

In this single-center, EHR-based study, we were unable to incorporate interventions (including antibiotics) and results from nonnetwork referral hospitals and prehospital environments. Even when included in EHR free-text, incorporation was not possible with our system’s current natural language processing capability. This limitation also precluded strict adherence to certain SBI definitions from Lautz et al.²⁷  Specifically, we were unable to extract pneumonia interpretations from radiograph result text nor specific symptoms from patient note text. Thus, we relied on urine culture for UTI identification and the VPS database for pneumonia identification.

To be pragmatic for future prospective evaluation and use, patients were not excluded for missing data. As such, blood pressure normalization required a novel approach, and missing laboratory results required imputation. Our imputation approach will facilitate real-time, prospective evaluation by minimizing required computational resources (potentially needed for other imputation strategies). However, it is unknown how this approach will affect model predictive ability, and future work should evaluate model types, such as gradient boosting, that may not require missing value imputation. Encouragingly, AUROC values for models created with and without imputation of missing laboratory results were similar, suggesting imputation did not create significant noise or bias.

Additionally, certain patients never had cultures drawn, and, for some, cultures were drawn after antibiotic administration, potentially obscuring certain SBI outcomes. Also, our inclusion criteria relied upon the ordering of specific studies (laboratory results, cultures, and radiographs) and thus, on clinician practice patterns. Though patient management likely varies from provider-to-provider (potentially introducing bias into the selected population), we believe focusing model development on the cohort of patients for whom there was some clinical concern for SBI will yield predictions most likely to drive changes in antibiotic-decision making once implemented prospectively.

It was difficult to assess the predictive utility of certain preselected biomarkers owing to the small number of patients with such laboratories available and difficulty assessing how a normal or abnormal value might impact the likelihood of pneumonia diagnosis by the clinician. Finally, this study was performed at an institution with an established, robust antibiotic stewardship program, and the median number of reducible antibiotic-days per patient might be lower than at other institutions.

High-dimensional, machine learning-based predictive models using EHR data are able to predict SBI status at the time of PICU admission. The models were able to identify many SBI-negative children who were continued on antibiotics after PICU arrival but did not perform as well among the SBI-negative children who were started on antibiotics in the PICU. Overall, these results suggest that predictive models have the potential to aid in antibiotic decision-making in children undergoing infectious evaluation at the time of PICU arrival. Incorporating model-based predictions of SBI risk has the potential to safely reduce antibiotic-days in low-risk children. Future work should include prospective model validation in a research context to test their readiness for clinical use.