Machine Learning To Predict Serious Bacterial Infections in Young Febrile Infants

We performed a secondary analysis of a multicenter prospective study performed between December 2008 and May 2013 by the Pediatric Emergency Care Applied Research Network (PECARN) in 26 EDs. The current study was performed by using a public use data set obtained from the parent study and was designated nonhuman subjects research by our institutional review board.²¹  Analyses were conducted by using R version 3.6.1 (R Foundation for Statistical Computing, Vienna, Austria).

Inclusion criteria of the parent study have been published.²²  A convenience sample of patients ≤60 days of age with fever (≥38°C in the preceding 24 hours) evaluated in study site EDs were included. Patients were excluded from the parent study for clinical sepsis, recent antibiotic use, prematurity, preexisting medical conditions, or soft tissue infections. We additionally excluded encounters that were missing any of the following: blood or urine cultures, serum procalcitonin (retaining levels performed on the first or second day of admission), white blood cell count (WBC), absolute neutrophil count (ANC), or urinalysis. We excluded encounters lacking cerebrospinal fluid (CSF) cultures or a phone or medical record review performed 8 to 14 days after the encounter visit to verify that discharged patients without lumbar puncture did not have bacterial meningitis. We developed our model with procalcitonin given the high diagnostic accuracy reported with respect to this predictor for SBI.^9,10,23,24  Similar inclusion criteria were used in the PECARN decision rule.⁷ 

Our outcome was SBI, defined as bacterial meningitis, bacteremia, and/or UTI. Cultures were reviewed by using standardized criteria from the parent study.²⁵  UTI was defined using as ≥1000 colony-forming units (CFUs) per milliliter from urine culture obtained via suprapubic aspiration, ≥50 000 CFU/mL from catheterization, or 10 000 to 49 999 CFU/mL from catherization with a positive urinalysis (leukocyte esterase, nitrite, or >5 WBC per high-power field).²²  Candidate features included age, attending or fellow clinical suspicion for risk of SBI (dichotomized as 1%–10% and >10%), sex, fever location (in enrolling ED or elsewhere), WBC, ANC, serum procalcitonin, and urinalysis.

We performed 2 steps to identify and retain features that were maximally predictive of outcome, while removing features which would limit accuracy. First, we performed univariate logistic regression to test for association of predictors with SBI, retaining variables with P < .10. Second, we performed principal component analysis (PCA), in which orthogonal transformation was performed to linearly uncorrelate variables and identify features that discriminated positive from negative cases of SBI. In PCA, a small angle between vectors implies a positive correlation, a large one suggests negative correlation, and a 90° angle indicates no correlation between characteristics. The farther away these vectors are from a principal component origin, the greater influence these carry in PCA.²⁶ 

We randomized patients into derivation and validation cohorts with equal partitions, balanced to ensure equal numbers of SBIs. We developed 4 supervised learning models.²⁷  First, we performed multivariable logistic regression. Logistic regression uses a logistic function to model a binary outcome, resulting in the generation of coefficients that can be used to predict the logarithm of the odds of SBI. This model was optimized to obtain the lowest Akaike Information Criterion using bidirectional stepwise regression. Second, we performed random forest modeling, an ensemble method (in which multiple algorithms are used) whereby numerous decision trees are developed on random subsets of features during model development. These are then run individually and averaged to calculate probabilities of an outcome. The hyperparameter (a parameter used to tune the training process) for the number of variables available for splitting at each tree node was derived from features identified during PCA and verified by using a 10-fold cross validation repeated 3 times. Third, we performed modeling using a support vector machine (SVM) using a radial kernel. SVM is an algorithm in which a boundary is created with the widest margins between each type of predetermined classification. Hyperparameters were identified using 10-fold cross validation. Fourth, we fitted a single-hidden layer neural network as a feed-forward multilayer perceptron. The neural network consists of 3 “layers” in which data are input, analyzed through a serious of transformations, then output into results. We used bootstrapped resampling performed 25 times to identify final values for size and decay functions. Analyses were performed using the MASS (v7.3–51.4), randomForest (version 4.6–14), e1071 (v1.7–2), and nnet (v7.3–12) packages.

Models were assessed by using the validation cohort. We estimated the area under the receiver operating curve (AUROC). Predictions from logistic regression were converted to probabilities using the inverse logit formula. Optimal thresholds were estimated for each model by using a misclassification cost algorithm in which the relative cost of false-negatives was 100 times greater than false-positives.⁷  For each model, we reported (with 95% confidence intervals [CIs]) parameters of diagnostic accuracy, area under the precision-recall curve (AUPRC), and the F1 score. We additionally compared models to 3 recently published models: PECARN, Step-by-Step, and Aronson (Supplemental Table 6).^6–8  For the Step-by-Step model, we a priori determined that an infant with a >10% physician suspicion of SBI was criteria for “ill-appearing,” and we classified intermediate-risk patients as high-risk.⁸  For the Aronson model, we used a score of ≥2 to classify patients into high- and low-risk groups.⁶  We assessed the lumbar punctures that may have been avoided by low-risk group identification from each model.

We performed an exploratory analysis using a modified inclusion criteria. Patients with complete and interpretable culture data were considered for inclusion. Predictors of interest that were missing in >15% of cases were removed, because imputations on data with high proportions of missing data can represent a source of bias.²⁸  For remaining cases, we performed imputation for missing values using predictive mean matching. We then validated the highest performing machine learning model. We performed a post hoc analysis whereby imputation was performed for all missing data from patients having complete and interpretable cultures. Second, we performed an analysis using an outcome limited to bacterial meningitis and bacteremia. Third, we redeveloped the model with highest accuracy using a modified outcome in which patients diagnosed with UTI in the primary analysis by colony counts of 10 000 to 49 999 CFU/hpf with positive urinalysis were considered non-SBIs. Our outcome remained SBI. Fourth, given the limited availability of procalcitonin in North America,^29,30  we performed an analysis creating a machine learning model without this test. We additionally evaluated the performance of such a model using only objective measures.

Of 7335 encounters, 1470 were included (Fig 1). A total of 1014 patients (69.0%) were >28 days old. A total of 138 of 1470 (9.3%) patients had SBIs, including 122 (8.3%), 20 (1.4%), and 8 (0.5%) with UTI, bacteremia, and bacterial meningitis, respectively. Eleven (0.7%) had concurrent SBIs. A total of 1240 (84.4%) patients had a lumbar puncture performed, and 1050 (71.4%) were admitted.

Predictors significantly associated with SBI were age, suspicion of SBI, sex, WBC, ANC, positive urinalysis, and procalcitonin (Table 1). In evaluation of non-SBIs cases on the PCA biplot (Supplemental Fig 4), age, urinalysis, and sex strongly influenced the second principal component (PC2). WBC, ANC, and procalcitonin influenced the first principal component (PC1). In particular, ANC and WBC were found to be positively correlated and had a small angle between their vectors in the PC1 dimension. For positive SBI cases, urinalysis, procalcitonin, physician impression of SBI, WBC, and ANC separated PC1. Age, and sex influenced PC2. A total of 4 variables, urinalysis, procalcitonin, ANC, and WBC, based on distances of the vectors from the origin and also how variables correlated with one another on the PCA biplot, were retained for modeling.

A total of 735 patients were placed into derivation and validation cohorts. Sixty-nine (9.4%) patients in each cohort had SBIs. Groups were similar with respect to predictor and outcome variables with the exception of WBC (Table 2).

The AUROC of models in the validation cohort were high with the highest found in the random forest model (AUROC: 0.96, 95% CI: 0.93–0.98) (Fig 2). This model had the highest specificity on the receiver operator curve (74.9%, 95% CI: 71.5%–78.2%) (Table 3), AUPRC, and F1 statistics (Table 4). The relative importance of features contained within the random forest model is provided in Fig 3. The random forest model demonstrated a sensitivity comparable to the PECARN, Aronson, and Step-by-Step models, but with specificity above the upper limits of the CI for each comparison model. Using the random forest model to classify infants into high and low-risk groups revealed that 849 of 1240 (68.5%) patients who underwent a lumbar puncture could have potentially avoided the procedure. When applying the PECARN, Step-by-Step, and Aronson rules to this study sample, each respectively would have resulted in 673 of 1240 (54.3%), 460 of 1240 (37.1%), and 319 of 1240 (25.7%) fewer lumbar puncture procedures.

No more than 4 SBIs were missed with any machine learning model. The random forest and logistic regression models missed 1 patient. The SVM and neural network models missed 4 cases (Table 5). The PECARN, Step-by-Step, and Aronson models had 3, 9, and 1 false-negatives, respectively.

In our first exploratory analysis, we used imputation to test model performance on a cohort of patients that had missing data elements in <15% of cases, keeping other inclusion criteria the same. A total of 1537 encounters were included, with 158 (10.3%) having SBIs (Supplemental Table 7). Application of the random forest model to this data set resulted in one missed case, with high sensitivity and specificity (Supplemental Table 8). In our second exploratory analysis, the model performed similarly to results reported in the primary analysis when using an outcome limited to bacterial meningitis and bacteremia (Supplemental Table 9). In our third exploratory analysis using a modified definition for UTI, 17 UTIs with lower colony counts were reclassified as non-UTIs. Model performance was similar in this analysis as for the primary study (Supplemental Table 10). In a post hoc analysis, we performed imputation for all missing data, including procalcitonin, while using the full SBI definition. This model demonstrated a sensitivity of 94.9% (95% CI: 92.3%–96.8%) and specificity of 74.3% (95% CI: 72.9%–75.7%; Supplemental Table 11).

In creation of a model without procalcitonin, 3989 patients were included (Supplemental Fig 5). Three hundred ninety (9.8%) had an SBI (335 [8.4%] UTIs, 73 [1.8%] bacteremia, and 21 [0.5%] bacterial meningitis; 38 [1.0%] with concurrent infections). The variables of physician suspicion of SBI, age, sex, urinalysis, and ANC were selected from the PCA biplot. This model achieved a sensitivity of 98.0% (95% CI: 94.9%–99.4%) and specificity of 42.4% (95% CI: 40.1%–44.7%) (Supplemental Tables 12 and 13). In development of a random forest model including only objective measures (ANC, urinalysis, patient age, and sex, omitting the clinician suspicion for SBI), the specificity from the validation cohort was lower compared with the model that included clinical assessment (Supplemental Tables 14 and 15).

We developed machine learning models to risk-stratify infants ≤60 days of age for SBI. We identified a random forest model that demonstrated high sensitivity (99%) and specificity (75%) compared with previously published models.^1–8,31  This model missed 1 patient out of 138 with SBI (0.7%). Machine learning models for the risk-stratification of febrile infants demonstrated high accuracy and may help support clinical decision-making to minimize unnecessary hospitalizations, antibiotics, and lumbar punctures.

Kuppermann et al,⁷  in their recent model, demonstrated a sensitivity of 98% and a specificity of 60%, using recursive partitioning. In our study, we use the public-use data set from that investigation, which may be a subset of the parent study. We also used slightly different inclusion criteria, such as the retention of procalcitonin levels performed within the first 2 days of admission. The Step-by-Step model, another recent predictive tool using ANC, urinalysis, procalcitonin, and C-reactive protein, demonstrated a sensitivity of 92% and specificity of 47%.⁸  A model reported by Aronson et al,⁶  for invasive bacterial infections, revealed a sensitivity of 99% and a specificity of 31%. Similar to these models, our model does not require CSF for risk-stratification. This model also demonstrates superior test characteristics compared with older models, including the Philadelphia (sensitivity 97%–99%, specificity 39%–42%)^2,31  and Rochester criteria (sensitivity 82%–97%, specificity 40%–50%).^3,5,8 

The use of supervised learning techniques carries potential to decrease medical interventions. Applying the random forest rule, for example, may have allowed lumbar punctures to be avoided when compared with previous rules.^6–8  Although incremental, this reduction in procedures represents an area of improvement. In addition to reduced procedures, more accurate classification of lower-risk infants may potentially result in reduced empirical antimicrobial therapy, hospitalizations, false-positive cultures, and nosocomial infection risk.³² 

Notably, one patient was classified as a false-negative in the random forest model with Enterobacter cloacae bacteremia. This patient was additionally misclassified as a false-negative in the PECARN and Step-by-Step rules.^7,8  According to Kuppermann et al,⁷  a repeated blood culture before the administration of antibiotics was negative, and the patient had a benign hospitalization. This misclassification stresses a continued need to develop more robust methods in the identification of febrile infants with SBI.

The random forest model was able to achieve the highest diagnostic accuracy among investigated machine-learning models. This occurred while using similar variables to those used in previous decision rules. Although a strength of a recursive partitioning approach is that it can be used by bedside clinicians without calculations, it does not fully exploit the computational power available in modern health care settings. In contrast, the random forest algorithm is computationally intense but likely better able to leverage the data, particularly with respect to continuous variables. The unbalanced nature of this data set, in which 10% of patients had SBIs, may have contributed to this result. With unbalanced data, standard classifiers such as SVMs, logistic regression, and decision trees tend to bias toward the majority (or negative) class and are further limited by smaller sample sizes.³³  In contrast, the random forest model, which provides predicted probabilities from multiple decision trees, may be better suited to these types of data.

Despite demonstrating superior test characteristics, our models carry important limitations compared with previously published guidelines. Unlike rules that provided parameter cutoffs,^1–8  these models require use of algorithms which are not intuitive. A sample tree, provided in Supplemental Fig 6, demonstrates one such decision tree made in the random forest model; the final model contains 5000 of such trees, the results of which are aggregated to provide a final probability. Integration of these findings into health information systems, including through interactive web applications or electronic medical record systems, may mitigate this disadvantage. For example, a validated model was recently developed into a web application to support the real time risk assessment of UTI in children 2 to 23 months of age.³⁴  Other investigators have reported on use of sepsis alerts to notify clinicians of at-risk patients using machine learning data.^35–37  A model to identify at-risk febrile infants may allow for greater diagnostic accuracy by leveraging available computational resources available through integration with the electronic medical record. Importantly, before clinical deployment, validation of such a model is paramount. Models may demonstrate a decline in performance when tested on other data sets.³⁸  This concern is especially relevant with respect to machine learning models.³⁹ 

Our findings are subject to limitations, including those with respect to use of a convenience sample and use of cultures to determine true-positives. However, rates of SBI from the parent study appear to be similar to those documented elsewhere, suggesting that this is not likely a major source of error.^8,13  Models were developed on a smaller-sized data set. We excluded patients with procalcitonin, which resulted in a large number of exclusions. This step was similarly performed in the PECARN model.⁷  We did not use the exact same inclusion criteria that was used in the PECARN model, or the same derivation and validation cohorts. This resulted in slightly different study numbers. Our models may be overfit and require external validation to assess generalizability.⁴⁰  Despite these limitations, the findings from this analysis suggest that machine learning models have the potential to perform well in the identification of SBI among populations of well-appearing febrile infants.

We evaluated machine-learning algorithms in the risk-stratification of well-appearing febrile infants ≤60 days old from a multicenter prospective study. Although they need external validation, we suggest that ensemble machine learning algorithms maintain the high sensitivity of recently published decision tools while providing higher specificity. This in turn would allow for more accurate identification of patients without disease, leading to reduced invasive procedures, antimicrobial agents, and hospitalizations.

ANC

absolute neutrophil count

AUPRC

area under the precision-recall curve

AUROC

area under the receiver operating curve

CFU

colony-forming unit

CI

confidence interval

CSF

cerebrospinal fluid

ED

emergency department

PC1

first principal component

PC2

second principal component

PCA

principal component analysis

PECARN

Pediatric Emergency Care Applied Research Network

SBI

serious bacterial infection

SVM

support vector machine

UTI

urinary tract infection

WBC

white blood cell count