Refinement and Validation of a Clinical-Based Approach to Evaluate Young Febrile Infants Free

Prompting >500 000 clinical encounters each year,^1,2  fever in young infants is a common clinical sign that may indicate a bacterial infection, including urinary tract infection (UTI), bacteremia, and meningitis.³  If undiagnosed, these infections result in sepsis and death. However, detection in febrile infants is challenging because infants may not demonstrate signs of severe illness.⁴  For decades, researchers have developed predictive models to identify infants at low risk of bacterial infections, characterized by high sensitivities, to avoid missing infants with infections.^5–9  To enhance prediction, researchers have incorporated sophisticated biomarkers, such as procalcitonin, into diagnostic algorithms.^10–12  However, these models reveal wide variation in clinical adoption.^13–15  This is likely due to implementation barriers, including the presence of multiple predictive models with different inclusion and exclusion criteria, clinician differences in risk tolerance/aversion, and challenges in obtaining predictor variables, such as serum biomarkers.^16,17  In addition, most children seek care in community hospitals and outpatient clinics,^18–20  with varying capabilities of performing onsite and timely testing with procalcitonin and other serum biomarkers.²¹  Thus, infants may receive disparate care related to available institutional resources.

To address these problems, we selected widely available predictors from previous research,^{5,6,9,10,13,22–25}  including demographic and clinical factors, plus urine studies, to derive bacterial infection predictive models using regression and machine learning methods.²⁶  We derived these models with 1 overarching goal: to guide clinicians’ decision-making for febrile infants ≤90 days of age in diverse settings, including emergency departments (EDs) and outpatient clinics. These predictive models were as sensitive, yet more specific, compared with the Rochester Low Risk model. As demonstrated previously,²⁷  it is important to externally validate models to ensure they perform reliably across diverse settings to avoid missed infections before proceeding to prospective validation. External validation also afforded an opportunity for refinement, focusing on 1 specific feature. We initially included insurance status as a predictor in the derivation model because it is often used as a proxy for socioeconomic status.²⁸  Our preliminary work,²⁹  and other studies in children and adults,^30–32  indicate that low socioeconomic status is associated with bacterial infection risk. However, insurance status holds a dual role in the United States because it can be a proxy for both socioeconomic status and access to care. In addition, it may exhibit different effects across states because of differences in Children’s Health Insurance Program eligibility and other local considerations. Thus, to reduce potential biases related to insurance status, the objective of this study was to both refine and externally validate the predictive models in a separate population.

We performed a retrospective cross-sectional study of all infants ≤90 days old with fever brought to 1 pediatric ED in the northeast from January 1, 2011 to December 31, 2018. This population is different and geographically distinct from the sample used for the derivation study. We obtained institutional review board approval for this study.

Study team members used a standard protocol and manually reviewed a complete data extract of ED encounters to identify eligible infants. We iteratively updated the protocol and study personnel met weekly to ensure that abstraction techniques were consistent. After abstraction, 1 study team member randomly coded 10% of predictor variable data in a blinded fashion to calculate interrater reliability. We used unweighted Cohen’s κ, weighted Cohen’s κ, and intraclass correlation coefficients for categorical, ordinal, and continuous predictor variables, respectively. Inclusion criteria consisted of (1) age 0 to 90 days old, (2) fever (ie, temperature ≥38°C) within 6 hours of ED arrival or reported by the caregiver from a measurement before arrival but during the current illness, (3) documented insurance type (eg, private), (4) documented gestational age, and (5) initial evaluation in the ED.

We previously derived machine learning (ie, super learner) and regression predictive models to detect bacterial infections in young, febrile infants.²⁶  To reduce potential biases related to insurance status, we used the derivation data to derive the super learner and regression models again, this time without insurance status. Thus, for this external validation study, we used 9 predictor variables for each refined model.²⁶  Categorical predictors included sex, presence of a chronic medical condition (yes or no), appearance (well, ill, or not documented), cough status (yes, no, or unknown), and urinary tract inflammation (yes, no, or not ordered), defined as ≥5 white blood cells/high power field of unspun urine or positive (≥trace) leukocyte esterase.²⁴  Two authors reviewed all records and categorized chronic medical conditions as present or absent on the basis of the likelihood that the condition could be related to a bacterial infection.²⁶  We calculated interrater reliability and discussed discrepant chronic medical conditions until we reached a consensus. To distinguish ill from well appearance, we used keywords such as toxic, limp, inconsolable, ill-appearing, listless, lethargic, irritable, and unresponsive.^26,33  A “Review of Systems” template, which noted the presence or absence of cough, was consistently completed in the ED per standard practice. Ordinal variables included age (days), caregiver report of gestational age at birth (weeks), and illness duration (any symptom) in days. We used 37.5 weeks for infants recorded only as “full term.”

The primary outcome was bacterial infection (UTI, bacteremia, or meningitis). We defined UTI as (1) a catheterized urine specimen that grew ≥10 000 CFU/mL of a pathogenic organism, (2) urinary tract inflammation (ie, ≥5 white blood cells/high power field of unspun urine or positive [≥trace] leukocyte esterase),^5,10,11,34  and (3) clinical management as a pathogen.^10,11,24,26  We defined bacteremia and meningitis as growth of a single pathogenic organism from blood and/or cerebrospinal fluid cultures that was treated clinically as a pathogen.²⁶  On the basis of a combination of a priori categorization³⁵  and review of the clinical course, two authors (a pediatric hospitalist and a pediatric infectious diseases physician) designated ambiguous culture results as pathogens or contaminants. We tracked infants for missed bacterial infections within 7 days of the index encounter through the electronic health record, which captures data from the other major health system.

We used descriptive statistics to characterize the demographic and clinical characteristics of the sample. We tested the refined regression and super learner models without insurance status as a predictor, applying them to detect bacterial infections in this distinct sample. A super learner model is an ensemble learning method that combines several machine learning algorithms to produce a single predictive model that is superior to each individual algorithm.³⁶  We used the “SuperLearner” R package (R Foundation for Statistical Computing, Vienna, Austria), which includes random forest, earth, generalized additive models, and a generalized linear model.²⁶  The output of each model is a bacterial infection risk estimate. We determined the area-under-the-receiver operating characteristic curve (AUC) for each model and used prespecified risk estimate thresholds (0.005, 0.01, 0.03, and 0.05) to calculate sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and likelihood ratios. To further illustrate model performance, we calculated F1 scores because they represent the harmonic mean between precision (positive predictive value) and recall (sensitivity).³⁷ 

Given the challenge of detecting bacterial infections and the varied approaches used to manage febrile infants, we performed exploratory analyses. First, because infants with invasive bacterial infections (IBIs) (ie, bacteremia and/or bacterial meningitis) are more likely to experience adverse outcomes,³⁸  we assessed each model’s ability to detect IBIs. Second, to align with the new Clinical Practice Guideline from the American Academy of Pediatrics and other studies,^11,39,40  we evaluated each model’s performance in full-term, well-appearing infants aged 8 to 60 days. Third, we calculated diagnostic characteristics for well-appearing infants because previous models have excluded ill-appearing infants.^12,39  Fourth, because neonatal infants with bacterial infections do not reliably appear ill¹⁰  and are often considered “not low risk”, we applied the super learner model to infants > 30 days of age. We evaluated the overall performance of this 31- to 90-day approach by hard-coding all infants ≤30 days old as not low-risk and used the regression and machine learning models to detect bacterial infections in this older group. We then calculated diagnostic characteristics of the entire sample. Last, because leukocyte esterase is a sensitive urinary biomarker to detect UTIs that can be done quickly and is readily available,⁴¹ we repeated the derivation and validation studies with leukocyte esterase as the sole marker of urinary tract inflammation. We used “R” to perform the statistical analysis. Data and code are available at: https://github.com/jmiahjones/pediatric-bi-val-2021.

Overall, 17 042 infants ≤90 days old were evaluated in the ED from 2011 to 2018 (Fig 1) and 1419 infants met inclusion criteria (median age 53 days [interquartile range (IQR) = 32–69]) (Table 1). Characteristics of the derivation sample are also shown in Table 1. The Cohen’s κ statistic for designating chronic medical conditions was 0.85, and we reached a consensus on the final designation of each condition (Supplemental Table 4). For manually abstracted predictors, interrater reliability scores ranged from 0.96 to 0.99. Of the 99 (7%) infants with bacterial infections, 22 (1.6%) had an IBI. Escherichia coli and Streptococcus agalactiae were the most common organisms isolated (Supplemental Table 5).

The AUCs of the super learner and regression models in detecting bacterial infections were 0.92 (95% confidence interval [CI] 0.89–0.94) and 0.90 (0.86–0.93) (Supplemental Fig 2) compared with 0.96 (0.94–0.98) and 0.95 (0.91–0.98) in the derivation study.²⁶ Table 2 reveals the performance of the super learner and regression models at each designated bacterial infection risk estimate. For example, the sensitivity and specificity of the super learner model using a bacterial infection risk estimate of 0.005 was 98.0% (94.7%–100%) and 54.2% (51.5%–56.9%).

For the exploratory analyses, estimates of sensitivities for the super learner model were greater than the regression model. Thus, we report findings for the super learner model here and in Table 3. Regression model results can be found in Supplemental Table 6. For the outcome of IBIs using the super learner model, we observed a sensitivity of 90.9% (77.3%–100%) and specificity of 51.2% (48.6%–53.8%). For the remaining exploratory analyses, sensitivities and specificities were at least 96% and 48%, respectively, to detect bacterial infections (Table 3). For IBIs, the 31- to 90-day approach revealed a sensitivity and specificity of 95.5% (84.2%–100%) and 43.3% (40.8%–45.9%), respectively (Table 3). Using a bacterial infection risk estimate of 0.005, the regression model misclassified 3 infants with IBIs as low-risk (0.5%), whereas the super learner model and the 31- to 90-day approach (using the super learner model) misclassified 2 (0.3%) and 1 (0.2%) infants with IBIs as low-risk, respectively (Supplemental Table 7). Characteristics of misclassified infants are shown in Supplemental Table 8.

Two infants were diagnosed with bacterial infections within 7 days of the index encounter, both with urinary tract infections (Escherichia coli). One infant did not have a urinalysis performed at the initial encounter and the other infant had a normal urinalysis at the index encounter. No infants were diagnosed with IBIs within 7 days of the initial encounter. Only 2.5% of predictor variables were coded as “unknown” (cough status) or “not ordered” (urinalysis). There were no missing data.

Findings in this validation study reveal that AUCs of the super learner and regression models were similar to results from the derivation study, even after removing insurance status as a predictor variable. This was important to reduce potential sources of bias related to unclear and likely varied effects of insurance status in different locations. Point estimates of the super learner model were greater, likely because, as an ensemble method, the super learner model is able to use a collection of machine learning algorithms to analyze data in linear and nonlinear manners, increasing the chances of accurate classification and generalizability.^36,42,43  Results suggest a clinical-based predictive model can accurately estimate bacterial infection risk in young infants with fever. Compared with the derivation study, the validation sample was composed of more ill-appearing infants and lower proportions of Hispanic infants and infants with public insurance,²⁶  demonstrating that it performs well across a range of settings and populations.

Compared with previous serum biomarker models,^5–12,39  our model revealed similar sensitivities and specificities to detect bacterial infections and IBIs for the entire sample, subsets of full-term, well-appearing infants 8 to 60 days old and well-appearing infants 0 to 90 days old. Similarly, the 31- to 90-day approach revealed high sensitivity and specificity in detecting bacterial infections and IBIs while avoiding missed infections among infants in the first month of life. Additionally, findings suggest that leukocyte esterase alone can be used as a marker of urinary tract inflammation. Prospective investigation is needed to confirm the clinical-based model can accurately detect bacterial infections and IBIs in febrile infants and to directly compare findings with current models.

Rather than identifying febrile infants at low risk for a bacterial infection, we sought to provide guidance to clinicians in any setting faced with evaluating and managing any young, febrile infant. When prospectively validated in an appropriately powered sample, this approach is positioned to reduce barriers related to clinical adoption and address possible inequities by increasing access to highly sensitive predictive models in resource-limited settings. This paradigm shift is characterized by a few key features, which are highlighted below.

Previous studies have excluded ill-appearing and premature infants,¹¹  focusing on a narrower age range (ie, ≤60 days old). This approach implies that all ill-appearing and premature infants are not low-risk for bacterial infections and merit full evaluation, including blood, urine, and cerebrospinal fluid cultures. Although bacterial infection risk may be as high as 25% in ill-appearing infants,¹⁰  there may be a subgroup of infants with intermediate or low risk. This designation change could alter management (eg, a subset of infants could be safely monitored in a hospital without antimicrobial therapy). By including ill-appearing infants, our clinical-based model is well-positioned to provide clinicians with a more personalized management strategy.

Even accurate predictive models reveal limited clinical adoption, often related to unanticipated implementation barriers.^16,17  Barriers typically refer to the ease or difficulty with which predictor variables can be obtained and are frequently measured in terms of time needed for collection, financial costs, and pain or stress of the patient or caregiver.⁴⁴  Current models to detect bacterial infections in febrile infants include at least 1 serum biomarker,^5–12,39,40  which requires time for venipuncture and to obtain, deliver, and process the specimen, ranging from minutes to hours and possibly delaying care.^45,46 

To improve clinical adoption, we have omitted serum biomarkers, which may reduce the time to reach a decision. During encounters with febrile infants, clinicians could obtain a urinalysis from a bag specimen,⁴⁰  placed during triage, with reflex catheterization to obtain cultures for samples with evidence of urinary tract inflammation. Simultaneously, clinicians could quickly obtain other clinical history variables and assess the infant’s appearance. Once urinalysis results are available, clinicians could simply enter predictor information into the web-based risk calculator⁴⁷  and obtain an accurate, highly sensitive bacterial infection risk estimate to inform clinical decisions. Other than an internet connection and urinalysis, no other resources would be needed, and invasive, painful procedures could be safely avoided for a subset of low-risk infants. The novel predictive model may be particularly relevant for outpatient clinic settings, as it could facilitate decision making about ED referral. When prospectively validated in an appropriately powered sample, clinicians could use the risk calculator to guide the management of any young, febrile infant. In addition, our exploratory analysis suggests that urine dipstick results, specifically leukocyte esterase, can be used as a predictor to detect bacterial infections with high sensitivity. Although future studies should investigate the extent to which this approach should be used, this finding is promising because the clinical-based model may reduce wait times and could represent an important decision support tool in resource-limited settings.

Each low-risk predictive model uses different inclusion/exclusion criteria, predictors, and thresholds,^5,6,8,10,11  increasing clinicians’ cognitive load in choosing and interpreting a predictive model. In contrast, our clinical-based approach uses inclusion criteria that are simple and intuitive (ie, age 0–90 days; temperature ≥38°C) and uses readily available predictors without the need for invasive and possibly challenging venipuncture or lumbar puncture procedures; there are also no low-risk threshold values for clinicians to remember. Our clinical-based approach considers all predictors and the entire spectrum of values.

Every predictive model will miss a small proportion of bacterial infections. Similar to other studies,¹⁰  the super learner model missed one 11-day-old infant with meningitis, indicating that, if the objective is to avoid all missed cases of meningitis, it may be prudent to consider the 31- to 90-day approach for which there were 0 missed cases of meningitis. Clinicians’ preferences vary along the spectrum of risk tolerance to risk aversion. Additionally, when encountered with an unfamiliar, yet potentially fatal disease, caregivers may be more likely to agree to a clinician’s preferences without a clear understanding of the possible harms and benefits of management options. Although it has remained elusive so far, the super learner model holds promise as an important tool in a shared decision-making approach. Instead of evaluating individual predictors in isolation, the super learner model considers all predictor variables simultaneously, just as clinicians do. Thus, no single predictor would automatically merit not low-risk status, and predictor variable threshold values would be unnecessary. The output of the super learner model is a bacterial infection risk estimate, allowing clinician–caregiver dyads to modify risk-estimate thresholds (and therefore sensitivities/specificities) on the basis of their collective risk tolerance/risk aversion preferences.

Limitations include data that were collected retrospectively from a single site and based on clinician documentation. Some variables may be recorded inaccurately. However, we employed a rigorous process of manual data abstraction to minimize discrepancies and observed high interrater reliability, and there were no missing data. Additionally, results are similar to those obtained in the derivation study, which was performed at a separate site with a different electronic health record vendor, suggesting that biases in documentation may be low. Moreover, although we used the medical literature to select predictor variables, there may be included factors that are not important and other excluded factors that may improve prediction. Additional investigation may be needed to identify the optimal set of predictor variables. Next, we used a lower colony count (10 000 CFU/mL) to define UTI compared with other studies.^8,48  We did so because urine culture results at the site for the derivation study did not specify growth from 10 000 to 100 000 CFU/mL. Instead, we used an evidence-based alternative of 10 000 CFU/mL plus urinary tract inflammation to avoid misclassifying cases of asymptomatic bacteriuria.^10,11,24,26  Future studies should examine whether diagnostic characteristics differ using a UTI definition with a higher colony count. Last, the models, including the exploratory analyses, should not be used for clinical purposes yet and require additional prospective evaluation.

In this retrospective validation study, the refined super learner and regression models performed similarly relative to the derivation study. This study provides additional evidence that our clinical-based approach can accurately estimate bacterial infection risk in young, febrile infants. In addition, the performance of the super learner model is similar to previously published predictive models that use serum biomarkers.^5–12,39  The super learner model is well-positioned to achieve high rates of clinical adoption and may have important implications related to equity across sites of care, safely avoiding unnecessary interventions, and promoting a shared decision-making approach. Clinical- and biomarker-based models (ie, American Academy of Pediatrics Clinical Practice Guidelines) may complement each other because there may be circumstances in which a biomarker-based predictive model is preferred, and other situations may be better suited for a clinical-based model. Additional investigation is needed to determine if the clinical-based model could enhance current biomarker-based models and the extent to which the 2 model types could be used in concert, perhaps as a tiered approach. Future studies should further refine predictor variables, prospectively test the super learner model and the 31- to 90-day approach to detect bacterial infections and IBIs, examine predictive capabilities in infants ≤60 days old, and investigate methods to optimize clinical adoption.

We thank Courtney Richfield, BS, MA, of the University of Rochester, Department of Pediatrics, for her assistance in formatting, proofing, and processing this manuscript. Special thanks to Dr. Lauren Solan, MD, Med, Associate Professor of Pediatrics, University of Rochester, Rochester, New York, and Dr. Tina Sosa, MD, MSc, Assistant Professor of Pediatrics, University of Rochester, Rochester, New York, for their thoughtful suggestions and revisions.