Performance of the Modified Boston and Philadelphia Criteria for Invasive Bacterial Infections

We performed a planned secondary analysis of the Pediatric Emergency Medicine Collaborative Research Committee Herpes Simplex Virus Study. The parent study was a retrospective, cross-sectional study of infants evaluated for meningitis at 1 of 23 tertiary-care emergency departments (EDs) located in the United States and Canada from January 2005 to December 2013.²⁷  The enrollment period varied by study center on the basis of the availability of electronic health record data. The study protocol was approved by the institutional review board of each participating center with agreements for data sharing.

In our secondary analysis, we include infants 29 to 60 days of age who had a CSF and blood culture obtained within 24 hours of ED arrival. We limited our analysis to infants in the second month of life who could potentially be managed as outpatients if at low risk for bacterial infections.^16,17  We excluded infants who were critically ill (ie, admitted to an ICU) because the Boston and Philadelphia criteria were both developed for the evaluation of well-appearing infants.

We abstracted the following from the medical record: demographics (age, sex, and study site), clinical data (triage temperature, admission status, and length of stay), laboratory results (complete blood count [CBC] and differential, CSF cell counts, CSF Gram-stain, urine dip, and/or urine microscopy), and microbiology results (blood, CSF, and urine cultures).

Our primary outcome was IBI, defined as growth of pathogenic bacteria form either a blood or CSF culture. We classified the following bacterial organisms a priori as contaminants: coagulase-negative staphylococci, Viridans group Streptococcus, Lactobacillus species, Micrococcus species, Bacillus species (non-cereus and non-anthracis), Propionibacterium acnes, and Corynebacterium species.²⁸  Infants with a positive CSF culture result were defined as having bacterial meningitis regardless of the blood culture results.

Our secondary outcome was SBI, defined as IBI or UTI. We defined UTI as growth of ≥50 000 colony-forming units of pathogenic bacteria from a urine culture or growth of ≥10 000 colony-forming units of pathogenic bacteria from a urine culture in the presence of pyuria (>10 WBC per high-power field [hpf] or positive leukocyte esterase result).²⁹ 

We modified the published Boston and Philadelphia criteria for application to our study cohort (Table 1). The original Philadelphia criteria include spun urine microscopy, including urine Gram-stain. Because most study infants did not have this test performed, we modified the high-risk Philadelphia predictors to include both spun and unspun urine results on the basis of each study site’s laboratory.³⁰  Both criteria required urine microscopy to stratify risk. However, not all infants underwent urine microscopy, including those for whom urine dip screening results were negative. Therefore, if infants had both urine dip leukocyte esterase testing and urine microscopy performed, urine microscopy results were used to categorize the infants as having pyuria. If urine dip leukocyte esterase testing was performed without microscopy, a positive urine leukocyte esterase result (trace to 3+) was classified as a positive result to maximize each criteria’s sensitivity.^31,32  In the parent study, urine microscopy results were coded as follows: 0 to 5 WBC per hpf, 6 to 10 WBC per hpf, 11 to 20 WBC per hpf, and >20 WBC per hpf. For this study, we modified both the Boston and Philadelphia criteria to define pyuria as >10 WBC per hpf.

We applied the modified Boston and Philadelphia criteria to the study cohort. Infants with any of the high-risk predictors were considered not to be at low risk for IBI. Infants missing data for any of the predictors were excluded unless they had one or more high-risk predictors, in which case they were considered not at low risk.

We report the diagnostic accuracy of the modified Boston and Philadelphia criteria to identify infants with IBI and SBI. We report sensitivity, specificity, positive predictive value, negative predictive value, positive likelihood ratio, and negative likelihood ratio. Comparisons of test characteristics between the modified Boston and Philadelphia criteria were performed by using χ² tests. Finally, we describe the infants with IBI who were misclassified as low risk by either the modified Boston or Philadelphia criteria.

Statistical analyses were performed by using SAS software (SAS Institute, Inc, Cary, NC).

Of the 11 679 infants 29 to 60 days of age, we included the 10 928 infants (93.6%) who had both blood and CSF cultures obtained (Fig 1). The median age in this study cohort was 42 days (interquartile range [IQR] 35–50 days); 6089 (55.9%) were boys, and 8620 (78.9%) were admitted to the hospital. Overall, 10 286 (94.1%) also had urine cultures obtained and were included in our SBI subanalysis.

Of the 10 928 study infants, 264 (2.4%) had an IBI, including 71 (0.6%) with bacterial meningitis and 193 (1.8%) with bacteremia. Fifty-one (72.9%) infants with bacterial meningitis had positive CSF culture results alone, and 20 (28.2%) had both a positive CSF and blood culture result. The most commonly identified bacterial pathogen was GBS, followed by Escherichia coli and Staphylococcus aureus (Table 2).

In our primary outcome analysis for IBI, we applied the modified Boston criteria to 8344 (76.4%) infants (Fig 1), of whom 212 (2.5%) had an IBI, and the modified Philadelphia criteria to 8131 (74.4%) infants, of whom 219 (2.7%) had an IBI. The clinical and laboratory characteristics of the infants in the modified Boston and Philadelphia cohorts were similar (Table 3). Overall, 2322 infants (21.2%) did not have sufficient laboratory results to be classified by either criterion. The rate of IBI among the unclassified and classified infants did not differ significantly (44 of 2322 [1.9%] unclassified versus 221 of 8606 [2.6%] classified by at least one set of criteria; P = .06).

The modified Boston criteria had lower sensitivity (62.7% Boston versus 71.7% Philadelphia [difference 9.0%; 95% confidence interval (CI) 0.1% to 17.7%]) but higher specificity (59.2% Boston versus 46.1% Philadelphia [difference 13.1%; 95% CI 11.6% to 14.6%]) than the modified Philadelphia criteria for the identification of IBI (Table 4). Of the 79 (0.9%) infants with IBI who were misclassified by the modified Boston criteria as low risk, 17 (0.2%) had bacterial meningitis and 62 (0.7%) had bacteremia. Of the 62 (0.8%) with IBI misclassified by the modified Philadelphia criteria, 13 (0.2%) had bacterial meningitis and 49 (0.6%) had bacteremia. Both criteria had similar negative predictive value (98.4% Boston versus 98.3% Philadelphia [difference 0.1%; 95% CI −0.4% to 0.7%]). Cases of IBI misclassified by the modified Boston and Philadelphia criteria are summarized in Supplemental Tables 5 and 6, respectively.

Among the 10 286 infants included in our secondary analysis of the modified Boston and Philadelphia criteria performance for identifying SBI, 1053 (10.2%) infants had an SBI, including 805 (7.8%) with an isolated UTI (ie, without associated bacteremia or bacterial meningitis). Among infants with an isolated UTI, E coli was the most common pathogen, accounting for 653 (81.1%) infections. In this cohort, we were able to apply the modified Boston criteria to 7982 (77.6%) infants, of whom 960 (12.0%) had an SBI, and the modified Philadelphia criteria to 7733 infants (75.2%), of whom 963 (12.5%) had an SBI. The modified Boston criteria had lower sensitivity (79.4% Boston versus 86.2% Philadelphia [difference 6.8%; 95% CI 3.4% to 10.2%]) and higher specificity (64.6% Boston versus 51.3% Philadelphia [difference 13.3%; 95% CI 11.7% to 14.9%]) than the modified Philadelphia criteria for the identification of SBI (Table 4).

We evaluated the performance of the modified Boston and Philadelphia criteria to identify bacterial infections in a large multinational cohort of infants between 29 and 60 days of age undergoing ED evaluation for meningitis. Approximately one-third of infants with an IBI were misclassified as low risk by both of these fever criteria. Although both criteria had a high negative predictive value (>98%), this was driven largely by the high prevalence of infants without IBI (>97%). Furthermore, only 4% of those infants identified as high risk (potentially leading to hospitalization and parenteral antibiotic exposure) actually had an IBI. Given well-described changes in the epidemiology of IBI as well as practice patterns for the management of febrile infants, new tools to risk stratify febrile infants are needed to enable the accurate and rapid identification of infants with IBI without overtesting or treating low-risk infants.

The Boston and Philadelphia criteria were developed in the early 1990s to identify young, febrile infants at low risk for SBI who may be safely managed as outpatients.^16,17,33  Because the Boston criteria derivation study only enrolled low-risk infants, the criteria sensitivity cannot be calculated.¹⁷  However, rates of SBI in infants identified as low risk by the Boston criteria have been reported between 3% and 5%, which is similar to our IBI rate.^17,34,35  In the single-center derivation study of 747 infants, the Philadelphia criteria had a sensitivity of 98% (95% CI 92% to 100%) for SBI,¹⁶  and in follow-up studies, rates of SBI in low-risk infants have been reported between 0% and 3%, which is also similar to our findings.^35–37  Although both the original Philadelphia and Boston criteria were designed to identify infants with SBI, we chose IBI as our primary outcome because delays in diagnosis could result in significant morbidity and mortality, whereas infants with UTI could be safely managed as outpatients.³⁸  Importantly, the Boston and Philadelphia criteria performed similarly for the identification of SBI.

Changes in the microbiology of invasive infections³⁹  may explain, in part, the modest sensitivity of both fever criteria because the host immune response differs by pathogen⁴⁰  and both rely on measures of host immune response. The epidemiology of IBIs in febrile infants has changed over the past few decades, reflecting widespread conjugate vaccination for Haemophilus influenzae as well as Streptococcus pneumoniae (including introduction of the 13-valent pneumococcal vaccination during our study period). The impact has extended to the youngest infants,^21,41  presumably because of herd immunity.²²  A decrease in the rates of IBI caused by these pathogens was seen in our study: H influenzae, 11% to 25% (Boston and Philadelphia derivation studies) versus 1% (current study); S pneumoniae, 11% to 15% (derivation studies) versus 5% (current study). Although maternal antibiotic prophylaxis has prevented early-onset GBS, rates of late-onset GBS infections (ie, beyond the first week of life) have remained largely unchanged.^42,43  Consistent with this observation, our observed GBS rates are similar: 22% to 43% (Boston and Philadelphia derivation studies) versus 27% (our study).^{16,17,23,25,26,44} 

Furthermore, with recent data suggesting that lumbar puncture may be safely avoided in some infants 29 to 60 days old,⁴⁵  clinicians may be obtaining CSF only from infants with a higher clinical concern for meningitis, thereby leading to the Boston and Philadelphia criteria being applied to a different patient population from which they were derived. Despite this concern, our observed IBI rates are similar to other recent ED febrile infant cohorts.^{2,9,28,39,46–48}  Although we cannot apply either the Boston or the Philadelphia criteria to those infants who did not have CSF obtained, newer strategies should focus on methods to identify febrile infants who may safely avoid diagnostic lumbar puncture.

Given the limitations of these febrile infant criteria, more accurate ways to risk stratify infants at risk for IBI are needed. Newer biomarkers, including procalcitonin and C-reactive protein, have shown promise in risk stratifying infants.^5,18,28,49  The Step-By-Step Approach, using leukouria, serum procalcitonin, peripheral WBC, and C-reactive protein, had a sensitivity of 92% and specificity of 46% in a large prospective cohort of infants.⁵  The Pediatric Emergency Care Applied Research Network Febrile Illness Working Group recently published a clinical prediction model to identify infants at low risk for SBI using a cohort of 1821 prospectively enrolled infants. Infants with negative urinalysis results, absolute neutrophil count <4090 /mm³, and a procalcitonin <1.71mg/dL were at very low risk of SBI (sensitivity 97.7% [95% CI 91.3% to 99.6%] and specificity 60.0% [95% CI 56.6% to 63.3%]).⁶  Multiplex bacterial polymerase chain reaction panels can be used to identify bacterial pathogens from both blood and CSF with a sufficiently short turnaround time to impact decisions regarding antibiotics and hospital admission.^50–52  Finally, novel RNA expression patterns measuring host response to infection have also shown promise in identifying infants without acute bacterial infections but require further development before real-time clinical application.^53,54 

Our study had several limitations. First, our sample cohort was limited to those infants who had a CSF culture obtained. Although clinical practice for febrile infants in the second month of life is changing,⁷  both of the studied criteria require CSF results for application. Second, we were unable to apply the Boston and Philadelphia criteria to the entire study cohort because of missing data. However, the IBI rates were similar between those we could and could not classify. Third, we were unable to determine infants’ clinical appearance, comorbidities, or indication for undergoing evaluation for IBI. We did, however, exclude infants who were admitted to the ICU as a proxy measure for critical illness. In a large prospective cohort of febrile infants, neither unstructured clinical appearance nor structured Yale Observation Scale score is a strong predictor of IBI.⁹  Fourth, we could not determine whether each infant had a fever at home or later in the ED course. Therefore, we may have inadvertently applied the criteria to some infants for whom these tools were not derived. However, we used the physician’s decision to send CSF and blood cultures as a proxy for concern for IBI. Fifth, we had to modify the criteria because some of the included predictors (ie, urine Gram-stain and peripheral bands) were not routinely performed and urine dipstick without microscopy was performed for some infants, and they may have performed better if these laboratories results could have been included.⁵⁵  Sixth, despite our large sample size, bacterial meningitis was uncommon, limiting our confidence in identifying this specific IBI. Finally, these criteria were developed to identify infants who could be safely managed as outpatients. Although our analysis focused on missed cases of IBI, it is not possible to determine if infants with IBI who were misclassified as low risk could have been safely managed as outpatients with appropriate close clinical follow-up.

The Boston and Philadelphia criteria misclassified a substantial number of infants with bacterial infections, including approximately one-third of infants with bacteremia and bacterial meningitis. Better risk-stratification tools, likely including novel biomarkers, are needed to enable the rapid and accurate identification of infants at low risk of IBI who may be safely managed as outpatients.

We acknowledge the following study site investigators for their contributions in data collection: Alesia H. Fleming, MD, MPH (School of Medicine, Emory University, Atlanta, GA); Rakesh D. Mistry, MD, MS (School of Medicine, University of Colorado, Aurora, CO); Joanna E. Thomson, MD, MPH (Cincinnati Children’s Hospital Medical Center, Cincinnati, OH); David Schnadower, MD, MPH (School of Medicine, Washington University in St Louis, St Louis, MO and Cincinnati Children’s Hospital Medical Center, Cincinnati, OH); Christopher M. Pruitt, MD (University of Alabama at Birmingham, Birmingham, AL and Medical University of South Carolina, Charleston, SC); Paul L. Aronson, MD, MHS (School of Medicine, Yale University, New Haven, CT); Sarah J. Curtis, MD, MSci (Faculty of Medicine and Dentistry, University of Alberta, Edmonton, Alberta, Canada); Paul T. Ishimine, MD (School of Medicine, University of California, San Diego, San Diego, CA); Suzanne M. Schmidt, MD (Feinberg School of Medicine, Northwestern University, Chicago, IL); Stuart A Bradin, DO (Medical School, University of Michigan, Ann Arbor, MI); Kendra L. Grether-Jones, MD (School of Medicine, University of California, Davis, Sacramento, CA); Aaron S. Miller, MD, MSPH (School of Medicine, Saint Louis University, St Louis, MO); Jeffrey Louie, MD (University of Minnesota Masonic Children’s Hospital, Minneapolis, MN); and Samir S. Shah, MD, MSCE (Cincinnati Children’s Hospital Medical Center, Cincinnati, OH).

CBC

complete blood count

CI

confidence interval

CSF

cerebrospinal fluid

ED

emergency department

GBS

group B Streptococcus

hpf

high-power field

IBI

invasive bacterial infection

IQR

interquartile range

SBI

serious bacterial infection

UTI

urinary tract infection

WBC

white blood cell count