BACKGROUND:

Improving the diagnosis of serious bacterial infections (SBIs) in the children’s emergency department is a clinical priority. Early recognition reduces morbidity and mortality, and supporting clinicians in ruling out SBIs may limit unnecessary admissions and antibiotic use.

METHODS:

A prospective, diagnostic accuracy study of clinical and biomarker variables in the diagnosis of SBIs (pneumonia or other SBI) in febrile children <16 years old. A diagnostic model was derived by using multinomial logistic regression and internally validated. External validation of a published model was undertaken, followed by model updating and extension by the inclusion of procalcitonin and resistin.

RESULTS:

There were 1101 children studied, of whom 264 had an SBI. A diagnostic model discriminated well between pneumonia and no SBI (concordance statistic 0.84, 95% confidence interval 0.78–0.90) and between other SBIs and no SBI (0.77, 95% confidence interval 0.71–0.83) on internal validation. A published model discriminated well on external validation. Model updating yielded good calibration with good performance at both high-risk (positive likelihood ratios: 6.46 and 5.13 for pneumonia and other SBI, respectively) and low-risk (negative likelihood ratios: 0.16 and 0.13, respectively) thresholds. Extending the model with procalcitonin and resistin yielded improvements in discrimination.

CONCLUSIONS:

Diagnostic models discriminated well between pneumonia, other SBIs, and no SBI in febrile children in the emergency department. Improvements in the classification of nonevents have the potential to reduce unnecessary hospital admissions and improve antibiotic prescribing. The benefits of this improved risk prediction should be further evaluated in robust impact studies.

What’s Known on This Subject:

Failure to identify serious infections in children results in adverse outcomes and a failure to rule out serious infections results in unnecessary antibiotic use and hospital admissions. Multivariable clinical-risk prediction models appear to discriminate well between serious and self-limiting infections.

What This Study Adds:

In a study of 1101 children of all ages, risk prediction models discriminated well between pneumonia, other serious bacterial infections, and none. A published model performed well on external validation, and model extension with procalcitonin and resistin improved discrimination.

Acute febrile illness is among the most common of all presentations to the children’s emergency department (ED).1 In this context, the probability of serious bacterial infections (SBIs) is ∼7%, predominantly lower than respiratory or urinary tract infection.2 

The prompt recognition of SBI is fundamental to effective management. Children with meningococcal disease are frequently missed at initial presentation,3 and delayed recognition increases mortality.4,5 Although rates of invasive infection have declined with the introduction of conjugate vaccines,6,8 SBIs remain an important contributor to childhood morbidity and mortality.9 

In the United Kingdom, as rates of invasive infections have declined, the number of children who are admitted to the hospital has increased.10 The greatest increase is in young children with uncomplicated admissions for acute infections.11 Supporting clinicians in ruling out SBIs may reduce unnecessary hospital admissions in children.12 

A number of studies have reported the diagnostic accuracy of clinical13 and laboratory14 variables in febrile children. More recently, risk prediction models that combine clinical variables have been evaluated,2,15 and in one, the addition of the C-reactive protein (CRP) improved diagnostic accuracy.16 We have previously reported the combined performance of procalcitonin (PCT), resistin, and neutrophil gelatinase-associated lipocalin (NGAL) in Malawian children.17 

Diagnostic accuracy studies of febrile children have so far failed to impact clinical practice. Restrictive inclusion criteria (such as age, temperature, or clinical syndrome18) have limited their external validity, and few have progressed to validation in external populations. We therefore set out to derive and internally validate a multivariable risk prediction model and to externally validate a previously published model16 for the diagnosis of SBIs in febrile children of all ages.

This was a prospective, diagnostic accuracy study of clinical and biomarker variables in the diagnosis of SBIs in children who presented to the Alder Hey Children’s Hospital ED. This is the busiest children’s ED in the United Kingdom, and it manages 60 000 attendances each year. Recruitment was undertaken between November 2010 and April 2012. The study is reported in line with the Standards for Reporting of Diagnostic Accuracy and Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis guidelines.19,20 

Children <16 years of age with fever (>38°C) or a history of fever were eligible if they required blood tests as part of their clinical management. Children with primary immunodeficiency were excluded. By using previous estimates of sensitivity (65%) and specificity (90%) and a rate of SBI of 15%, a sample size of 2300 was proposed. For skin and soft tissue infections, the reference standard for an SBI was that children were deemed by the clinical team to require intravenous antibiotics. Because the outcome diagnosis was solely based on a clinical decision (and because this was true of all such cases), these children (n = 82) were excluded (Fig 1).

FIGURE 1

Flow diagram of the study. The exclusion of children with a clinical reference standard is explained in the text. PID, primary immunodeficiency.

FIGURE 1

Flow diagram of the study. The exclusion of children with a clinical reference standard is explained in the text. PID, primary immunodeficiency.

The Generation R Young Persons’ Advisory Group (www.generationr.org.uk), initiated by the National Institute for Health Research, helped design patient information leaflets for young people and families. In the course of the study, the group explored improvements in the recognition of serious infections and they discussed diagnostic tests by using various samples (such as saliva or blood). This involvement has informed the design of subsequent studies.

Relevant clinical and biomarker variables were identified from the literature, including 2 large systematic reviews.13,14 Clinical data were entered onto a proforma at the time of the clinical assessment. When possible, this was done by the attending clinician. When the proforma was incomplete, missing clinical information was retrieved from the clinical notes when explicitly referenced. Paper proformas were collected daily by the study team. All proformas were cross-checked against the clinical notes, which were electronically scanned and stored. Missing or ambiguous data were recorded as missing. Data collection and entry into the database was blinded to the final outcomes.

The tests performed on study subjects are recorded in Supplemental Table 4. All samples were processed in Clinical Pathology Accredited laboratories. Blood (0.5–1 mL) inoculated into culture bottles was monitored by using the BacT/ALERT 3D system (BioMerieux, Marcy l'Etoile, France) . Positive cultures were processed in line with UK standards for microbiology investigations developed by Public Health England.21 Specific Streptococcus pneumoniae and Neisseria meningitidis polymerase chain reaction (PCR) assays were performed at the Meningococcal Reference Unit in Manchester, United Kingdom.22,23 Urine and cerebrospinal fluid (CSF) underwent microscopy and culture on agar gel plates and were processed in line with UK standards. Multiplex PCR was performed on respiratory (respiratory syncytial virus, influenza A and B, parainfluenza 1-3, adenovirus, rhinovirus, and human metapneumovirus) and CSF (herpes simplex virus 1 and 2, varicella-zoster, and enterovirus) samples at the regional laboratory in Manchester. Starting in April 2011, respiratory PCRs were performed by using the FilmArray respiratory viral panel (BioMerieux; Marcy l'Etoile, France) and additionally identified parainfluenza 4, rhinovirus, enterovirus, and coronavirus 1-4.

Blood (0.5–1 mL) was collected into lithium heparin and plasma stored in Sarstedt microtubes (Sarstedt AG & Co; Nümbrecht, Germany) at −80°C within 1 hour. Before analysis samples were thawed, vortex was mixed and centrifuged to remove bubbles and particulate matter. PCT analysis was undertaken on the B.R.A.H.M.S. Kryptor (Thermo Fisher Scientific Inc; Raleigh, NC) according to the manufacturer’s instructions. Quality-control samples were analyzed with each run. NGAL and resistin were analyzed by using validated commercial enzyme-linked immunosorbent assay.

In common with other published studies, outcome diagnoses were determined by a composite reference standard that incorporated clinical, microbiological, and radiologic features (Supplemental Table 5).14,15,24,25 By using these predefined criteria, a pediatric research fellow and a pediatric infectious-disease consultant independently attributed the outcome diagnosis. In the case of disagreement, a second pediatric infectious-disease consultant determined the final outcome. Children who failed to meet the predefined criteria for SBIs were considered to have no SBI. Subjects were followed up for 28 days to reduce misclassification.

Analysis was undertaken in R, version 3.0.1 (R Foundation for Statistical Computing, Vienna, Austria).26 Missing data were handled by 10-fold multiple imputation by using fully conditional specification implemented by the MICE package.27 In this method, missing values are replaced by values that are drawn from a conditional distribution that is specific to each predictor variable and is defined by its own imputation model. Data were assumed to be missing at random. The proportion of missing data relating to each clinical variable is recorded in Supplemental Table 6.

The data set was randomized into a split-sample derivation and validation set. Univariate analysis of clinical and biomarker variables was undertaken by using logistic regression for the outcomes of SBIs. Explanatory variables were examined for evidence of collinearity. Scatter plots and generalized additive model plots28 fitted by using the gam() function in the mgcv package were examined for evidence of nonlinearity on the log-odds scale.29 Piecewise and polynomial transformations were undertaken when appropriate. Plausible interaction terms were explored, including interactions between age, heart rate, and respiratory rate. A multivariable model was derived by using a forward stepwise method. Improvements in model fit were tested by means of a likelihood ratio test (α = .05), and variables associated with a significant improvement were retained. Having identified a parsimonious model for SBIs, these variables were then included in a multinomial regression model for the categorical outcomes of pneumonia, other SBI, and no SBI.

External validation of the model published by Nijman et al16 was undertaken by using the published coefficients. A comparison of study participants is given in Supplemental Table 7. The model was updated by refitting variables and estimating the individual coefficients then extended by the inclusion of PCT and resistin. This strategy preserved the original model structure and avoided deriving an entirely new model. The biomarkers were chosen by having observed their values in our earlier model derivation. Additional clinical variables were not investigated because they appeared less predictive in our model derivation, and plausible clinical variables were adequately represented by the published model.

Performance characteristics of the fitted models at various risk thresholds were estimated by using the epiR package.30 Discrimination was measured by using the concordance statistic (c statistic) and illustrated by receiver operating characteristic curves using the pROC package.31 The c statistic estimates the probability that a randomly selected subject with the outcome of interest has a higher predicted probability than a randomly selected subject without it. Comparison of the c statistic was undertaken by using the DeLong method.32 For the multinomial regression model, the c statistic estimated discrimination between pairs of patients (a patient with pneumonia and a patient with no SBI, or a patient with other SBI and a patient with no SBI). The 95% confidence intervals (CIs) were estimated with a bootstrapping process by using 2000 bootstrap replicates. Calibration of the models (how closely risk predictions fit observed cases) was illustrated by using multinomial calibration plots.33 

In the absence of established methods to report classification in multinomial risk-prediction models, we compared crude classification (that is, the most likely diagnosis predicted by the multinomial models) in the updated model with the extended model. To investigate potential clinical utility, we estimated the ability of the models to rule out (predictions for both categories of SBIs <5%) or rule in (prediction of either category >20%) SBIs. These thresholds represent approximately half and double, respectively, the observed event rate in the study population.

Approval for the study was granted by the Greater Manchester West Research Ethics Committee (10/H1014/53) and by the Alder Hey Children’s Hospital R&D department.

Between November 1, 2010 and April 3, 2012, 7949 children presented to the Alder Hey Children’s ED with fever. Of these, 1872 were eligible for inclusion, and 1101 were recruited to the study (Fig 1). The median age was 2.4 years (interquartile range 0.9–5.7 years), and 55% were boys. Approximately one-third of the children had significant comorbidities (Table 1). Two hundred and sixty-four children (24.0%) were diagnosed with SBIs (Supplemental Fig 5).

TABLE 1

Characteristics of Study Subjects

Overall (N = 1101)Pneumonia (n = 108)Other SBI (n = 156)No SBI (n = 837)
MedianIQRMedianIQRMedianIQRMedianIQR
Demographics         
 Age 2.39 0.88–5.73 3.51* 1.60–6.29 2.28 0.43–7.54 2.21 0.92–5.35 
  Proportion 95% CI Proportion 95% CI Median IQR Proportion 95% CI 
 Boy  0.55 0.52 to 0.58 0.48 0.39 to 0.57 0.59 0.51–0.66 0.56 0.52 to 0.59 
 PMH 0.31 0.28 to 0.34 0.47* 0.38 to 0.57 0.26 0.19–0.33 0.30 0.27 to 0.33 
         
Clinical variables Median IQR Median IQR Median IQR Median IQR 
 Temperature 37.8 37.0–38.6 37.9** 37.1–38.9 38.0** 37.2–38.8 37.7 36.9–38.6 
 Heart rate 140 121–166 147** 132–170 148** 122–175 139 120–163 
 Respiratory rate 30 24–38 38* 28–48 30 24–38 28 24–36 
         
Biomarkers Median IQR Median IQR Median IQR Median IQR 
 CRP, mg/L 19.6 5.8–54.0 49.0* 21.1–119 68.3* 28.9–137 14.3 4.0–36.5 
 WCC, ×109/L 11.5 7.9–15.8 11.8** 8.4–18.5 15.0* 10.9–20.5 10.8 7.7–14.7 
 Neutrophils, ×109/L 6.9 3.8–10.8 8.0* 4.8–13.4 10.0* 5.9–14.8 6.2 3.4–9.7 
 NGAL, ng/L 77.1 52.5–121 92.1* 65.9–162 120* 74.4–170 69.7 49.5–103 
 PCT, µg/L 0.23 0.10–0.83 0.49* 0.12–2.85 1.10* 0.15–5.85 0.18 0.09–0.53 
 Resistin, ng/L 40.3 21.1–73.4 67.3* 31.4–107 60.6* 29.7–113 35.7 19.8–64.3 
         
Outcomes Median IQR Median IQR Median IQR Median IQR 
 Length of stay, d 0–3 3* 2–6 4.5* 2–7 0–2 
 n (%) 95% CI n (%) 95% CI n (%) 95% CI n (%) 95% CI 
 Antibiotic use 855 (78) 75 to 80 108* (100) 96 to 100 156* (100) 97 to 100 509 (61) 57 to 64 
 Hospital admission 844 (77) 74 to 79 102* (94) 88 to 98 148* (95) 90 to 98 516 (62) 58 to 65 
 PICU 19 (1.73) 1.11 to 2.68 5** (4.63) 2.00 to 10.4 10** (6.41) 3.52 to 11.4 4 (0.48) 0.19 to 1.22 
 Mortality 1 (0.09) 0.01 to 0.51 0 to 3.40 1 (0.65) 0.12 to 3.55 0 to 0.46 
Overall (N = 1101)Pneumonia (n = 108)Other SBI (n = 156)No SBI (n = 837)
MedianIQRMedianIQRMedianIQRMedianIQR
Demographics         
 Age 2.39 0.88–5.73 3.51* 1.60–6.29 2.28 0.43–7.54 2.21 0.92–5.35 
  Proportion 95% CI Proportion 95% CI Median IQR Proportion 95% CI 
 Boy  0.55 0.52 to 0.58 0.48 0.39 to 0.57 0.59 0.51–0.66 0.56 0.52 to 0.59 
 PMH 0.31 0.28 to 0.34 0.47* 0.38 to 0.57 0.26 0.19–0.33 0.30 0.27 to 0.33 
         
Clinical variables Median IQR Median IQR Median IQR Median IQR 
 Temperature 37.8 37.0–38.6 37.9** 37.1–38.9 38.0** 37.2–38.8 37.7 36.9–38.6 
 Heart rate 140 121–166 147** 132–170 148** 122–175 139 120–163 
 Respiratory rate 30 24–38 38* 28–48 30 24–38 28 24–36 
         
Biomarkers Median IQR Median IQR Median IQR Median IQR 
 CRP, mg/L 19.6 5.8–54.0 49.0* 21.1–119 68.3* 28.9–137 14.3 4.0–36.5 
 WCC, ×109/L 11.5 7.9–15.8 11.8** 8.4–18.5 15.0* 10.9–20.5 10.8 7.7–14.7 
 Neutrophils, ×109/L 6.9 3.8–10.8 8.0* 4.8–13.4 10.0* 5.9–14.8 6.2 3.4–9.7 
 NGAL, ng/L 77.1 52.5–121 92.1* 65.9–162 120* 74.4–170 69.7 49.5–103 
 PCT, µg/L 0.23 0.10–0.83 0.49* 0.12–2.85 1.10* 0.15–5.85 0.18 0.09–0.53 
 Resistin, ng/L 40.3 21.1–73.4 67.3* 31.4–107 60.6* 29.7–113 35.7 19.8–64.3 
         
Outcomes Median IQR Median IQR Median IQR Median IQR 
 Length of stay, d 0–3 3* 2–6 4.5* 2–7 0–2 
 n (%) 95% CI n (%) 95% CI n (%) 95% CI n (%) 95% CI 
 Antibiotic use 855 (78) 75 to 80 108* (100) 96 to 100 156* (100) 97 to 100 509 (61) 57 to 64 
 Hospital admission 844 (77) 74 to 79 102* (94) 88 to 98 148* (95) 90 to 98 516 (62) 58 to 65 
 PICU 19 (1.73) 1.11 to 2.68 5** (4.63) 2.00 to 10.4 10** (6.41) 3.52 to 11.4 4 (0.48) 0.19 to 1.22 
 Mortality 1 (0.09) 0.01 to 0.51 0 to 3.40 1 (0.65) 0.12 to 3.55 0 to 0.46 

Statistical comparisons between pneumonia or other SBI and no SBI. Continuous data were compared by means of the Kruskal-Wallis test, and proportions were compared by means of the Pearson’s χ2 statistic. Rare events (such as admission to the PICU or death) were compared by means of a Monte Carlo simulation. IQR, interquartile range; PMH, past medical history; WCC, white blood cell count.

*

P < .001;

**

P < .05.

The probability of pneumonia and other SBIs increased linearly with heart rate, respiratory rate, and temperature. Consistent with other studies, increased work of breathing (odds ratio [OR] 10.4, 95% CI 6.69 to 16.2), hypoxia (9.29, 95% CI 5.35 to 16.1), and other respiratory variables were significantly associated with pneumonia. These features reduced the probability of other SBIs. Neck stiffness, a bulging fontanelle, irritability, and dysuria were associated with other SBIs. Prolonged capillary refill time was associated with other SBIs (1.43, 95% CI 1.05 to 1.97) but not pneumonia, whereas the presence of a rash reduced the probability of both pneumonia and other SBIs. Univariate ORs are presented in Supplemental Fig 6. CRP, PCT, NGAL, and resistin were all associated with SBIs (Supplemental Table 8).

The derived model included the variables respiratory rate and normal air entry alongside CRP, PCT, and resistin (Supplemental Table 9). Fitting CRP as a piecewise term improved the model fit. The model discriminated well on internal validation (c statistic 0.84, 95% CI 0.78 to 0.90 for pneumonia; and 0.77, 95% CI 0.71 to 0.83 for other SBIs). Calibration plots suggested that the model overestimated the risk of pneumonia (Fig 2).

FIGURE 2

Parametric nominal calibration plot of predicted risks and observed outcomes in the validation set.

FIGURE 2

Parametric nominal calibration plot of predicted risks and observed outcomes in the validation set.

The published model of Nijman et al16,34 was validated in the complete data set (n = 1101). By using the published coefficients, the model discriminated well between pneumonia and no SBI although not as well between other SBIs and no SBI (c statistic 0.85 and 0.76, respectively, Supplemental Fig 7). Model calibration was poor, although calibration plots indicated that predicted risks and observed outcomes were highly correlated (Fig 3).

FIGURE 3

Parametric nominal calibration plot of the original Nijman model on external validation.

FIGURE 3

Parametric nominal calibration plot of the original Nijman model on external validation.

By observing the correlation between predicted probabilities and observed outcomes in the poorly calibrated model, we updated the model by re-estimating the individual coefficients. No attempt was made to adjust the functional form of the predictor variables. The refitted model discriminated well (c statistic 0.88 and 0.82 for pneumonia and other SBIs, respectively) and was well calibrated (Fig 4). The model was then extended by the inclusion of PCT and resistin. This improved discrimination of pneumonia (c statistic increased from 0.88 to 0.90, P = .03) and other SBI models (from 0.82 to 0.84, P = .03), and calibration remained good (Supplemental Figure 8).

FIGURE 4

Parametric nominal calibration plot of the Nijman model with coefficients refitted to the validation data set.

FIGURE 4

Parametric nominal calibration plot of the Nijman model with coefficients refitted to the validation data set.

The performance characteristics of the updated and extended models are summarized in Table 2. At a low-risk threshold of 5%, the extended pneumonia model had a sensitivity of 92% (95% CI 85% to 96%) and a negative likelihood ratio of 0.12 (0.06 to 0.23). For other SBIs, model sensitivity was 92% (86% to 95%), and negative likelihood ratio was 0.21 (0.12 to 0.35). At a high-risk threshold (>20%), specificity was 89% (95% CI 87% to 91%) for pneumonia, with a positive likelihood ratio of 6.69 (5.30 to 8.44) and 86% (83% to 88%) and a positive likelihood ratio of 4.96 (4.07 to 6.03) for other SBIs.

TABLE 2

Performance Characteristics of the Updated (Top) and Extended Nijman Models Including the Biomarkers PCT and Resistin (Bottom) at the stated risk thresholds.

Sensitivity95% CISpecificity95% CIPPV95% CINPV95% CIPLR95% CINLR95% CI
Updated Nijman Model 
 Pneumonia, % 
  2.5 0.93 (0.86 to 0.97) 0.51 (0.47 to 0.55) 0.20 (0.16 to 0.24) 0.98 (0.96 to 0.99) 1.91 (1.75 to 2.08) 0.14 (0.07 to 0.28) 
  5 0.89 (0.81 to 0.94) 0.70 (0.67 to 0.73) 0.28 (0.23 to 0.33) 0.98 (0.97 to 0.99) 2.98 (2.63 to 3.37) 0.16 (0.09 to 0.27) 
  10 0.81 (0.79 to 0.88) 0.82 (0.79 to 0.84) 0.36 (0.30 to 0.43) 0.97 (0.95 to 0.98) 4.41 (3.72 to 5.23) 0.24 (0.16 to 0.35) 
  20 0.69 (0.60 to 0.78) 0.89 (0.87 to 0.91) 0.45 (0.38 to 0.53) 0.96 (0.94 to 0.97) 6.46 (5.12 to 8.14) 0.34 (0.26 to 0.46) 
  30 0.60 (0.92 to 0.96) 0.94 (0.92 to 0.96) 0.58 (0.48 to 0.67) 0.95 (0.93 to 0.96) 10.5 (7.66 to 14.4) 0.42 (0.33 to 0.53) 
 Other SBI, % 
  2.5 0.99 (0.96 to 1.0) 0.09 (0.07 to 0.11) 0.17 (0.15 to 0.19) 0.99 (0.93 to 1.00) 1.09 (1.06 to 1.12) 0.07 (0.01 to 0.50) 
  5 0.97 (0.93 to 0.99) 0.24 (0.21 to 0.27) 0.19 (0.17 to 0.22) 0.98 (0.95 to 0.99) 1.28 (1.22 to 1.34) 0.13 (0.06 to 0.31) 
  10 0.83 (0.77 to 0.89) 0.58 (0.55 to 0.62) 0.27 (0.23 to 0.31) 0.95 (0.93 to 0.97) 1.99 (1.79 to 2.21) 0.29 (0.20 to 0.41) 
  20 0.56 (0.48 to 0.64) 0.89 (0.87 to 0.91) 0.49 (0.41 to 0.56) 0.92 (0.90 to 0.93) 5.13 (4.04 to 6.50) 0.49 (0.41 to 0.59) 
  30 0.40 (0.32 to 0.48) 0.95 (0.94 to 0.97) 0.61 (0.51 to 0.70) 0.89 (0.87 to 0.91) 8.31 (5.80 to 11.9) 0.63 (0.56 to 0.72) 
Extended Nijman Model (including PCT and resistin) 
 Pneumonia, % 
  2.5 0.94 (0.87 to 0.97) 0.52 (0.49 to 0.56) 0.20 (0.17 to 0.24) 0.98 (0.97 to 0.99) 1.96 (1.79 to 2.13) 0.12 (0.06 to 0.25) 
  5 0.92 (0.85 to 0.96) 0.69 (0.66 to 0.72) 0.28 (0.23 to 0.33) 0.98 (0.97 to 0.99) 2.96 (2.64 to 3.33) 0.12 (0.06 to 0.23) 
  10 0.85 (0.77 to 0.91) 0.82 (0.79 to 0.84) 0.38 (0.31 to 0.44) 0.98 (0.96 to 0.99) 4.66 (3.96 to 5.49) 0.18 (0.12 to 0.29) 
  20 0.70 (0.61 to 0.79) 0.89 (0.87 to 0.91) 0.46 (0.39 to 0.54) 0.96 (0.94 to 0.97) 6.69 (5.30 to 8.44) 0.33 (0.25 to 0.44) 
  30 0.62 (0.52 to 0.71) 0.94 (0.92 to 0.95) 0.56 (0.47 to 0.65) 0.95 (0.93 to 0.96) 9.99 (7.38 to 13.5) 0.4 (0.32 to 0.52) 
 Other SBI, % 
  2.5 0.97 (0.94 to 0.99) 0.18 (0.15 to 0.20) 0.18 (0.16 to 0.21) 0.97 (0.93 to 0.99) 1.18 (1.14 to 1.23) 0.15 (0.05 to 0.39) 
  5 0.92 (0.86 to 0.95) 0.40 (0.37 to 0.44) 0.22 (0.19 to 0.26) 0.96 (0.94 to 0.98) 1.54 (1.43 to 1.65) 0.21 (0.12 to 0.35) 
  10 0.85 (0.79 to 0.90) 0.61 (0.58 to 0.65) 0.29 (0.25 to 0.34) 0.96 (0.94 to 0.97) 2.21 (1.98 to 2.46) 0.24 (0.16 to 0.35) 
  20 0.70 (0.62 to 0.77) 0.86 (0.83 to 0.88) 0.48 (0.41 to 0.55) 0.94 (0.92 to 0.95) 4.96 (4.07 to 6.03) 0.35 (0.28 to 0.45) 
  30 0.53 (0.45 to 0.61) 0.94 (0.92 to 0.95) 0.61 (0.52 to 0.69) 0.91 (0.89 to 0.93) 8.40 (6.23 to 11.3) 0.50 (0.42 to 0.59) 
Sensitivity95% CISpecificity95% CIPPV95% CINPV95% CIPLR95% CINLR95% CI
Updated Nijman Model 
 Pneumonia, % 
  2.5 0.93 (0.86 to 0.97) 0.51 (0.47 to 0.55) 0.20 (0.16 to 0.24) 0.98 (0.96 to 0.99) 1.91 (1.75 to 2.08) 0.14 (0.07 to 0.28) 
  5 0.89 (0.81 to 0.94) 0.70 (0.67 to 0.73) 0.28 (0.23 to 0.33) 0.98 (0.97 to 0.99) 2.98 (2.63 to 3.37) 0.16 (0.09 to 0.27) 
  10 0.81 (0.79 to 0.88) 0.82 (0.79 to 0.84) 0.36 (0.30 to 0.43) 0.97 (0.95 to 0.98) 4.41 (3.72 to 5.23) 0.24 (0.16 to 0.35) 
  20 0.69 (0.60 to 0.78) 0.89 (0.87 to 0.91) 0.45 (0.38 to 0.53) 0.96 (0.94 to 0.97) 6.46 (5.12 to 8.14) 0.34 (0.26 to 0.46) 
  30 0.60 (0.92 to 0.96) 0.94 (0.92 to 0.96) 0.58 (0.48 to 0.67) 0.95 (0.93 to 0.96) 10.5 (7.66 to 14.4) 0.42 (0.33 to 0.53) 
 Other SBI, % 
  2.5 0.99 (0.96 to 1.0) 0.09 (0.07 to 0.11) 0.17 (0.15 to 0.19) 0.99 (0.93 to 1.00) 1.09 (1.06 to 1.12) 0.07 (0.01 to 0.50) 
  5 0.97 (0.93 to 0.99) 0.24 (0.21 to 0.27) 0.19 (0.17 to 0.22) 0.98 (0.95 to 0.99) 1.28 (1.22 to 1.34) 0.13 (0.06 to 0.31) 
  10 0.83 (0.77 to 0.89) 0.58 (0.55 to 0.62) 0.27 (0.23 to 0.31) 0.95 (0.93 to 0.97) 1.99 (1.79 to 2.21) 0.29 (0.20 to 0.41) 
  20 0.56 (0.48 to 0.64) 0.89 (0.87 to 0.91) 0.49 (0.41 to 0.56) 0.92 (0.90 to 0.93) 5.13 (4.04 to 6.50) 0.49 (0.41 to 0.59) 
  30 0.40 (0.32 to 0.48) 0.95 (0.94 to 0.97) 0.61 (0.51 to 0.70) 0.89 (0.87 to 0.91) 8.31 (5.80 to 11.9) 0.63 (0.56 to 0.72) 
Extended Nijman Model (including PCT and resistin) 
 Pneumonia, % 
  2.5 0.94 (0.87 to 0.97) 0.52 (0.49 to 0.56) 0.20 (0.17 to 0.24) 0.98 (0.97 to 0.99) 1.96 (1.79 to 2.13) 0.12 (0.06 to 0.25) 
  5 0.92 (0.85 to 0.96) 0.69 (0.66 to 0.72) 0.28 (0.23 to 0.33) 0.98 (0.97 to 0.99) 2.96 (2.64 to 3.33) 0.12 (0.06 to 0.23) 
  10 0.85 (0.77 to 0.91) 0.82 (0.79 to 0.84) 0.38 (0.31 to 0.44) 0.98 (0.96 to 0.99) 4.66 (3.96 to 5.49) 0.18 (0.12 to 0.29) 
  20 0.70 (0.61 to 0.79) 0.89 (0.87 to 0.91) 0.46 (0.39 to 0.54) 0.96 (0.94 to 0.97) 6.69 (5.30 to 8.44) 0.33 (0.25 to 0.44) 
  30 0.62 (0.52 to 0.71) 0.94 (0.92 to 0.95) 0.56 (0.47 to 0.65) 0.95 (0.93 to 0.96) 9.99 (7.38 to 13.5) 0.4 (0.32 to 0.52) 
 Other SBI, % 
  2.5 0.97 (0.94 to 0.99) 0.18 (0.15 to 0.20) 0.18 (0.16 to 0.21) 0.97 (0.93 to 0.99) 1.18 (1.14 to 1.23) 0.15 (0.05 to 0.39) 
  5 0.92 (0.86 to 0.95) 0.40 (0.37 to 0.44) 0.22 (0.19 to 0.26) 0.96 (0.94 to 0.98) 1.54 (1.43 to 1.65) 0.21 (0.12 to 0.35) 
  10 0.85 (0.79 to 0.90) 0.61 (0.58 to 0.65) 0.29 (0.25 to 0.34) 0.96 (0.94 to 0.97) 2.21 (1.98 to 2.46) 0.24 (0.16 to 0.35) 
  20 0.70 (0.62 to 0.77) 0.86 (0.83 to 0.88) 0.48 (0.41 to 0.55) 0.94 (0.92 to 0.95) 4.96 (4.07 to 6.03) 0.35 (0.28 to 0.45) 
  30 0.53 (0.45 to 0.61) 0.94 (0.92 to 0.95) 0.61 (0.52 to 0.69) 0.91 (0.89 to 0.93) 8.40 (6.23 to 11.3) 0.50 (0.42 to 0.59) 

NPV, negative predictive value; PPV, positive predictive value.

Classification (determined by likeliest outcome category) was similar between the updated and extended models (893 of 1101 vs 917 of 1101, 2.2% improvement, 95% CI −1.1% to 5.4%, Supplemental Table 10). By using the extended model, SBI was correctly ruled out in 31 additional children (3.7%, 95% CI −1.0% to 8.4%), and there were 5 fewer potentially missed SBI diagnoses (14 of 264 vs 19 of 264, 1.8% reduction, 95% CI −2.6% to 6.4%, Table 3).

TABLE 3

Outcomes According to Risk Classification for the Updated and Extended Models

Outcome CategoryUpdatedExtendedn
ROIMRIROIMRI
PneuOtherPneuOther
No SBI 269 355 76 137 300 352 74 111 837 
Pneumonia 19 70 13 16 71 16 108 
Other SBI 13 29 107 33 107 156 
Total 288 403 153 257 314 401 152 234 1101 
Outcome CategoryUpdatedExtendedn
ROIMRIROIMRI
PneuOtherPneuOther
No SBI 269 355 76 137 300 352 74 111 837 
Pneumonia 19 70 13 16 71 16 108 
Other SBI 13 29 107 33 107 156 
Total 288 403 153 257 314 401 152 234 1101 

SBI was considered ruled out if the predicted probabilities of both pneumonia and other SBI were <5%, whereas SBI was considered ruled in if the probability of either outcome was >20%. All other subjects were considered to be at intermediate risk. IM, intermediate; Pneu, pneumonia; RI, rule in; RO, rule out.

In this large, prospective study of febrile children of all ages who presented to the ED, multinomial risk-prediction models discriminated well between pneumonia, other SBIs, and no SBIs. A newly derived model performed well on internal validation and identified PCT, resistin, and CRP as biomarkers of potential value. A published model performed well on external validation, and the addition of PCT and resistin improved discrimination. At a low-risk threshold (<5%), an NLR of 0.12 (pneumonia) or 0.21 (other SBIs) may help to rule out SBIs, whereas at a high-risk threshold (>20%), PLRs of 6.69 and 4.96, respectively, may expedite treatment.

Strengths

We present data on multiple biomarkers of SBIs in >1000 children. We have evaluated children irrespective of age, past medical history, and clinical syndrome and obtained comparable discrimination to other studies with more restrictive inclusion criteria. In common with other recent data,2,16 we have demonstrated the value of combining clinical and biomarker variables.

This is the first broad, external validation of the published multivariable model by Nijman et al.16 The model discriminated well but was poorly calibrated. Specifically, there was a problem with calibration overall: the model predicted too few cases in our population. However, correlation between model predictions and observed cases suggested the overall structure of the model was appropriate to our data set, and our approach of re-estimating the model coefficients resulted in a well-calibrated model.

Limitations

This is a single-center study, and although we have performed internal validation of our derived model, external validity would require demonstration in an alternative setting. We have grouped other SBI into a single outcome category. It would be preferable to model outcomes such as septicemia and meningitis separately, but the infrequency of these outcomes makes this challenging. A pragmatic response is to advocate additional diagnostic testing (including urgent urine or CSF microscopy) of children who are considered at high risk for other SBIs.

Diagnostic studies with imperfect reference standards require a pragmatic approach to determine outcomes. An established approach to this is to use predefined, composite reference standards, as we have done. The universal application of respiratory viral assays may have yielded additional evidence on which to base classification, but such testing was undertaken at the discretion of the clinical team and not applied systematically. Our use of a radiologic diagnosis of pneumonia despite its limitations is common in this setting.15,34 We included a category of probable SBI to account for the lack of sensitivity of conventional diagnostic testing in children. This category accounted for only a small number of cases (8) and was defined in advance. By establishing clear criteria for each outcome diagnosis, we have sought to minimize verification bias.

We studied children who were already considered at risk for SBIs and in whom the clinical team had initiated additional investigation. This unmeasured risk evaluation limits the external validity of our findings. The proportion of SBIs (24%) is significantly higher than that which is observed in all febrile children in the ED, and we agree with other authors who have stressed the importance of diagnostics research in low-risk populations (such as all children attending the ED or primary care).18 Of our sample, ∼80% were admitted to the hospital and received antibiotics, including 60% of those who did not have SBIs. Decision-making on the basis of a low-risk threshold of 5% may reduce admissions and antibiotic use, but it does not (by definition) eliminate risk. Clinicians would need to combine risk evaluation with appropriate safety-netting.

Our finding that clinical variables such as hypoxia, abnormal respiratory findings, irritability, and dehydration increase the probability of an SBI is consistent with similar studies.2,13,16 We failed to demonstrate the value of more subjective assessments such as ill appearance and parental concern, although there was a significant problem of missing data for each.

Our results support a growing body of research suggesting that risk prediction models improve the identification of SBIs in the children’s ED. Such models have yet to translate into improved clinical decision-making. Two recent impact studies challenged the assumption that accurate risk prediction will necessarily improve decision-making. In the first, the use of the laboratory score (a decision rule that combines CRP, PCT, and urinalysis) failed to reduce antibiotic prescriptions in children in the ED.35 A second evaluated the use of the Nijman risk prediction model to guide decisions, and no impact on antibiotic prescribing or hospital admission was observed.36 

Future impact studies need to evaluate the behaviors associated with decision-making. This has been of considerable importance in evaluating interventions to rationalize antibiotic prescribing.37 To translate estimates of risk into safe clinical decisions and improve the management of children in the ED, it will be necessary to involve clinicians and families. The risk thresholds we have proposed are not yet established in the context of SBIs in the children’s ED, and more work is necessary to determine if they (and the clinical decisions they guide) are appropriate.

A diagnostic model that combined clinical and biomarker variables discriminated well between pneumonia, other SBIs, and no SBI in febrile children of all ages in the ED. External validation of a previously derived risk model yielded encouraging diagnostic accuracy and was improved by the addition of PCT and resistin. Future work should establish the value of decision rules based on risk prediction models in robust impact studies. Such studies must address the complex behaviors that are associated with clinical decisions to yield clinical benefit.

     
  • c statistic

    concordance statistic

  •  
  • CI

    confidence interval

  •  
  • CRP

    C-reactive protein

  •  
  • CSF

    cerebrospinal fluid

  •  
  • ED

    emergency department

  •  
  • NGAL

    neutrophil gelatinase-associated lipocalin

  •  
  • OR

    odds ratio

  •  
  • PCR

    polymerase chain reaction

  •  
  • PCT

    procalcitonin

  •  
  • SBI

    serious bacterial infection

Dr Irwin oversaw the running of the study, collected the data, determined outcome diagnoses, performed laboratory assays and statistical analysis, wrote the first draft of the manuscript, and revised the final manuscript; Ms Grant and Ms R. Williams supervised the collection of data and contributed to the writing of the manuscript; Dr Kolamunnage-Dona oversaw the running of the study, performed statistical analysis, and contributed to the writing of the manuscript; Dr Drew contributed to the study design and the writing of the manuscript; Dr Paulus determined outcome diagnoses, oversaw the running of the study, and contributed to the writing of the manuscript; Mr Jeffers and Ms Chesters performed laboratory assays, acquired and interpreted data, and contributed to the writing of the manuscript; Ms K. Williams and Dr Marzouk designed and oversaw the study, collected the data, and contributed to the writing of the manuscript; Dr Breen and Ms Preston oversaw the running of the study and contributed to the writing of the manuscript; Dr Appelbe supported the study design and data acquisition and contributed to the writing of the manuscript; Dr Newland and Prof McNamara designed the study and contributed to the writing of the manuscript; Prof Diggle performed statistical analysis, contributed to the writing of the manuscript, and reviewed the final manuscript; Prof Carrol designed and oversaw the running of the study, determined outcome diagnoses, contributed to the writing of the manuscript, and reviewed the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the National Institute for Health Research’s Research for Innovation, Speculation, and Creativity Programme grant RC-PG-0309-10053 to Prof Carrol and the Alder Hey Charity funding grant 8037 to Dr Irwin and Prof Carrol. The funding organizations had no role in the design or undertaking of this study. This article presents independent research that is funded by the National Institute for Health Research’s Research for Innovation, Speculation, and Creativity Programme. The views expressed are those of the author(s) and are not necessarily those of the NHS Foundation Trust, the National Institute for Health Research, or the Department of Health.

We thank all the children and their parents for agreeing to participate, and we thank the significant number of ED and pediatric clinical and nursing staff who work in the Alder Hey Children’s ED for their substantial contributions to the study. In particular, we thank the children of the Medicines for Children Research Network Young Persons’ Advisory Group. We thank Prof Matthew Peak and Ms Dot Lambert for their logistical support from the Alder Hey Research Department. We thank Ms Elaine Hanmer, Ms Laura Medway, and Ms Gemma Boydell for their informatics support and Mrs Sarah Olsen for her administrative support in the running of the study.

The University of Liverpool acknowledges the support of the National Institute for Health Research through the Comprehensive Clinical Research Network.

1
Sands
R
,
Shanmugavadivel
D
,
Stephenson
T
,
Wood
D
.
Medical problems presenting to paediatric emergency departments: 10 years on.
Emerg Med J
.
2012
;
29
(
5
):
379
382
[PubMed]
2
Craig
JC
,
Williams
GJ
,
Jones
M
, et al
.
The accuracy of clinical symptoms and signs for the diagnosis of serious bacterial infection in young febrile children: prospective cohort study of 15 781 febrile illnesses.
BMJ
.
2010
;
340
:
c1594
[PubMed]
3
Thompson
MJ
,
Ninis
N
,
Perera
R
, et al
.
Clinical recognition of meningococcal disease in children and adolescents.
Lancet
.
2006
;
367
(
9508
):
397
403
[PubMed]
4
Launay
E
,
Gras-Le Guen
C
,
Martinot
A
, et al
.
Suboptimal care in the initial management of children who died from severe bacterial infection: a population-based confidential inquiry.
Pediatr Crit Care Med
.
2010
;
11
(
4
):
469
474
[PubMed]
5
Inwald
DP
,
Tasker
RC
,
Peters
MJ
,
Nadel
S
;
Paediatric Intensive Care Society Study Group (PICS-SG)
.
Emergency management of children with severe sepsis in the United Kingdom: the results of the Paediatric Intensive Care Society sepsis audit.
Arch Dis Child
.
2009
;
94
(
5
):
348
353
[PubMed]
6
Koshy
E
,
Murray
J
,
Bottle
A
,
Sharland
M
,
Saxena
S
.
Impact of the seven-valent pneumococcal conjugate vaccination (PCV7) programme on childhood hospital admissions for bacterial pneumonia and empyema in England: national time-trends study, 1997-2008.
Thorax
.
2010
;
65
(
9
):
770
774
[PubMed]
7
Whitney
CG
,
Farley
MM
,
Hadler
J
, et al;
Active Bacterial Core Surveillance of the Emerging Infections Program Network
.
Decline in invasive pneumococcal disease after the introduction of protein-polysaccharide conjugate vaccine.
N Engl J Med
.
2003
;
348
(
18
):
1737
1746
[PubMed]
8
Balmer
P
,
Borrow
R
,
Miller
E
.
Impact of meningococcal C conjugate vaccine in the UK.
J Med Microbiol
.
2002
;
51
(
9
):
717
722
[PubMed]
9
Ladhani
S
,
Pebody
RG
,
Ramsay
ME
,
Lamagni
TL
,
Johnson
AP
,
Sharland
M
.
Continuing impact of infectious diseases on childhood deaths in England and Wales, 2003-2005.
Pediatr Infect Dis J
.
2010
;
29
(
4
):
310
313
[PubMed]
10
Saxena
S
,
Bottle
A
,
Gilbert
R
,
Sharland
M
.
Increasing short-stay unplanned hospital admissions among children in England; time trends analysis ’97-’06.
PLoS One
.
2009
;
4
(
10
):
e7484
[PubMed]
11
Gill
PJ
,
Goldacre
MJ
,
Mant
D
, et al
.
Increase in emergency admissions to hospital for children aged under 15 in England, 1999-2010: national database analysis.
Arch Dis Child
.
2013
;
98
(
5
):
328
334
[PubMed]
12
Irwin
AD
,
Wickenden
J
,
Le Doare
K
,
Ladhani
S
,
Sharland
M
.
Supporting decisions to increase the safe discharge of children with febrile illness from the emergency department: a systematic review and meta-analysis.
Arch Dis Child
.
2016
;
101
(
3
):
259
266
[PubMed]
13
Van den Bruel
A
,
Haj-Hassan
T
,
Thompson
M
,
Buntinx
F
,
Mant
D
;
European Research Network on Recognising Serious Infection investigators
.
Diagnostic value of clinical features at presentation to identify serious infection in children in developed countries: a systematic review.
Lancet
.
2010
;
375
(
9717
):
834
845
[PubMed]
14
Van den Bruel
A
,
Thompson
MJ
,
Haj-Hassan
T
, et al
.
Diagnostic value of laboratory tests in identifying serious infections in febrile children: systematic review.
BMJ
.
2011
;
342
:
d3082
[PubMed]
15
Brent
AJ
,
Lakhanpaul
M
,
Thompson
M
, et al
.
Risk score to stratify children with suspected serious bacterial infection: observational cohort study.
Arch Dis Child
.
2011
;
96
(
4
):
361
367
[PubMed]
16
Nijman
RG
,
Vergouwe
Y
,
Thompson
M
, et al
.
Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: diagnostic study.
BMJ
.
2013
;
346
:
f1706
[PubMed]
17
Irwin
AD
,
Marriage
F
,
Mankhambo
LA
, et al;
IPD Group Study
.
Novel biomarker combination improves the diagnosis of serious bacterial infections in Malawian children.
BMC Med Genomics
.
2012
;
5
:
13
[PubMed]
18
Thompson
M
,
Van den Bruel
A
,
Verbakel
J
, et al
.
Systematic review and validation of prediction rules for identifying children with serious infections in emergency departments and urgent-access primary care.
Health Technol Assess
.
2012
;
16
(
15
):
1
100
[PubMed]
19
Bossuyt
PM
,
Reitsma
JB
,
Bruns
DE
, et al;
Standards for Reporting of Diagnostic Accuracy
.
Towards complete and accurate reporting of studies of diagnostic accuracy: the STARD initiative.
Ann Intern Med
.
2003
;
138
(
1
):
40
44
[PubMed]
20
Collins
GS
,
Reitsma
JB
,
Altman
DG
,
Moons
KG
.
Transparent reporting of a multivariable prediction model for individual prognosis or diagnosis (TRIPOD): the TRIPOD statement.
BMJ
.
2015
;
350
:
g7594
[PubMed]
21
UK Standards for Microbiology Investigations
.
Investigation of Blood Cultures (for Organisms Other Then Mycobacterium Species)
.
London, United Kingdom
:
Public Health England
;
2014
22
Corless
CE
,
Guiver
M
,
Borrow
R
,
Edwards-Jones
V
,
Fox
AJ
,
Kaczmarski
EB
.
Simultaneous detection of Neisseria meningitidis, Haemophilus influenzae, and Streptococcus pneumoniae in suspected cases of meningitis and septicemia using real-time PCR.
J Clin Microbiol
.
2001
;
39
(
4
):
1553
1558
[PubMed]
23
Gray
SJ
,
Trotter
CL
,
Ramsay
ME
, et al;
Meningococcal Reference Unit
.
Epidemiology of meningococcal disease in England and Wales 1993/94 to 2003/04: contribution and experiences of the Meningococcal Reference Unit.
J Med Microbiol
.
2006
;
55
(
pt 7
):
887
896
[PubMed]
24
Rutjes
AW
,
Reitsma
JB
,
Coomarasamy
A
,
Khan
KS
,
Bossuyt
PM
.
Evaluation of diagnostic tests when there is no gold standard. A review of methods.
Health Technol Assess
.
2007
;
11
(
50
):
iii
, ix–51
25
Galetto-Lacour
A
,
Zamora
SA
,
Andreola
B
, et al
.
Validation of a laboratory risk index score for the identification of severe bacterial infection in children with fever without source.
Arch Dis Child
.
2010
;
95
(
12
):
968
973
[PubMed]
26
R Core Team
.
R: A Language and Environment for Statistical Computing
.
Vienna, Austria
:
R Foundation for Statistical Computing
;
2013
27
van Buuren
S
.
Multiple imputation of discrete and continuous data by fully conditional specification.
Stat Methods Med Res
.
2007
;
16
(
3
):
219
242
[PubMed]
28
Hastie
T
,
Tibshirani
R
.
Generalized additive models for medical research.
Stat Methods Med Res
.
1995
;
4
(
3
):
187
196
[PubMed]
29
Wood
SN
.
Fast stable restricted maximum likelihood and marginal likelihood estimation of semiparametric generalized linear models.
J R Stat Soc Series B Stat Methodol
.
2011
;
73
(
1
):
3
36
30
Stevenson
M
.
epiR: Tools for the Analysis of Epidemiological Data. R Package Version 0.9-62 ed2015
31
Robin
X
,
Turck
N
,
Hainard
A
, et al
.
pROC: an open-source package for R and S+ to analyze and compare ROC curves.
BMC Bioinformatics
.
2011
;
12
:
77
[PubMed]
32
DeLong
ER
,
DeLong
DM
,
Clarke-Pearson
DL
.
Comparing the areas under two or more correlated receiver operating characteristic curves: a nonparametric approach.
Biometrics
.
1988
;
44
(
3
):
837
845
[PubMed]
33
Van Hoorde
K
,
Vergouwe
Y
,
Timmerman
D
,
Van Huffel
S
,
Steyerberg
EW
,
Van Calster
B
.
Assessing calibration of multinomial risk prediction models.
Stat Med
.
2014
;
33
(
15
):
2585
2596
[PubMed]
34
Nijman
RG
,
Zwinkels
RL
,
van Veen
M
, et al
.
Can urgency classification of the Manchester triage system predict serious bacterial infections in febrile children?
Arch Dis Child
.
2011
;
96
(
8
):
715
722
[PubMed]
35
Lacroix
L
,
Manzano
S
,
Vandertuin
L
,
Hugon
F
,
Galetto-Lacour
A
,
Gervaix
A
.
Impact of the lab-score on antibiotic prescription rate in children with fever without source: a randomized controlled trial.
PLoS One
.
2014
;
9
(
12
):
e115061
[PubMed]
36
de Vos-Kerkhof
E
,
Nijman
RG
,
Vergouwe
Y
, et al
.
Impact of a clinical decision model for febrile children at risk for serious bacterial infections at the emergency department: a randomized controlled trial.
PLoS One
.
2015
;
10
(
5
):
e0127620
[PubMed]
37
Little
P
,
Stuart
B
,
Francis
N
, et al;
GRACE consortium
.
Effects of internet-based training on antibiotic prescribing rates for acute respiratory-tract infections: a multinational, cluster, randomised, factorial, controlled trial.
Lancet
.
2013
;
382
(
9899
):
1175
1182
[PubMed]

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: Prof Carrol serves on a childhood systemic infections panel for BioFire Ltd; the other authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data