Validation of Prediction Models for Pneumonia Among Children in the Emergency Department

We performed a secondary analysis of a prospective cohort study, Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine (CARPE DIEM), which was conducted at Cincinnati Children’s Hospital Medical Center (CCHMC) ED between July 2013 and December 2017. The CCHMC ED is part of a tertiary care specialty pediatric hospital that evaluates an average of 61 990 pediatric encounters per year between 2013 and 2017, of which 823 per year (1.3%) had a diagnosis of pneumonia. The CCHMC and Ann and Robert H Lurie Children’s Hospital Institutional Review Boards approved this study. A radiographic pneumonia prediction model was previously derived and published using data from CARPE DIEM.¹⁵ 

Patients 3 months to 18 years of age with signs and symptoms of lower respiratory tract infection lower respiratory tract infection (defined based on previous work as new or different cough or sputum production, chest pain, dyspnea, tachypnea, or abnormal auscultatory findings)²²  and who had CXR performed for clinical suspicion of community-acquired pneumonia (CAP). We excluded patients with a recent (≤14 days) hospitalization, history of aspiration, medically complex conditions (eg, immunodeficiency, chronic corticosteroid use, chronic lung disease, malignancy, sickle cell disease, congenital heart disease, tracheostomy use, and neuromuscular disorders impacting respiration). Potential patients were prospectively enrolled by research coordinators who used a computerized ED tracking board and who then collaborated with the treating physician to confirm eligibility criteria before enrollment. Research coordinators obtained informed consent from caregivers, and assent from children ≥11 years old. Medical history was collected from patients and guardians by the research coordinators, and physical examination data were collected by the clinical care team.

CXRs were independently interpreted by 2 radiologists masked to clinical information. Radiologists classified CXRs into 1 of 4 categories: (1) normal lungs, (2) definite atelectasis, (3) atelectasis versus pneumonia, and (4) definite pneumonia. We defined our primary outcome, radiographic pneumonia, as an interpretation of atelectasis versus pneumonia or definite pneumonia. We included equivocal radiographs in our outcome measure based on previous literature, which suggests that most clinicians prescribe antibiotics in these cases.²³ 

We evaluated 5 previously published models for pneumonia: Mahabee-Gittens et al,¹³  Lynch et al,¹¹  Neuman et al,¹⁴  Oostenbrink et al,¹²  and Lipsett et al.¹⁶  These models were chosen as they used an outcome of radiographic pneumonia, in contrast to other models that included pneumonia as a part of a composite variable defining serious bacterial infections.^17–20  Models were derived from prospective studies, defined radiographic pneumonia as the outcome, and had similar inclusion criteria, although age ranges varied (Table 1). When validating a model with a narrower age range, the validation data set was limited to the appropriate age range for each associated model.

We matched variables within the validation dataset to those in the derivation studies. We categorized continuous variables using the same criteria from each derivation study. For the classification of tachypnea, we used guidelines provided by the model for each definition: Neuman et al used age-specific thresholds, Oostenbrink used World Health Organization criteria,²⁴  Lynch et al used Pediatric Risk of Admission criteria,²⁵  and Mahabee-Gittens used a criteria of >50 respirations per minute. For the variable of “ill appearance” described by Neuman and Oostenbrink, we used a variable for general appearance based on exam by an ED clinician, which was classified into 5 categories: well, mildly ill or distressed, moderately ill or distressed, and severely ill or distressed, and designated the presence of mild, moderate or severely ill or distressed as “ill appearing.”^12,14 

We imputed missing data via multiple imputation by chained equations, creating 5 imputed data sets over which all subsequent results were averaged.²⁶  We generated predicted probabilities for each of the previously published models under 2 methods: (1) using the values of the regression coefficients as originally published (“coefficients as published”), and (2) using the CARPE DIEM dataset to estimate new regression coefficients for the variables included in each model (“coefficient derived from data”). For 3 models,^11,13,14  model intercepts were not published. We estimated intercepts for these models in the “coefficients as published” analysis by fixing the regression coefficients of the included variables at their published values and estimating the intercepts on the CARPE DIEM data set.

We conducted receiver operating characteristic analyses of the predicted probabilities generated by each model and calculated area under the receiver operating characteristic curve (AUROC) with 95% confidence intervals (95% CI). We constructed calibration graphs of the predicted probabilities against the observed prevalence of pneumonia in both continuous and decile-categorized formats.^27,28  We identified optimal cutoffs for the predicted probabilities from each model using the Euclidean distance method²⁹  and compared diagnostic accuracy statistics (sensitivity, specificity, positive and negative predictive values, positive and negative likelihood ratios).

Given the large proportion of CARPE DIEM patients with missing c-reactive protein (CRP) data, we repeated our validation analyses in only those patients with an observed CRP (ie, not imputed) to validate with the Oostenbrink model. Additionally, given potential differences in CAP etiology by age, we evaluated the performance of all models for the subset of children <5 years of age. Analysis was performed with R, version 4.0.3 (R Foundation for Statistical Computing, Vienna, Austria).

Of 1142 patients enrolled, the median patient age was 3.3 years (interquartile range, 1.4–7.1 years) and 54% were male. Radiographic pneumonia was found in 253 (22%) patients (203 with definite pneumonia and 50 with pneumonia versus atelectasis). Characteristics of the cohort, with rates of missing data before imputation, are provided in Table 2.

The models by Neuman and Lipsett included the whole CARPE DIEM cohort. The remaining 4 were only applied to subsets of the cohort because of different age ranges used in the original models. The percentage of patients with radiographic pneumonia were similar in the validation cohorts for the models of Neuman, Lipsett, Oostenbrink, and Lynch (22% to 24%), but was lower in the validation cohort of the model of Mahabee-Gittens (12%). Supplemental Table 4 describes differences in the variables included in model between those with and without radiographic pneumonia for their respective age-based validation cohorts.

Using the coefficients as published, the highest AUROC was demonstrated by the models of Neuman and Lipsett (0.72, 95% CI 0.68–0.75 for both models, Fig 1). The AUROCs for the remaining 3 models were substantially lower than the Neuman and Lipsett models. When utilizing coefficients estimated from the CARPE DIEM dataset, the model of Neuman exhibited the highest AUROC (0.79, 95% CI 0.75–0.82) followed by Lipsett (0.76, 95% CI 0.73–0.80). The AUROC for the model of Oostenbrink increased substantially when coefficients were estimated from CARPE DIEM (from 0.53 [95% CI 0.49–0.57] to 0.72 [95% CI 0.68–0.77]), whereas the models of Lynch and Mahabee-Gittens exhibited modest increases.

In the coefficients-as-published analysis, the model of Neuman demonstrated a sensitivity of 70.0% and specificity of 65.4% and the models of Lynch and Lipsett demonstrated high sensitivity (83.0% and 81.7%, respectively) with lower specificity (30.0% and 52.6% respectively). The Oostenbrink model exhibited lower sensitivity (63.4%) and specificity (49.8%). The model of Mahabee-Gittens classified nearly the entire cohort (670 of 725 cases, or 92%) as positive for pneumonia, resulting in a high sensitivity (95.3%) but low specificity (8.0%). In the coefficients derived from the CARPE DIEM cohort, 1 or more performance characteristics improved for each of the models (Table 3).

None of the models calibrated well using the coefficients as published, each predicting higher risk of pneumonia than what was observed, and an observed risk of pneumonia that did not increase with predicted risk (Supplemental Fig 3). Calibration for all models substantially improved when the coefficients were derived from CARPE DIEM data. Model performance at intervals of predicted risk is provided in Fig 2. The model of Neuman calibrated best among the 5 models, followed by the model of Lipsett. A notable feature of the Lynch, Mahabee-Gittens, and, to a lesser extent, Oostenbrink models was the limited range of predicted probabilities exhibited by each when applied to the validation dataset.

In an additional analysis, externally validating the Oostenbrink model among the 432 CARPE DIEM patients with CRP data available, the AUROC of originally published coefficients was 0.55 (95% CI 0.49–0.60), which improved to 0.75 (95% CI 0.70–0.80) when using coefficients derived from the CARPE DIEM dataset. Model discriminatory performance declined slightly when analyses were limited to children <5 years of age (Supplemental Fig 4) and in the performance at an optimally-selected cutpoint (Supplemental Table 5).

We externally evaluated 5 previously published prediction models for radiographic pneumonia using a prospective cohort of children with suspected CAP. Of these, the models reported by Neuman, Lipsett, and Oostenbrink demonstrated the highest performance, with AUROCs >0.7 after fitting model variables to the validation dataset. No model provided clear discrimination between patients with and without radiographic pneumonia at a single cutoff.

The Neuman, Lipsett, and Oostenbrink models demonstrated the highest performance in this external validation, though the variables used in these models are different. The Neuman model requires extensive data, including historical and physical examination variables.¹⁴  Some of these variables, such as chest pain, may be difficult to ascertain in younger patients. Others, such as those relating to accessory muscle use and auscultatory findings, may be challenging to ascertain (eg, patients with high body mass index) and may be subject to limitations of interrater reliability.³⁰  The model reported by Lipsett retained excellent performance with fewer clinical variables (age, presence of fever, wheeze, rales, and oxygen desaturation). Fewer clinical data are required by the Oostenbrink model, though this model requires the performance of a laboratory test (CRP) and 1 of the measures (ill-appearance) is inherently subjective.¹²  Reliance on a blood biomarker raises challenges with respect to institutional availability of point of care testing, time and cost, and discomfort required for venipuncture. Furthermore, the ability to apply the Oostenbrink model in a less ill-appearing population may be challenging, as blood testing is infrequently performed in these patients.

Predictive performance was weaker for the other 2 models we studied. The study by Mahabee-Gittens had limited performance after fitting the model with covariates derived from our dataset. This is likely because of the Mahabee-Gittens cohort being limited to a younger patient sample (2 months–5 years) where the radiographic pneumonia is less common.¹³  Additionally, the prediction of radiographic pneumonia among younger children is challenging because of the high incidence of bronchiolitis and viral pneumonia, which have similar clinical manifestations but variable findings on CXR.³¹  This combination of factors underscores the challenges in developing a prediction model for pediatric pneumonia limited to young children.

By externally validating decision models for radiographic pneumonia using a population of patients distinct in time and place but with a similar inclusion criteria from the derivation studies, this study demonstrates the limited generalizability of some of the studied models.³²  A decline in performance is generally expected when externally validating a model in a population distinct from the derivation cohort.²¹  The Mahabee-Gittens model, for example, had a decline in the AUROC from the derivation study (0.81; 95% CI 0.75–0.87) compared with our validation study (0.58; 95% CI 0.52–0.64). In contrast, the model by Lipsett demonstrated an increase in AUROC from the original derivation study (0.71, 95% CI 0.67–0.75) to the present external validation (0.76; 95% CI 0.73–0.80), suggesting that this model may have better transportability. As such, this model, which carries the benefit of having only 6 variables and does not require laboratory biomarkers, may be most beneficial in clinical practice. If the predicted probability of disease is low, then other disease states may be considered, and radiography may be avoided. Alternatively, if the predicted probability is high, the patient may be treated for pneumonia without a confirmatory chest radiography.

Many clinical prediction models in emergency medicine, such as those used for young febrile infants for serious bacterial infection³³  or for children with head trauma for clinically important traumatic brain injury,³⁴  are used in a “1-way” fashion to determine patients at a low risk of an outcome but generally cannot be used to determine high risk. None of the radiographic pneumonia models we examined provided satisfactory discrimination between patients with and without radiographic pneumonia at a single threshold using a binary cutoff. This challenge, additionally noted in the CARPE DIEM rule,¹⁵  means that no clear, singular cutoff can be used to characterize patients as low- or high-risk in a dichotomous fashion to make decisions about CXR or empirical antimicrobial use. This likely relates to the heterogeneity of pneumonia presentation among children, differences in disease etiology, and variability in radiograph interpretation. At the statistically derived cutoffs, the Oostenbrink model demonstrated the greatest ability to “rule in” pneumonia with the highest specificity and positive likelihood ratio. The Neuman model demonstrated moderate ability to rule in but the highest performance to rule out, as reflected by the sensitivity and negative likelihood ratio. The internally derived CARPE DIEM radiographic pneumonia prediction model contains 3 variables: duration of fever, focal decreased breath sounds, and age.¹⁵  Evaluated on this subset of patients, this model demonstrated an AUROC of 0.81 (95% CI 0.66–0.84). At an optimally selected cutpoint, this model demonstrated a sensitivity of 73.1% and a specificity of 75.8%, though this model has not yet been externally validated. None of the rules demonstrated a likelihood ratio of greater than 5 or a negative likelihood ratio less than 0.2, which is generally suggestive of a substantial change in pretest to posttest probability. Although performance of the models declined slightly when models were examined in children younger than 5 years of age, most of these were not derived specifically in this age group, and thus a decline in performance is expected.

Prediction rules for pneumonia are not widely used in clinical practice. The reasons for their lack of use may include the lack of external validation, inability to sufficiently differentiate patients with and without radiographic pneumonia with a single threshold, and challenges related to implementation of best practice regarding defining diagnosis and etiology. Successful implementation would potentially decrease overuse of CXR³  and antibiotics³⁵  and facilitate a decrease in unnecessary practice variation.³⁶  We have taken the initial step to externally validate several of these previously derived models. A prediction rule may be implemented in multiple ways. Online calculators, such as in 1 recently described for urinary tract infection risk stratification in young febrile children,³⁷  may allow for the dynamic risk stratification of children considered for a specific disease wherein the logistic regression formula is applied, and a predicted probability is returned to the user. Alternatively, a model may be directly embedded into the electronic medical record, similar to efforts that have been described with pediatric sepsis.³⁸  These predicted probabilities may in turn guide management decisions (eg, a decision to perform CXR in children at intermediate risk, or a decision regarding use of antibiotics without CXR in children at very low or high risk). Finally, a future step may be to evaluate the validity of these models in other settings, including for children seen in the primary care offices or for admitted patients.

Our study has limitations. Certain variables, such as ill appearance used in Oostenbrink were not objectively defined. In addition, most of the studied models did not include complete data required for external validation (ie, model coefficients and intercepts),^11,12,14  requiring additional steps to estimate the model performance on the study data. As such, we used 2 analytical approaches toward external validation. We performed multiple imputation for missing data in the CARPE DIEM dataset. This was particularly important in validating the Oostenbrink where a larger proportion of patients had missing CRP. However, the performance of the Oostenbrink model even when limited to those with CRP obtained was similar to the primary analysis performed with imputation. Additionally, all the models studied were developed for ED use and were validated on the data from another pediatric ED. Therefore, the applicability of these model to other settings remains unknown.

Implementation of well validated models can facilitate CXR or antibiotic use, particularly when CXR is not easily available. By providing a probability of radiographic pneumonia based on clinical factors, clinicians can integrate an evidence-based risk estimate into their clinical decision-making. In this study, 3 models revealed superior performance during validation.

CARPE DIEM

Catalyzing Ambulatory Research in Pneumonia Etiology and Diagnostic Innovations in Emergency Medicine

AUROC

area under the ROC curve

CAP

community-acquired pneumonia

CCHMC

Cincinnati Children’s Hospital Medical Center

CXR

chest radiograph

ED

emergency department