Substantial morbidity and excessive care variation are seen with pediatric pneumonia. Accurate risk-stratification tools to guide clinical decision-making are needed.
We developed risk models to predict severe pneumonia outcomes in children (<18 years) by using data from the Etiology of Pneumonia in the Community Study, a prospective study of community-acquired pneumonia hospitalizations conducted in 3 US cities from January 2010 to June 2012. In-hospital outcomes were organized into an ordinal severity scale encompassing severe (mechanical ventilation, shock, or death), moderate (intensive care admission only), and mild (non–intensive care hospitalization) outcomes. Twenty predictors, including patient, laboratory, and radiographic characteristics at presentation, were evaluated in 3 models: a full model included all 20 predictors, a reduced model included 10 predictors based on expert consensus, and an electronic health record (EHR) model included 9 predictors typically available as structured data within comprehensive EHRs. Ordinal regression was used for model development. Predictive accuracy was estimated by using discrimination (concordance index).
Among the 2319 included children, 21% had a moderate or severe outcome (14% moderate, 7% severe). Each of the models accurately identified risk for moderate or severe pneumonia (concordance index across models 0.78–0.81). Age, vital signs, chest indrawing, and radiologic infiltrate pattern were the strongest predictors of severity. The reduced and EHR models retained most of the strongest predictors and performed as well as the full model.
We created 3 risk models that accurately estimate risk for severe pneumonia in children. Their use holds the potential to improve care and outcomes.
We appreciate the comment by Dr. van de Maat and colleagues regarding our article, “Predicting Severe Pneumonia Outcomes in Children.” We welcome their enthusiasm for our work and attempt to extend it. This is a critical step towards developing useful predictive models for disease severity in pediatric pneumonia, something that is urgently needed. Dr. van de Maat et al. built a model using six of the nine predictor variables included in our electronic health record (EHR) model and assessed its performance among children presenting to the emergency department with suspected pneumonia. The adjusted odds ratios estimated from our original EHR model were very similar to those estimated using this reduced set of six predictor variables (Table). We caution, however, that Dr. van de Maat et al. used hospitalization as the outcome, which is different from the 3 level ordinal in-hospital outcome used in our study. Further, due to the complexity of our model, which included interaction terms and restricted cubic splines for selected variables, we reported adjusted odds ratios for categorical predictors and presented only selected values along with graphical displays for continuous predictors. As acknowledged by Dr. van de Maat’s team, proper validation would require use of the original coefficients estimated from our model. We also agree that inclusion of better biomarkers, such as C-reactive protein or procalcitonin, in place of white blood cell count might improve our model’s predictive accuracy, although these biomarkers were not initially measured in the EPIC study.
Importantly, Dr. van de Maat and colleagues’ application of our EHR model in a different setting using a simplified outcome and a reduced set of covariates showed good performance and seemed useful and informative. Nonetheless, complete external validation of our EHR prediction tool is still needed prior to clinical application.
Williams et al. developed three different models to predict moderate or severe outcomes of childhood pneumonia (1). All models perform well with C-indices ranging from 0.78 to 0.81. One of the models, the so-called EHR-model, included predictors that are routinely available in electronic health records (EHR). In this way clinical relevance and practical utility were taken into account, in addition to statistical significance. The aim of the authors to guide and standardize clinical decision-making based on outcomes is very promising.
To evaluate the feasibility in another setting, we applied this EHR-model to data from our Emergency Department (ED) as collected during ongoing diagnostic research projects. We observed however, that in our database of routinely collected data only 6 out of the 9 variables are available (2). Race, systolic blood pressure and white blood count (WBC) were not routinely collected. These variables contributed in small amount to the original model. In addition, blood pressure is known to be a late sign of deterioration and rarely deviates from normal at the ED. In contrast to WBC, C-reactive protein (CRP) is used at our ED as a preferred determinant of serious infections. This is in line with diagnostic studies, proving higher diagnostic value for CRP than for WBC (3).
The authors included children hospitalized for signs of respiratory illness and with radiographic evidence of pneumonia. Selection of severe patients influences the predictive value of developed models and their generalizability (4). As the authors mention in their discussion, one would be interested in selecting children at risk for severe pneumonia from the vast majority of children with respiratory symptoms presenting at the ED. We therefore validated the EHR-model to a prospective cohort of 248 children (aged < 5 years) visiting the ED suspected of pneumonia (based on presentation with fever and cough and/or dyspnea), using hospitalization as the outcome. Compared to the original population, our study population can be considered as a low-prevalence setting with 55 cases (23%) hospitalized, of whom only 2 at the intensive care unit. Due to missing variables, we excluded race, systolic blood pressure and WBC from the model, and used a proxy for PF-ratio based on oxygen saturation. Using the coefficients for categories of variables as presented in the article, we applied the model to our multiply imputed database which resulted in a C-index of 0.71 (95% CI 0.64-0.79). Lower predictive value in our population may be explained by case mix differences and identify some overfitting of the developed model. Next, use of the original (continuous) coefficients may improve the performance. Finally, predictive value of the model will increase by adding laboratory tests (CRP).
In conclusion, with the above mentioned limitations in our validation method, we find that the EHR-model also helps to identify children that require in-hospital treatment for pneumonia among the vast majority of children presenting at the ED with suspicion of lower airway infection. Improvement could be achieved by adding CRP. However, not all variables of the EHR-model are documented in routine practice.
1. Williams DJ, Zhu Y, Grijalva CG, Self WH, Harrell FE, Jr., Reed C, et al. Predicting Severe Pneumonia Outcomes in Children. Pediatrics. 2016.
2. Nijman RG, Vergouwe Y, Thompson M, Veen MV, Van Meurs AHJ, Van Der Lei J, et al. Clinical prediction model to aid emergency doctors managing febrile children at risk of serious bacterial infections: Diagnostic study. BMJ (Online). 2013;346(7905).
3. Van Den Bruel A, Thompson MJ, Haj-Hassan T, Stevens R, Moll H, Lakhanpaul M, et al. Diagnostic value of laboratory tests in identifying serious infections in febrile children: Systematic review. BMJ. 2011;342(7810).
4. Oostenbrink R, Moons KG, Bleeker SE, Moll HA, Grobbee DE. Diagnostic research on routine care data: prospects and problems. J Clin Epidemiol. 2003;56(6):501-6.