The goal of this study was to determine the incidence, prognostic performance, and generalizability of the Pediatric Organ Dysfunction Information Update Mandate (PODIUM) organ dysfunction criteria using electronic health record (EHR) data. Additionally, we sought to compare the performance of the PODIUM criteria with the organ dysfunction criteria proposed by the 2005 International Pediatric Sepsis Consensus Conference (IPSCC).
Retrospective observational cohort study of critically ill children at 2 medical centers in the United States between 2010 and 2018. We assessed prevalence of organ dysfunction based on the PODIUM and IPSCC criteria for each 24-hour period from admission to 28 days. We studied the prognostic performance of the criteria to discriminate in-hospital mortality.
Overall, 22 427 PICU admissions met inclusion criteria, and in-hospital mortality was 2.3%. The cumulative incidence of each PODIUM organ dysfunction ranged from 15% to 30%, with an in-hospital mortality of 6% to 10% for most organ systems. The number of concurrent PODIUM organ dysfunctions demonstrated good-to-excellent discrimination for in-hospital mortality (area under the curve 0.87–0.93 for day 1 through 28) and compared favorably to the IPSCC criteria (area under the curve 0.84–0.92, P < .001 to P = .06).
We present the first evaluation of the PODIUM organ dysfunction criteria in 2 EHR databases. The use of the PODIUM organ dysfunction criteria appears promising for epidemiologic and clinical research studies using EHR data. More studies are needed to evaluate the PODIUM criteria that are not routinely collected in structured format in EHR databases.
Various criteria for pediatric organ dysfunction have been developed and used over the years. Increasingly, it’s become common to leverage clinical databases to perform epidemiologic studies and evaluate the ability of the different organ dysfunction criteria to discriminate outcomes.
We present the first evaluation of the PODIUM organ dysfunction criteria in 2 electronic health record databases. The PODIUM criteria have good-to-excellent discrimination of mortality and appear promising for epidemiologic and clinical research studies leveraging electronic health record data.
Identifying and supporting dysfunctional organ systems are cornerstones of pediatric critical care.1 Multiple organ dysfunction (MOD) is a final common pathway for death in critically ill children, and even among survivors it is associated with long-term functional impairment and disability.2–4 However, defining single and MODs is challenging because no “gold standards” exist and we lack a complete understanding of the complex pathobiology underlying most forms of organ dysfunction.5 Nevertheless, various diagnostic criteria for pediatric organ dysfunction, such as those proposed by Wilkinson, Proulx, or the 2005 International Pediatric Sepsis Consensus Conference (IPSCC), have been developed, validated, and used clinically over the years.6–8
As the knowledge and practice of pediatric critical care have continued to evolve, so have the criteria for diagnosing single and MOD. Examples of these include the increased use of oxygen saturation-based measures to describe hypoxemic respiratory failure or the evolving understanding of fluid overload and its associations with outcomes.9,10 In response to these and other advances in the field, the Pediatric Organ Dysfunction Information Update Mandate (PODIUM) convened a group of >60 international experts to develop an updated set of diagnostic criteria for single and multiple organ dysfunction.11
Given the growth of the digital infrastructure in health care, it has become increasingly common to use electronic clinical databases to perform epidemiologic studies of MOD or evaluate the ability of organ dysfunction scoring systems to discriminate outcomes in critically ill children.3,12 In the case of sepsis, for example, measures of organ dysfunction derived from electronic health record (EHR) databases have been used to develop and validate diagnostic criteria in adults, characterize phenotypes with prognostic and therapeutic relevance, and perform automatic surveillance in children.13–15 Furthermore, it is widely expected that the use of EHR data and data-driven systems will continue to grow in the critical care setting, and it is therefore paramount that an evaluation of the face validity, prognostic performance, and generalizability of a new set of organ dysfunction criteria be performed in EHR databases.16,17
In this study, we present the first evaluation of the PODIUM organ dysfunction criteria using structured data from 2 large EHR databases of critically ill children. The goal of this study was to test the face validity, prognostic performance, and generalizability of the PODIUM criteria using EHR data.
Methods
Study Design and Data Sources
We conducted a retrospective observational cohort study of critically ill children admitted to 2 academic medical centers. We included all children aged 0 to 17 years admitted to the PICU between 2010 and 2018 (Ann & Robert H. Lurie Children’s Hospital of Chicago) or 2010 and 2016 (University of Chicago Comer Children’s Hospital). Patients were excluded if they had congenital heart disease or were recovering from cardiac surgery. Data were extracted from the 2 institutional databases using structured queries and underwent quality checks for conformity, completeness, and plausibility.18
Only the information pertaining to the first PICU encounter and up to 28 days, discharge, or death (whichever came first) in each hospitalization was included, as well as whether the patient survived the hospitalization or not. Each admission was treated independently. The primary outcome was in-hospital mortality. Oncologic comorbidity was determined using Feudtner’s complex chronic condition classification.19
Assessment of the PODIUM Organ Dysfunction Criteria in the EHR Data Sets
The goal of this study was to test the face validity (ie, the degree to which an assessment appears effective in terms of its stated aims), prognostic performance, and generalizability of the PODIUM criteria using EHR data. To do this, only commonly recorded structured data elements from the EHRs were included in the analysis. Structured EHR data include laboratory test results, medications administered, diagnostic codes, and clinician-charted observations and interventions (eg, vital signs, intake/output, ventilator settings, etc).16 Unstructured data contained in EHRs and other clinical information systems (eg, clinician notes, diagnostic reports, imaging studies) may also contain important information related to organ dysfunction. However, extracting information from these data sources in a generalizable way can be challenging and impractical, and hence we excluded them from the analysis. Additionally, structured data elements that were recorded in <5% of patients were excluded to maximize the face validity and generalizability of the results using EHR data.
We assessed organ dysfunction criteria for each 24-hour period starting at PICU admission and up to 28 days, discharge, or death (whichever came first). When >1 criterion for a single organ dysfunction was recorded in a 24-hour period, the worst value was used. If no values were recorded in a 24-hour period, they were assumed to be normal (ie, not meeting criteria for organ dysfunction). The degree of severity of individual criteria (which was only graded in the respiratory, cardiovascular, and hematologic systems) was not considered for this analysis. Specific criteria used to determine organ dysfunction in the EHR data sets are detailed in the Supplemental Information.
Prognostic Performance and Comparison With the IPSCC Criteria
We assessed the prognostic performance of the PODIUM organ dysfunction criteria using the number of concurrent PODIUM organ dysfunctions as predictors and in-hospital mortality as the outcome and using the area under the receiver operating characteristic curve to measure discrimination. We assessed the prognostic performance of the number of concurrent PODIUM organ dysfunctions on day 1 of PICU admission, as well as the maximum number of concurrent PODIUM organ dysfunctions met by day 3, day 7, and day 28 of admission. We compared the performance of these measures with the maximum number of concurrent organ dysfunctions met by the IPSCC organ dysfunction criteria at the same time intervals.6 Specific criteria used to determine organ dysfunction by the IPSCC criteria are detailed in the Supplemental Information.
New, Progressive, and Persistent Multiple Organ Dysfunction
We examined the prevalence of MOD on day 1 of PICU stay and the development of new or progressive MOD (NPMOD) and persistent MOD by day 7. Based on the consensus PODIUM criteria, MOD syndrome is considered a subset of MOD in which there is a shared common cause for the organ dysfunctions. MOD, on the other hand, is defined as the presence of ≥2 concurrent organ dysfunctions, regardless of cause. New MOD was defined as MOD that developed between days 2 and 7 in patients who had single or no organ dysfunction on day 1. Progressive MOD was defined as the development of one or more additional organ dysfunctions or death between days 2 and 7 in patients who already met MOD criteria on day 1. NPMOD was defined as patients who met either new or progressive MOD criteria.5 Persistent MOD was defined as the presence of MOD on day 7 among patients who already met MOD criteria on day 1.
Statistical Analyses
Data were analyzed using R version 4.0 (R Foundation for Statistical Computing, Vienna, Austria). Categorical variables were compared using the χ2 test and continuous variables using the nonparametric Kruskal-Wallis test. In-hospital mortality discrimination was assessed using the area under the curve (AUC) and compared using the DeLong method.
The institutional review boards at Ann & Robert H. Lurie Children’s Hospital of Chicago and The University of Chicago approved this study with a waiver of informed consent. The reporting of this observational cohort study was performed using the Strengthening the Reporting of Observational Studies in Epidemiology checklist for cohort studies.20
Results
Study Population
Overall, 22 427 PICU admissions by 15 447 unique patients met inclusion criteria during the study period. The median age on admission was 4.6 years (interquartile range 1.4–11.6), and in-hospital mortality was 2.3%. Table 1 presents the demographic and clinical characteristics of patents who died and those who survived the admission.
Demographic and Clinical Characteristics of PICU Admissions
Variables . | Survived . | Died . | P . |
---|---|---|---|
n (%) | 21913 (97.7) | 514 (2.3) | — |
Age, y, median (IQR) | 4.6 (1.4–11.6) | 4.4 (1–11.9) | .49 |
Male, n (%) | 12075 (55.1) | 291 (56.6) | .53 |
Race/ethnicity, n (%) | .02 | ||
Black | 7410 (33.8) | 159 (30.9) | |
White | 6956 (31.7) | 155 (30.2) | |
Hispanic | 5494 (25.1) | 132 (25.7) | |
Other | 2053 (9.4) | 68 (13.2) | |
Oncology diagnosis | 2003 (9.1) | 146 (28.4) | <.001 |
Maximum number of concurrent organ dysfunctions by day 28 | |||
PODIUM, median (IQR) | 1 (0–2) | 5 (4–7) | <.001 |
IPSCC | 1 (0–2) | 4 (3–5) | <.001 |
Length of stay, d, median (IQR) | 4 (2–8.1) | 5 (1.7–20) | <.001 |
Hospital A versus B, n (%) | 13849 (63.2) | 307 (59.7) | .12 |
Variables . | Survived . | Died . | P . |
---|---|---|---|
n (%) | 21913 (97.7) | 514 (2.3) | — |
Age, y, median (IQR) | 4.6 (1.4–11.6) | 4.4 (1–11.9) | .49 |
Male, n (%) | 12075 (55.1) | 291 (56.6) | .53 |
Race/ethnicity, n (%) | .02 | ||
Black | 7410 (33.8) | 159 (30.9) | |
White | 6956 (31.7) | 155 (30.2) | |
Hispanic | 5494 (25.1) | 132 (25.7) | |
Other | 2053 (9.4) | 68 (13.2) | |
Oncology diagnosis | 2003 (9.1) | 146 (28.4) | <.001 |
Maximum number of concurrent organ dysfunctions by day 28 | |||
PODIUM, median (IQR) | 1 (0–2) | 5 (4–7) | <.001 |
IPSCC | 1 (0–2) | 4 (3–5) | <.001 |
Length of stay, d, median (IQR) | 4 (2–8.1) | 5 (1.7–20) | <.001 |
Hospital A versus B, n (%) | 13849 (63.2) | 307 (59.7) | .12 |
IQR, interquartile range; —, not applicable.
Assessment of the PODIUM Organ Dysfunction
Cumulative Incidence and Mortality.
The cumulative incidence of 7 of the 9 individual PODIUM organ dysfunctions by day 28 ranged from 15% to 30%, with an associated in-hospital mortality of 6% to 10% (Fig 1). Liver and coagulation dysfunction were more rare (cumulative incidence 3% and 5%, respectively) and associated with higher in-hospital mortality (30% and 16%, respectively).
Prevalence and associated in-hospital mortality for each of the PODIUM organ dysfunction criteria studied.
ALC, absolute lymphocyte count; ANC, absolute neutrophil count; CI, confidence interval; ECMO, extracorporeal membrane oxygenation; FO, fluid overload; GCS, Glasgow Coma Scale; GCS-m, GCS motor response; Gluc, blood glucose level; HE, hepatic encephalopathy (of note, GCS used as proxy); Hgb, hemoglobin; INR, international normalized ratio; OI, oxygenation index; Onc, oncology; OSI, oxygen saturation index; PF, PaO2/FiO2 ratio; Plts, platelet count; RRT, renal replacement therapy; SCr, serum creatinine; SF, SpO2/FiO2 ratio; UOP6h, low urine output for 6 hours; UOP12h, low urine output for 12 hours; VA, veno-arterial; WBC, white blood cell count.
Prevalence and associated in-hospital mortality for each of the PODIUM organ dysfunction criteria studied.
ALC, absolute lymphocyte count; ANC, absolute neutrophil count; CI, confidence interval; ECMO, extracorporeal membrane oxygenation; FO, fluid overload; GCS, Glasgow Coma Scale; GCS-m, GCS motor response; Gluc, blood glucose level; HE, hepatic encephalopathy (of note, GCS used as proxy); Hgb, hemoglobin; INR, international normalized ratio; OI, oxygenation index; Onc, oncology; OSI, oxygen saturation index; PF, PaO2/FiO2 ratio; Plts, platelet count; RRT, renal replacement therapy; SCr, serum creatinine; SF, SpO2/FiO2 ratio; UOP6h, low urine output for 6 hours; UOP12h, low urine output for 12 hours; VA, veno-arterial; WBC, white blood cell count.
When stratified by age group, the cumulative incidence and associated in-hospital mortality were similar, with some exceptions (Supplemental Fig 1). Infants <1 year of age had higher cumulative incidence of respiratory and cardiovascular dysfunction, and school-aged children >6 years had higher cumulative incidence of renal, endocrine, and immune dysfunction. When stratified by hospital, the cumulative incidence and associated mortality were comparable across all organ dysfunctions (Supplemental Fig 2).
The number of concurrent PODIUM organ dysfunction criteria met was associated with in-hospital mortality in a dose-dependent fashion (Fig 2). This association was stronger when the organ dysfunction criteria were met concurrently on the same day as opposed to cumulatively over the course of the admission. For example, meeting criteria for 6 organ dysfunctions by day 28 was associated with an in-hospital mortality of 13%, whereas meeting 6 organ dysfunctions on the same day was associated with an in-hospital mortality of 29% (Supplemental Fig 3).
Distribution and associated in-hospital mortality of the maximum number of concurrent organ dysfunctions based on the PODIUM criteria (left) and the 2005 IPSCC criteria (right) by day 28.
Distribution and associated in-hospital mortality of the maximum number of concurrent organ dysfunctions based on the PODIUM criteria (left) and the 2005 IPSCC criteria (right) by day 28.
The number of concurrent organ dysfunctions met by the IPSCC criteria had a similar dose-dependent relationship with in-hospital mortality (Fig 2).
Timing.
Among patients who met any organ dysfunction criteria, 80% to 90% met criteria by day 3 and >90% met criteria by day 7 (Supplemental Fig 4). In general, endocrine and immunologic dysfunction criteria were met earlier and accumulated faster (∼80% on day 1 and 90% by day 3). Renal and coagulation dysfunction criteria were met later and accumulated slower (∼55%–60% on day 1 and 90% by day 6).
Prognostic Performance of the PODIUM Criteria
The number of concurrent PODIUM organ dysfunctions had good to excellent discrimination of in-hospital mortality at different intervals during the admission (Table 2). The AUCs ranged from 0.87 for the number of criteria met on day 1 to 0.93 for the maximum number of criteria met by day 28. In general, the performance of the PODIUM criteria was better than the IPSCC criteria (Table 2).
Comparison of the Prognostic Performance of the Number of Concurrent PODIUM and the 2005 International Pediatric Sepsis Consensus Conference (IPSCC) Organ Dysfunction Criteria Met at Different Intervals
Concurrent Organ Dysfunctions . | PODIUM AUC (95% CI) . | IPSCC AUC (95% CI) . | P . |
---|---|---|---|
On day 1 | 0.87 (0.85–0.89) | 0.84 (0.82–0.86) | <.001 |
Maximum by day 3 | 0.89 (0.87–0.90) | 0.87 (0.85–0.89) | .02 |
Maximum by day 7 | 0.91 (0.89–0.92) | 0.89 (0.87–0.90) | <.001 |
Maximum by day 28 | 0.93 (0.91–0.94) | 0.92 (0.91–0.93) | .06 |
Concurrent Organ Dysfunctions . | PODIUM AUC (95% CI) . | IPSCC AUC (95% CI) . | P . |
---|---|---|---|
On day 1 | 0.87 (0.85–0.89) | 0.84 (0.82–0.86) | <.001 |
Maximum by day 3 | 0.89 (0.87–0.90) | 0.87 (0.85–0.89) | .02 |
Maximum by day 7 | 0.91 (0.89–0.92) | 0.89 (0.87–0.90) | <.001 |
Maximum by day 28 | 0.93 (0.91–0.94) | 0.92 (0.91–0.93) | .06 |
New, Progressive, and Persistent Multiple Organ Dysfunction
Based on the PODIUM criteria, 30.2% of patients had MOD on day 1. These patients had an in-hospital mortality rate of 6.4% and an odds ratio (OR) of 13.6 (95% confidence interval [CI] 10.7–17.4) for mortality compared with those who did not meet MOD criteria on day 1 (Supplemental Table 1). MOD on day 1 by PODIUM criteria had a sensitivity of 85% and a specificity of 71% for in-hospital mortality. When using the IPSCC criteria, 24.2% of patients had MOD on day 1, of whom 7.4% died (OR 12.2 for mortality, 95% CI 9.9–15.1), resulting in a sensitivity of 78% and a specificity of 77% for in-hospital mortality.
NPMOD or persistent MOD based on PODIUM criteria by day 7 occurred in 17.7% of patients, with an associated in-hospital mortality of 11.7%, and an OR of 47.8 (95% CI 36.1–64.8) for mortality compared with those who did not meet criteria. Meeting NPMOD or persistent MOD criteria had a sensitivity of 90% and a specificity of 84% for in-hospital mortality. When using the IPSCC criteria, 15.3% of patients had NPMOD or persistent MOD by day 7, of whom 12.5% died (OR 32.6 for mortality, 95% CI 25.9–41.7), resulting in a sensitivity of 84% and a specificity of 86% for in-hospital mortality (Supplemental Table 1).
Supplemental Fig 5 presents the average trajectory of organ dysfunction by type and day during the first 7 days among patients with MOD on day 1 based on the PODIUM criteria. Supplemental Table 2 presents this same information with the prevalence and associated in-hospital mortality for each individual organ dysfunction. In general, endocrine dysfunction was common in most patients on day 1. Nonsurvivors and those with progressive or persistent MOD had more cardiovascular, neurologic, and respiratory dysfunctions earlier in the course and persistent neurologic, respiratory, and hematologic dysfunction later in the course (Supplemental Fig 5). New, progressive, and persistent respiratory dysfunction was associated with the largest number of deaths compared with the other individual organ dysfunctions (Supplemental Table 2).
Discussion
In this study, we present the first evaluation of the PODIUM organ dysfunction criteria in 2 large EHR databases. Seven of the 9 individual PODIUM organ dysfunctions had a cumulative incidence ranging from 15% to 30% by day 28, with an associated in-hospital mortality of 6% to 10%. The other 2 organ dysfunctions, liver and coagulation dysfunction, occurred in 3% to 5% of patients and were associated with an in-hospital mortality of 16% to 30%. The cumulative incidence and associated mortality were relatively stable across age groups and the 2 hospitals in the study, demonstrating generalizability. The number of concurrent PODIUM organ dysfunction criteria met by individual patients on day 1 of PICU stay demonstrated good discrimination for in-hospital mortality and better than the IPSCC criteria (AUC 0.87 vs 0.84, P < .001). Additionally, the maximum number of concurrent PODIUM organ dysfunction criteria met by individual patients by days 3, 7, and 28 demonstrated good to excellent discrimination for in-hospital mortality and compared favorably to the IPSCC criteria. Approximately 30% of patients met criteria for MOD on day 1 based on the PODIUM criteria and these had a 14-fold higher likelihood of dying than those who did not meet criteria. Meeting PODIUM criteria for MOD on day 1 was associated with higher sensitivity, lower specificity, and higher odds for in-hospital mortality than the IPSCC criteria. Finally, ∼18% of patients had new, progressive, or persistent MOD by day 7 based on the PODIUM criteria and these had a 48-fold higher likelihood of dying than those who did not meet criteria. Meeting PODIUM criteria for new, progressive, or persistent MOD by day 7 was also associated with higher sensitivity, lower specificity, and higher odds for in-hospital mortality than the IPSCC criteria.
Most of the difference we found in sensitivity and specificity between the PODIUM and IPSCC criteria is likely because PODIUM includes 9 organ dysfunctions versus the 6 proposed by IPSCC.6 The larger number and clearer delineation of organ dysfunctions in PODIUM, such as the addition of the immunologic and endocrine criteria and the distinction of the hematologic and coagulation dysfunctions, are desirable features in a set of organ dysfunction criteria. In recent years, there has been an increasing interest in understanding the effect that different organ dysfunction-based phenotypes have on outcome and response to therapy.13,21,22 That is, the focus has not been on describing the severity of organ dysfunctions alone, but rather the actual type and combination of organ dysfunctions and their impact on outcomes. Thus, having a larger set of organ dysfunctions with clearer delineations between organ systems is likely to benefit this important line of research. Additionally, the number of concurrent PODIUM organ dysfunction criteria met had a favorable prognostic performance compared with the IPSCC criteria, which is also a desirable characteristic given the strong association of organ dysfunction burden and mortality in critically ill children.
In our study, we show that progressive MOD is more frequent and is associated with higher mortality than new MOD, similar to previous analyses.23–25 Expanding on the MOD-based outcome literature, we also present the prevalence and associated mortality of persistent MOD, which captures patients with MOD on day 1 who, regardless of whether they develop progressive organ dysfunctions or not, are persistently affected by MOD a week after their presentation. Among all PICU patients in our study, almost 1 in 5 developed new, progressive, or persistent MOD during the first 7 days of stay, and those patients were almost 50 times more likely to die than those who either never had MOD or those who had nonprogressive, nonpersistent MOD. Using new, progressive, or persistent MOD based on the PODIUM criteria as an outcome in pediatric clinical trials appears promising, similar to what has been proposed and used previously.5,24,26
Our study has several strengths and limitations. Study data were derived from EHR databases corresponding to 2 large, urban, academic PICUs. Although the size and heterogeneity of the patient population is an advantage from a generalizability standpoint, our cohort may not be representative of all PICU populations, particularly in smaller, less urban, or nonacademic settings. Additionally, our aim was to evaluate the face validity, prognostic performance, and generalizability of the PODIUM criteria using structured EHR data, which we were able to achieve for a majority of the criteria. However, we did not evaluate a number of criteria, either because of the low number of patients with recorded values (eg, CD4+ T lymphocyte counts or total thyroxine levels) or the absence of the criteria as structured variables in the EHR (eg, electroencephalography results or type of respiratory failure). Some of these criteria may only be useful for studies with prospective data and blood specimen collection and should be further studied. However, we do highlight some of the variables that could be processed and included in EHRs in structured formats (eg, left ventricular ejection fraction from echocardiographic studies) to allow their integration with data-driven systems. As the development of data-driven systems continues to expand, work in this area is needed.
Conclusions
We present the first evaluation of the PODIUM organ dysfunction criteria in 2 large EHR databases. The prevalence and associated in-hospital mortality of the different PODIUM criteria is relatively similar across age groups and between the 2 PICUs in the study. The number of concurrent PODIUM organ dysfunction criteria demonstrated good to excellent discrimination for in-hospital mortality and compared favorably to the IPSCC organ dysfunction criteria. Similarly, MOD on day 1 and new, progressive, or persistent MOD by day 7 based on the PODIUM criteria demonstrated higher sensitivity and higher odds for in-hospital mortality than the IPSCC criteria. The use of the PODIUM organ dysfunction criteria appears promising for epidemiologic and clinical research studies using EHR data. More studies are needed to evaluate the PODIUM criteria that are not routinely collected in structured format or are infrequently recorded in EHR databases.
FUNDING: Dr Sanchez-Pinto receives support from the National Institutes of Health award number R21HD096402 and Dr Bembea receives support from the National Institutes of Health/National Institute of Neurological Disorders and Stroke award number R01NS106292 (MMB). Funded by the National Institutes of Health (NIH).
Dr Sanchez-Pinto conceptualized and designed the study, conducted the initial analyses, drafted the initial manuscript, and reviewed and revised the manuscript; Drs Bembea, Farris, Hartman, Odetola, Spaeder, Watson, Zimmerman, and Bennett conceptualized and designed the study and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
References
Competing Interests
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
Comments