BACKGROUND:

A 56 US hospital collaborative, Improving Pediatric Sepsis Outcomes, has developed variables, metrics and a data analysis plan to track quality improvement (QI)–based patient outcomes over time. Improving Pediatric Sepsis Outcomes expands on previous pediatric sepsis QI efforts by improving electronic data capture and uniformity across sites.

METHODS:

An expert panel developed metrics and corresponding variables to assess improvements across the care delivery spectrum, including the emergency department, acute care units, hematology and oncology, and the ICU. Outcome, process, and balancing measures were represented. Variables and statistical process control charts were mapped to each metric, elucidating progress over time and informing plan-do-study-act cycles. Electronic health record (EHR) abstraction feasibility was prioritized. Time 0 was defined as time of earliest sepsis recognition (determined electronically), or as a clinically derived time 0 (manually abstracted), identifying earliest physiologic onset of sepsis.

RESULTS:

Twenty-four evidence-based metrics reflected timely and appropriate interventions for a uniformly defined sepsis cohort. Metrics mapped to statistical process control charts with 44 final variables; 40 could be abstracted automatically from multiple EHRs. Variables, including high-risk conditions and bedside huddle time, were challenging to abstract (reported in <80% of encounters). Size or type of hospital, method of data abstraction, and previous QI collaboration participation did not influence hospitals’ abilities to contribute data. To date, 90% of data have been submitted, representing 200 007 sepsis episodes.

CONCLUSIONS:

A comprehensive data dictionary was developed for the largest pediatric sepsis QI collaborative, optimizing automation and ensuring sustainable reporting. These approaches can be used in other large-scale sepsis QI projects in which researchers seek to leverage EHR data abstraction.

What’s Known on This Subject:

Institutions have focused on small scale quality improvement efforts for pediatric sepsis. These efforts have revealed that defining the sepsis cohort uniformly and standardizing time 0 for a sepsis event is challenging. The optimal variables and metrics are not fully known.

What This Study Adds:

We have created a uniform sepsis cohort for quality improvement efforts across a large multi-institution collaborative with a standardized sepsis time 0. Optimal variables and metrics have been created and automated for centralized reporting and have demonstrated abstraction sustainability over time.

Severe sepsis is a leading cause of mortality for children worldwide, accounting for >8 million deaths annually.1  From 2004 to 2012, the prevalence of pediatric severe sepsis in the United States increased from 3.7% to 4.4%, with 176 000 hospitalizations and an 8.2% mortality.24  Despite the disease burden, significant gaps in ideal sepsis care exist.5  Ideal care has been recommended through various national and international organizations, including the National Institute for Health and Care excellence guidelines, the American College of Critical Care Medicine guidelines, and the Surviving Sepsis Campaign guidelines.68  Barriers in tertiary pediatric emergency department (ED) settings have resulted in poor adherence to rapid administration of intravenous (IV) fluids, vasoactive agents, and antibiotics.912  A recent New York state regulatory initiative demonstrated that, despite improvement in several sepsis-related measures across hospitals, only 25% of patient encounters had adherence to completion of a 1-hour sepsis treatment bundle.13  Time 0 (T0) definitions and the sepsis cohort were not standardized across hospitals, making it difficult to truly compare performance or delineate the optimal metrics for improving outcomes. A key component of ensuring delivery of quality care includes development of appropriate measures and variables that capture the key processes within a system.

In several previous quality improvement (QI) collaboratives, researchers have focused on pediatric sepsis care; however, a common challenge has been the establishment of standardized metrics across participating centers. In 2012, the Children’s Hospital Association (CHA) undertook a 12-hospital QI collaborative to understand the local burden of disease.14  A limited set of variables, corresponding metrics, and sepsis cohort definitions were not standardized, but rather were delegated to the individual institutions to determine definitions and identification strategies.

After this pilot sepsis collaborative, in 2016, the American Academy of Pediatrics launched the ED-focused Pediatric Severe Sepsis Collaborative (PSSC). This collaborative developed the first standard sepsis definitions for QI work, ensuring uniformity among the cohort, however, inconsistent reporting of 172 variables was reported (R.P., H.D., unpublished observations).

Few QI collaboratives have described their work in metric development. The Institute for Healthcare Improvement (Cambridge, MA) recommends an architecture building on that described by Donabedian.15  Collectively, these frameworks include outcome metrics (those that truly matter to the patient), process metrics (granular components of a system that directly influence the outcome metrics), and balancing metrics (those that ensure a QI intervention does not carry unintended consequences for patients and processes).

Once metrics are developed, specific variables that map to these metrics must be abstracted from the health system, predominantly from the electronic health record (EHR). Identifying factors that can affect reliable variable abstraction must be a priority for QI collaboratives. Such factors include institution size, electronic versus paper health records, EHR platform type, leadership support within the institution, and potential case prevalence at each site.16 

Outcome and process metrics must be effectively linked and are increasingly part of pay-for-performance efforts and accountable care organizations.1720  To this end, several groups have attempted to refine and create manageable sets of measures that best capture a disease process and have acceptable external validity.2125 

In 2016, CHA sponsored the Improving Pediatric Sepsis Outcomes (IPSO) QI collaborative with a goal of surmounting these challenges. This QI learning collaborative currently includes 56 hospitals and has developed comprehensive variables, metrics, and a data analysis plan to track QI-based patient outcomes over time. With IPSO, researchers aim to expand on previous pediatric sepsis QI efforts by increasing electronic data capture and uniformity across participating sites, not only in the ED setting, but also in other inpatient settings. Herein, we describe the process of metric and variable development for the IPSO collaborative.

Previous sepsis definitions from Goldstein et al,26  namely The International Pediatric Sepsis Consensus Conference definitions, were developed for research purposes and require intensive chart review for accurate abstraction, often resulting in delayed reporting, an experience reported by the CHA rapid cycle as well as the PSSC.14  As a QI collaborative anticipating a significant number of patients with disparate institutional resources, we prioritized automated patient identification to ensure timely and reliable data submission and to ensure sites could respond in real time to dynamic fluctuations in metric adherence.

IPSO developed and retrospectively analyzed 3 major cohorts of patients over time: IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis (Fig 1).27  Raw variable submission was provided by individual sites, and CHA was responsible for derived measures to have granular data and to ensure consistent calculations.

FIGURE 1

IPSO definitions for IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis cohorts: patient identification and stratification. a Treatment: antibiotic and 1 bolus and second bolus or pressor (all within 6 hours) and blood culture within 72 hours. ICD-10, International Classification of Diseases, 10th Revision.

FIGURE 1

IPSO definitions for IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis cohorts: patient identification and stratification. a Treatment: antibiotic and 1 bolus and second bolus or pressor (all within 6 hours) and blood culture within 72 hours. ICD-10, International Classification of Diseases, 10th Revision.

Close modal

Before initiation of the formal IPSO Collaborative, a steering committee and workgroups were assembled in 2015. Metric development initially included systematic review of existing evidence incorporated into an evidence table (Supplemental Information).28  This evidence table was then translated into a key driver diagram (KDD)28  and metrics were derived from this document, a methodology described by the Institute for Healthcare Improvement.2931  In cases in which evidence was not available, consensus was obtained from subject matter experts composing the 17-person data and analytics workgroup. Representation was present from across the sepsis care continuum (EDs, inpatient units, ICUs, hematology and oncology, infectious disease, and surgery). Stakeholders included experts in QI methodology and research and clinical pediatric sepsis care, including nurses, physicians, and infection control specialists, as well as CHA personnel, including administrative liaisons, QI consultants, data specialists, and statisticians.

A modified Delphi approach was used to define the cohort and to develop and vet metrics as delineated by the National Quality Forum.3234  This process was conducted over 9 virtual conferences and 4 in-person meetings, including 3 distinct voting rounds. An initial 41 metrics were posed to the group, ranked and reduced to 24 final metrics prioritizing those with supporting evidence, process metrics historically linked with improved outcomes, representation of the sepsis care continuum, generalizability, EHR abstraction feasibility, and categorization within process, outcome, and balancing domains.

In February 2020, the pediatric Surviving Sepsis Campaign guidelines were published, delineating that slower fluid and antibiotic administration may be appropriate for those with sepsis without organ dysfunction or shock.8  Hourly therapeutic targets for those who have septic shock have remained consistent. As such, our evidence table, recommendations, and collaborative goals changed to reflect differing targets for IPSO sepsis and IPSO noncritical and critical sepsis cohorts.

An initial outcome measure included death from any cause. Chart review at several sites determined that death attributed to sepsis represented significantly fewer cases than all-cause mortality. Therefore, 1 year into the collaborative, sites further determined which of their deaths were sepsis attributable on the basis of Goldstein definitions (as this was now retrospective review), and the data portal was altered to reflect this.

Previous local and collaborative QI work revealed that IV access difficulties did not substantially contribute to timely delivery of therapeutics and was unreliably reported.11  Therefore, this metric was not retained in last-round voting. Sites, however, were encouraged to drill down within their local process as they investigated delays in care.

We captured patients arriving from home and referring hospitals. As interventions from a referring center are difficult to glean accurately from manual chart review, these patients were excluded from final analysis. Emergency medical systems from the field may rarely initiate fluids; however, start and stop times are unreliably captured in the EHR.35  Analysis of these interventions would have been prohibitive on a large scale, given the extensive manual chart review needed.

T0 for time-bound metrics was defined as time of earliest possible sepsis recognition, as determined electronically, denoted as functional T0 (Fig 2). Because there is not yet a singular biomarker to identify sepsis, EHR surrogates, such as time a sepsis screen alerted, were used. Researchers at many sites felt that retrospective clinical chart review could better define a true physiologic T0, so this was offered as an optional variable to report and denoted as “clinically derived T0.” Calculations for performance metrics, however, were uniformly based on functional T0.

FIGURE 2

Functional T0 definition as automatically abstracted from the EHR was used in an attempt to identify the earliest point of sepsis recognition. Manually abstracted and clinically derived T0 supplemented this definition and was used to attempt to identify the earliest physiologic onset of sepsis.

FIGURE 2

Functional T0 definition as automatically abstracted from the EHR was used in an attempt to identify the earliest point of sepsis recognition. Manually abstracted and clinically derived T0 supplemented this definition and was used to attempt to identify the earliest physiologic onset of sepsis.

Close modal

Each site completed institutional regulatory requirements necessary for use of a limited data set for QI purposes, including an institutional review board approval at some sites and a participation agreement at all sites. Dates were the only protected health information submitted because they were needed to calculate several metrics, including age, unit transfer times, and outcome metric data. Several error alerts were built into the data portal to alert sites to variable discrepancies before submission (Supplemental Information). Additionally, discrepancies in incidence by month were resolved in real time to address cohort identification issues.

Continuous QI necessitates evaluation of process and outcomes over time. Therefore, statistical process control (SPC) charts were mapped to each metric and generated with a frequency dictated by the data.36  Most metrics were reported monthly, with rarer events, such as mortality metrics, reported quarterly. Percentage charts (p-charts) represented percent adherence to a certain goal; X bar charts were chosen for “time to” interventions and u-charts for rare outcomes, such as mortality.

Once sites submitted the baseline and first year of prospective data, they had access to centralized data portal reports managed by CHA as well as a third party (Prometheus Research, an IQVIA Company, New Haven, CT). Access to the reports was retained if the site maintained regular data submission. The portal provided visualization of individual and aggregate progress across the collaborative. Control charts allowed for monthly informing of plan-do-study-act cycles and benchmarking across sites.

We measured sites’ submission rate of variables to the portal. Sites were responsible for manual or electronic identification of the sepsis cohorts, followed by subsequent collection and submission of variables related to these cohorts. Although some sites abstracted data manually, others initially or eventually automated their abstraction efforts. We analyzed site characteristics that were associated with faster variable submission within the portal, including method of variable submission, size of hospital, freestanding children’s hospital versus not, and whether the site had participated in previous sepsis collaboratives. We used Wilcoxon rank sum statistics for 2 level comparisons or Kruskal-Wallis tests for >2 level comparisons.

On the basis of the KDD and the iterative process described above, metrics, variables and SPC charts were developed. Twenty-four metrics reflected both timely and appropriate interventions and included process, outcome and balancing metrics (Fig 3). A comprehensive data dictionary included 44 final variables, 40 of which could be abstracted automatically from the EHR, including Epic, Cerner, and Meditech platforms and were reliably reported for most encounters (Table 1). Most variables mapped to metrics, but some were abstracted to describe the patient population and inform future guidelines, such as serum lactate and hypotension time. Until December 2019, 90% of expected data have been submitted, representing 200 007 sepsis episodes (Fig 4).

FIGURE 3

ISPO collaborative metrics with measures, associated variables, and control charts.

FIGURE 3

ISPO collaborative metrics with measures, associated variables, and control charts.

Close modal
TABLE 1

Abridged Data Dictionary With Variables, Definitions, and Percentages Reported in the IPSO Sepsis Cohort

Variable (Ordered by Reporting Frequency)Data Dictionary Variable DefinitionIPSO Sepsis Cohort (Reported in Descending Order), %
Arrival time Time patient arrived at the ED or was admitted to the hospital directly 100.0 
Functional T0 See T0 diagram, Fig 1  99.3 
Bolus 1 time Time that first fluid bolus began; vol <5 mL/kg should not be counted as a bolus 98.4 
First antibiotic time Time that first IV antibiotic (antibacterial) was started, whether the antibiotic was effective or not; intramuscular or intraosseous used for sepsis also acceptable 97.5 
IV antibiotic days The No. days the patient is on IV antibiotics within the 30 d after T0; use whole numbers only; if the patient is on IV antibiotics for one-half a day, count as 1 d; if the patient is on multiple antibiotics, each antibiotic counts as a day (for example, 3 d of 2 antibiotics = 6 d) 97.1 
Bolus 3 volume Volume of the third bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 97.1 
Bolus 2 time Time that second fluid bolus began; vol <5 mL/kg should not be counted as a bolus 95.8 
Vasoactive agent type Indicates what kind of vasoactive agent was first used; the target time is within 24 h after T0 95.8 
Birthdate Patient’s date of birth 95.5 
First vasoactive time Time the first vasoactive agent was used 95.5 
ICU d No. days the patient was in the ICU for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include ICU days before T0; if the patient was not in the ICU, report 0 d 95.3 
Bolus 3 time Time that third fluid bolus began; vol <5 mL/kg should not be counted as a bolus 95.3 
Bolus 2 volume Volume of the second bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 94.4 
Time hematology or oncology from ED Time of patient transfer between hematology and oncology service and the ED 93.9 
Wt Patient’s wt in kilograms 93.5 
Time ICU from hematology or oncology Time of patient transfer between the ICU and hematology or oncology service 93.2 
Positive pressure ventilation d The No. days the patient was on ventilation for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; include days in which the patient had noninvasive ventilation, such as BIPAP or CPAP; do include ventilator days before T0 93.1 
Time ICU from general care Time of patient transfer between the ICU and acute care or general care service 93.1 
Vasoactive d No. days the patient was on pressors for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include vasoactive days before T0 93.0 
Lactic acid time Time first lactic acid was obtained; do not capture lactic acid from an outside hospital, only the first one at your institution 93.0 
Time general care from ED Time of patient transfer between the ED and the acute care or general care service 93.0 
Time ICU from ED Time of patient transfer between the ICU and the ED 92.9 
T0 location The location of the patient at the time of functional T0 92.8 
Order set Time of first use of either (1) any component of a severe sepsis order set or (2) a related infection‐specific order set that includes core severe sepsis components; for example, a pneumonia order set 92.4 
Bolus 1 volume Volume of the first bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 92.1 
Lactic acid value Initial lactic acid value 91.5 
Sepsis screen Time of initial screening process to identify possible severe sepsis in patient when screen result was positive; the initial screening process may consist of an electronic alert, a checklist, Pediatric Early Warning Scores (absolute value and/or change), bedside nursing screens, or other identification tools; the initial screening process may use paper‐based tools or electronic tools 91.3 
Extracorporeal membrane oxygenation Indicates if patient was placed on extracorporeal membrane oxygenation within 30 d of T0 88.9 
First hypotension time If the patient was hypotensive, this is the time of the first documented hypotension. Use what your institution classifies as hypotensive. 87.7 
Central venous line time Time of placement of central venous line in patient, if within time frame of 72 h before T0 to 72 h after T0 87.0 
Outside hospital specified Indicates if a patient arrived at your hospital from another hospital or facility in the 24 h before T0 86.7 
Blood culture result positive Indicates whether there was a positive blood culture result for the patient; blood culture results from an outside hospital or primary care provider can be used 85.7 
Chronically ventilated Indicates whether the patient was chronically ventilated or used BIPAP or CPAP as baseline for any portion of the day 84.5 
Time surgical source control Time of surgical procedure to control the source of the patient’s infection (including definitive management of abscess, peritonitis, line infection, bone or joint infection, empyema, or infected hardware); if the procedure was within 48 h before T0 to 48 h after T0 75.2 
Huddle time Time of sepsis team huddle to review clinical findings and determine if that patient is on a severe sepsis pathway in cases in which the huddle result was positive 74.2 
Mixed venous oxygen saturation time Time of obtaining mixed venous saturation for patient, if within time frame of 72 h (72 h before T0 to 72 h after T0) 69.2 
Organ dysfunction Indicate if and when organ dysfunction occurred after. Organ dysfunction would include ≥1 of the following: cardiovascular, respiratory, renal, hepatic, hematologic, or neurologic per Goldstein et al26  definitions. These data may require clinician review of the patient’s chart. 53.0 
High-risk condition other than not applicable The patient’s underlying high-risk conditions documented in ED or on admission; may require manual chart review to determine malignancy, asplenia (including sickle cell), bone marrow transplant, indwelling line or catheter, solid organ transplant, global developmental delay or cerebral palsy, immunocompromise or suppression, technology dependence (gastrostomy tube, tracheostomy tube, or ventriculoperitoneal shunt) 41.3 
Clinically derived T0 The date and time of the patient’s onset of physiologic sepsis, as determined by manual chart review; we recommend using the Goldstein et al26  definitions; this definition is different from functional T0 32.2 
Risk score Using your hospital’s standard risk-scoring tool, report the risk score for the patient. You may use the Pediatric Index of Mortality, PRISM, or another risk-scoring tool. 11.4 
Risk score method Name of the tool or scoring method used to assess a risk of mortality score 10.8 
Other International Classification of Diseases, 10th Revision sepsis codes A02.1, A20.7, A21.7, A22.7, A24.1, A26.7, A32.7, A39.2, A39.3, A39.4, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.50, A41.51, A41.52, A41.53, A41.59, A41.81, A41.89, A41.9, A42.7, A54.86, B00.7, B37.7 N/A 
International Classification of Diseases, 10th Revision severe sepsis, septic shock R65.20, R65.21 N/A 
Variable (Ordered by Reporting Frequency)Data Dictionary Variable DefinitionIPSO Sepsis Cohort (Reported in Descending Order), %
Arrival time Time patient arrived at the ED or was admitted to the hospital directly 100.0 
Functional T0 See T0 diagram, Fig 1  99.3 
Bolus 1 time Time that first fluid bolus began; vol <5 mL/kg should not be counted as a bolus 98.4 
First antibiotic time Time that first IV antibiotic (antibacterial) was started, whether the antibiotic was effective or not; intramuscular or intraosseous used for sepsis also acceptable 97.5 
IV antibiotic days The No. days the patient is on IV antibiotics within the 30 d after T0; use whole numbers only; if the patient is on IV antibiotics for one-half a day, count as 1 d; if the patient is on multiple antibiotics, each antibiotic counts as a day (for example, 3 d of 2 antibiotics = 6 d) 97.1 
Bolus 3 volume Volume of the third bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 97.1 
Bolus 2 time Time that second fluid bolus began; vol <5 mL/kg should not be counted as a bolus 95.8 
Vasoactive agent type Indicates what kind of vasoactive agent was first used; the target time is within 24 h after T0 95.8 
Birthdate Patient’s date of birth 95.5 
First vasoactive time Time the first vasoactive agent was used 95.5 
ICU d No. days the patient was in the ICU for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include ICU days before T0; if the patient was not in the ICU, report 0 d 95.3 
Bolus 3 time Time that third fluid bolus began; vol <5 mL/kg should not be counted as a bolus 95.3 
Bolus 2 volume Volume of the second bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 94.4 
Time hematology or oncology from ED Time of patient transfer between hematology and oncology service and the ED 93.9 
Wt Patient’s wt in kilograms 93.5 
Time ICU from hematology or oncology Time of patient transfer between the ICU and hematology or oncology service 93.2 
Positive pressure ventilation d The No. days the patient was on ventilation for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; include days in which the patient had noninvasive ventilation, such as BIPAP or CPAP; do include ventilator days before T0 93.1 
Time ICU from general care Time of patient transfer between the ICU and acute care or general care service 93.1 
Vasoactive d No. days the patient was on pressors for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include vasoactive days before T0 93.0 
Lactic acid time Time first lactic acid was obtained; do not capture lactic acid from an outside hospital, only the first one at your institution 93.0 
Time general care from ED Time of patient transfer between the ED and the acute care or general care service 93.0 
Time ICU from ED Time of patient transfer between the ICU and the ED 92.9 
T0 location The location of the patient at the time of functional T0 92.8 
Order set Time of first use of either (1) any component of a severe sepsis order set or (2) a related infection‐specific order set that includes core severe sepsis components; for example, a pneumonia order set 92.4 
Bolus 1 volume Volume of the first bolus in milliliters; vol <5 mL/kg should not be counted as a bolus 92.1 
Lactic acid value Initial lactic acid value 91.5 
Sepsis screen Time of initial screening process to identify possible severe sepsis in patient when screen result was positive; the initial screening process may consist of an electronic alert, a checklist, Pediatric Early Warning Scores (absolute value and/or change), bedside nursing screens, or other identification tools; the initial screening process may use paper‐based tools or electronic tools 91.3 
Extracorporeal membrane oxygenation Indicates if patient was placed on extracorporeal membrane oxygenation within 30 d of T0 88.9 
First hypotension time If the patient was hypotensive, this is the time of the first documented hypotension. Use what your institution classifies as hypotensive. 87.7 
Central venous line time Time of placement of central venous line in patient, if within time frame of 72 h before T0 to 72 h after T0 87.0 
Outside hospital specified Indicates if a patient arrived at your hospital from another hospital or facility in the 24 h before T0 86.7 
Blood culture result positive Indicates whether there was a positive blood culture result for the patient; blood culture results from an outside hospital or primary care provider can be used 85.7 
Chronically ventilated Indicates whether the patient was chronically ventilated or used BIPAP or CPAP as baseline for any portion of the day 84.5 
Time surgical source control Time of surgical procedure to control the source of the patient’s infection (including definitive management of abscess, peritonitis, line infection, bone or joint infection, empyema, or infected hardware); if the procedure was within 48 h before T0 to 48 h after T0 75.2 
Huddle time Time of sepsis team huddle to review clinical findings and determine if that patient is on a severe sepsis pathway in cases in which the huddle result was positive 74.2 
Mixed venous oxygen saturation time Time of obtaining mixed venous saturation for patient, if within time frame of 72 h (72 h before T0 to 72 h after T0) 69.2 
Organ dysfunction Indicate if and when organ dysfunction occurred after. Organ dysfunction would include ≥1 of the following: cardiovascular, respiratory, renal, hepatic, hematologic, or neurologic per Goldstein et al26  definitions. These data may require clinician review of the patient’s chart. 53.0 
High-risk condition other than not applicable The patient’s underlying high-risk conditions documented in ED or on admission; may require manual chart review to determine malignancy, asplenia (including sickle cell), bone marrow transplant, indwelling line or catheter, solid organ transplant, global developmental delay or cerebral palsy, immunocompromise or suppression, technology dependence (gastrostomy tube, tracheostomy tube, or ventriculoperitoneal shunt) 41.3 
Clinically derived T0 The date and time of the patient’s onset of physiologic sepsis, as determined by manual chart review; we recommend using the Goldstein et al26  definitions; this definition is different from functional T0 32.2 
Risk score Using your hospital’s standard risk-scoring tool, report the risk score for the patient. You may use the Pediatric Index of Mortality, PRISM, or another risk-scoring tool. 11.4 
Risk score method Name of the tool or scoring method used to assess a risk of mortality score 10.8 
Other International Classification of Diseases, 10th Revision sepsis codes A02.1, A20.7, A21.7, A22.7, A24.1, A26.7, A32.7, A39.2, A39.3, A39.4, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.50, A41.51, A41.52, A41.53, A41.59, A41.81, A41.89, A41.9, A42.7, A54.86, B00.7, B37.7 N/A 
International Classification of Diseases, 10th Revision severe sepsis, septic shock R65.20, R65.21 N/A 

BIPAP, bilevel positive airway pressure; CPAP, continuous positive airway pressure; N/A, not applicable.

FIGURE 4

A, Data submission over time for primary sepsis cohorts. B, Data submission over time for primary sepsis cohorts and subcohorts.

FIGURE 4

A, Data submission over time for primary sepsis cohorts. B, Data submission over time for primary sepsis cohorts and subcohorts.

Close modal

The data submission curve revealed a slow submission initially, a steady ramp-up phase, then a plateau. We observed a 6 to 9-month delay from the onset of the data portal opening to first submission of data and an 18-month delay before >80% of the data were submitted (Fig 4). Hospital size, electronic versus manual data abstraction, and previous sepsis QI work before IPSO were not associated with increased median time to >80% of data submission (Table 2). At collaborative onset, the percentage of sites reporting good to excellent personnel support in the following domains was as follows: clinical informatics (78%), data analyst or statistician (71%), QI consultant (82%), data entry personnel (100%), and project manager (3%). With the onset of COVID-19 in the Unites States, sites maintained similar levels of commitment, although clinical informatics full support dropped to 56%.

TABLE 2

Characteristics of Hospital Sites With Rate of Data Submission (48 Sites Analyzed)

Site CharacteristicNo. SitesMedian No. Months to >80% Data Submission25th Percentile75th PercentileP
Hospital bed count     .765 
 <100 15 20  
 100–199 12 16 12 19.5  
 200–299 10 15 12 19  
 300+ 17 16 15 18  
Previous QI collaborative participation     .404 
 Yes 18 16.5 15 20  
 No 30 15 19  
Date capture process     .578 
 Electronic 24 16 15 19.5  
 Manual initially, then electronic 13 10 20  
 Manual 16 16 11 21.5  
Hospital type     .655 
 Freestanding children’s hospital 23 16 13 20  
 Not freestanding 25 15 11 19  
Site CharacteristicNo. SitesMedian No. Months to >80% Data Submission25th Percentile75th PercentileP
Hospital bed count     .765 
 <100 15 20  
 100–199 12 16 12 19.5  
 200–299 10 15 12 19  
 300+ 17 16 15 18  
Previous QI collaborative participation     .404 
 Yes 18 16.5 15 20  
 No 30 15 19  
Date capture process     .578 
 Electronic 24 16 15 19.5  
 Manual initially, then electronic 13 10 20  
 Manual 16 16 11 21.5  
Hospital type     .655 
 Freestanding children’s hospital 23 16 13 20  
 Not freestanding 25 15 11 19  

“Sepsis-attributable death” was a midcollaborative added variable. The PSSC had previously revealed that a midcollaborative change in the data portal resulted in a 1-year delay in reporting of modified variables (R.P., H.D., unpublished observations). Here, 95% of sites were able to provide this variable within 5 months.

32% of patients had a clinically derived T0 reported. Clinically derived T0 was the same as functional T0 in 7675 episodes and was earlier in 2369 episodes, with an overall median difference of 0 minutes (interquartile range: 0–19).

We have developed a set of measures with related variables and SPC charts to better understand the spectrum of pediatric sepsis care and track improvement using a QI learning collaborative model across 56 hospitals. We found that centers could reliably and consistently provide data for 24 established metrics to define >200 000 sepsis episodes in pediatric patients.

Identification of a sepsis patient cohort is a challenge, given the unique difficulties in defining sepsis. Unlike medical conditions that adhere to a narrow clinical phenotype, such as asthma, sepsis is heterogeneous and often subtle in initial presentation, making recognition and retrospective identification difficult, leading to unreliable coding.37  Furthermore, there is significant overlap with other conditions, making automated identification problematic.

Many sites struggled with whether to exclude patients who met the strict definition of IPSO sepsis but on chart review did not appear to have sepsis (eg, patients with transient hypotension due to seizure or those receiving multiple boluses as pretreatment of chemotherapy). The collaborative was aware that the IPSO sepsis definition was designed to be sensitive to mitigate missed case identification, yet also understood that specificity was able to be captured within the IPSO critical sepsis definition. Therefore, sites were instructed to remain consistent in their identification methods over time, with emphasis placed on adherence to the a priori, broader identification criteria.

A myriad of reasons resulted in delayed variable reporting. We noted that sites that used manual reporting had earlier submission of data, with the first site to submit data relying on manual abstraction alone. However, once sites that automated their process established their data pipeline, they were able to submit data in a complete and timely fashion thereafter. Sites spent considerable time investing in data analytic resources to create the coding structure for automated abstraction. Once created, clinical review ensured validation of the code.

Coding for abstraction of many sepsis variables, such as chronic ventilation, organ dysfunction, and sepsis screen time, was time-consuming and onerous. Bedside huddle time was reported as particularly problematic to abstract, given it is an often-performed but undocumented process and thereby difficult to capture in an automated fashion. Chart review was often needed to capture a huddle because natural language processing was not used in abstraction. Sites that are beginning their efforts should focus on creating discrete flowsheet rows within their EHR for processes such as huddle time.

Most sites had a primary sepsis QI champion responsible for early and sustainable data submission. When primary champions left the institution or were reassigned to other roles, a temporary lag in data submission ensued. Attention should be paid to developing infrastructure that can function despite primary champions transitioning.

Several nonfreestanding sites reported that they were mirroring their efforts on previous adult-based sepsis QI projects. Doing so proved to be problematic and a less sustainable strategy, given disparate therapeutic recommendations for pediatric and adult sepsis and the increased prevalence of sepsis in adults.38  For example, adult sepsis QI efforts abstract time to 30 mL/kg of IV fluids and place a larger emphasis on certain variables, such as lactate.

It was hypothesized that sites that had participated in previous sepsis QI collaboratives or had completed significant institutional sepsis QI work would be able to submit data sooner than those that had not. New York State passed Rory’s Regulations in 2013 to ensure that all hospitals caring for pediatric patients had sepsis QI protocols and submitted data to the state.39  Because many of the variables reported were similar to those reported for IPSO, these sites could potentially report data earlier because of personnel, data infrastructure, and executive support already in place. Unfortunately, this hypothesis was unfounded. Sites reported that the definitions and variables needed for IPSO submission were substantially different from those needed for previous work and thus time to submission could not be expedited. Having a robust registry already in place at times hindered the ability to change one’s workflow and to modify alignment of data abstraction with the IPSO collaborative.

Historically, identifying and automating T0 for sepsis onset has posed a significant challenge.11,12,14  Because of the resources required for retrospective chart review for clinically derived T0, this variable proved to be underreported. In the majority of reported cases, clinically derived T0 and functional T0 were equal, but further investigation is needed for the cases in which they differed.

Outcome variables can be sensitive to individual patient factors, such as severity of illness and idiosyncratic patient physiology. We thus included variables allowing for risk stratification, such as the Pediatric Risk of Mortality Score (PRISM).40  The PRISM was being reported by PICUs before IPSO through the Virtual PICU Systems initiative (VPS, LLC, Los Angeles, CA). Sites could also optionally calculate severity scores for patients in their initial presenting hours of non-PICU care. Because of the labor-intensive nature of calculating this score, this value was also rarely reported.

We had initially facilitated the data abstraction process by streamlining our variables, such as presence or absence of organ dysfunction. We felt collecting specific systems of organ dysfunction present for a single encounter would be too onerous. In retrospect, inclusion of this granular data would have aided in future analysis and better aligned with current research.

Variables, metrics and control charts to measure recognition, appropriate therapeutic interventions, and outcomes across 56 sites were created for the largest pediatric sepsis QI collaborative. Variables were amenable to automated and sustainable abstraction. Size or type of hospital, method of data abstraction, and previous QI collaboration participation did not influence hospitals’ abilities to contribute meaningful data. Sites initiating and optimizing their sepsis QI journey can benefit from these efforts in the future.

The following are IPSO Collaborative investigators:

  • Audrey H Barnett, MSN, RN, RNC-NIC, CPHQ, Department of Quality and Safety, Children’s Memorial Hermann Hospital, Houston, Texas;

  • Sopnil N Bhattarai, CPHQ, Department of Performance Improvement, Children’s National Hospital, Washington, District of Columbia;

  • Michael T Bigham, MD, FAAP, FCCM, Critical Care Medicine, Akron Children’s Hospital, Akron, Ohio, Associate Professor, Pediatrics, Northeast Ohio Medical University, Rootstown, Ohio, Akron, Ohio;

  • Kristi Booker, RN, MSN, CDN, Quality Department, Children’s Hospital at University of Oklahoma Medicine, Oklahoma City, Oklahoma;

  • Renee M Bruhn, ME, Center for Healthcare Quality and Analytics, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania;

  • David G Bundy, MD, MPH, Department of Pediatrics, Medical University of South Carolina, Charleston, South Carolina;

  • Stephanie A. Burrus, DO, Department of Pediatrics, Children’s Mercy Kansas City, Division of Pediatric Hospital Medicine, Children’s Mercy Kansas City, Kansas City, Missouri;

  • Pavan Kumar Chundi, MS, James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio;

  • Emily C Dawson, MD, Department of Pediatrics, Advocate Children’s Hospital, Section of Pediatric Critical Care and Pediatric Emergency Medicine, Oak Lawn, Illinois;

  • Theodore K M DeMartini, MD, Department of Pediatrics, Penn State University, Division of Pediatric Critical Care Medicine, Penn State Children’s Hospital, Hershey, Pennsylvania;

  • Susan J Duffy, MD, MPH, Department of Emergency Medicine and Pediatrics Alpert Medical School, Brown University, Pediatric Emergency Medicine, Hasbro Children’s Hospital, Providence, Rhode Island;

  • Jill B Dykstra Nykanen, RN, MSN, CPHQ, Department of Pediatrics, Orlando Health Arnold Palmer Hospital for Children, Orlando, Florida;

  • Julie C Fitzgerald, MD, PhD, MSCE, Department of Anesthesiology and Critical Care, University of Pennsylvania Perelman School of Medicine, Department of Anesthesiology and Critical Care, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania;

  • Meg Frizzola, DO, FAAP, Department of Pediatrics, Nemours/Alfred I. duPont Hospital for Children, Department of Pediatrics, Thomas Jefferson University, Wilmington, Delaware;

  • Elizabeth J. Haines, DO, MAS, FACEP, Ronald O. Perelman Department of Emergency Medicine, NYU Langone Health, Hassenfeld Children’s Hospital, NYU Langone Health, New York, New York;

  • Hana Hakim, MD, MS, Department of Infectious Diseases, St. Jude Children’s Research Hospital, Memphis, Tennessee;

  • Lauren M Hess, MD, Department of Pediatrics, Baylor College of Medicine, Section of Hospital Medicine, Texas Children’s Hospital, Houston, Texas;

  • Sarah B Kandil, MD, Department of Pediatrics, Yale University, School of Medicine, Section of Critical Care Medicine, Yale New Haven Children’s Hospital, New Haven, Connecticut;

  • Kimberly D Kato, RN, BSN, MS, Clinical Effectiveness and Improvement, Ann and Robert H. Lurie Children’s Hospital of Chicago; Chicago, Illinois;

  • Daniel P Kelly, MD, Department of Pediatrics, Division of Medical Critical Care, Boston Children’s Hospital, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts;

  • Raed M. Khoury, MA, MPH, CIC, MT (ASCP), CLS (CA), ARM, CPHQ, CHSP, CJCP, CSSBB, FAPIC, Quality & Patient Safety and Medical Affairs, Madera, California;

  • V. Matt Laurich, MD, Department of Pediatrics, University of Connecticut, Division of Pediatric Emergency Medicine, Connecticut Children’s, Hartford, Connecticut;

  • Stephanie M. Lavin, MSN, RN, CPN, Clinical Collaborative, Cook Children’s Medical Center, Ft Worth, Texas;

  • Jennifer L Liedel, MD, Department of Pediatrics, Nemours Children’s Hospital, Divisions of Critical Care and Neonatology, Nemours Children’s Hospital, Orlando, Florida;

  • Jeremy M Loberger, MD, Department of Pediatrics, Division of Pediatric Critical Care Medicine, University of Alabama at Birmingham, Birmingham, Alabama;

  • Justin M Lockwood, MD, MSCS, Department of Pediatrics, Section of Hospital Medicine, University of Colorado School of Medicine, Aurora, Colorado;

  • Merrick R Lopez, MD, FAAP, Department of Pediatrics, Loma Linda University Children’s Hospital, Section of Pediatric Critical Care, Loma Linda University Children’s Hospital, Loma Linda, California;

  • Erica A Michiels, MD, Department of Emergency Medicine, Helen DeVos Children’s Hospital, Department of Emergency Medicine, Michigan State University, Grand Rapids, Michigan;

  • Kristina J Murphy, DO, Department of Pediatric Critical Care, Cohen Children’s Medical Center, New Hyde Park, New York;

  • Gregory P Priebe, MD, Department of Anesthesiology, Critical Care and Pain Medicine, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts;

  • Sandhya Ramachandran, MPH, QI Services, Nationwide Children’s Hospital, Columbus, Ohio;

  • Faisal Razzaqi, MD, Valley Children’s Hospital, Madera, California;

  • Johanna R Rosen, MD, Department of Pediatrics, University of Pittsburgh School of Medicine, Division of Emergency Medicine, UPMC Children’s Hospital of Pittsburgh, Pittsburgh, Pennsylvania;

  • Lori E Rutman, MD, MPH, Department of Pediatrics, University of Washington, Division of Emergency Medicine, Seattle Children’s Hospital, Seattle, Washington;

  • Amanda M Sebring, MD, Department of Pediatrics, Levine Children’s Hospital, Section of Pediatric Critical Care, Levine Children’s Hospital, Charlotte, North Carolina;

  • Jonathan A Silverman, MD, MPH, Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Division of Pediatric Emergency Medicine, Children’s Hospital of Richmond, Richmond, Virginia;

  • Erika L Stalets, MD, MS, Department of Pediatrics, University of Cincinnati College of Medicine, Division of Critical Care Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio;

  • Nathan E Thompson, MD, PharmD, MS, Department of Pediatrics, Medical College of Wisconsin, Section of Critical Care Medicine, Children’s Wisconsin, Milwaukee, Wisconsin;

  • Zebulon J Timmons, MD, FAAP, Department of Pediatrics, Phoenix Children’s Hospital, Section of Pediatric Emergency Medicine, Phoenix Children’s Hospital, Phoenix, Arizona;

  • Jennifer J Wilkes, MD MSCE, Department of Pediatrics, University of Washington, Division of Cancer and Blood Disorders, Seattle, Washington; and

  • Jennifer K Workman, MD, MSCI, Department of Pediatrics, University of Utah, Division of Pediatric Critical Care, Primary Children’s Hospital, Salt Lake City, Utah.

Individual participant data that underlie the results reported in this article, after deidentification, including the study protocol and statistical analytic plan, will be available beginning 9 months and ending 36 months after article publication and collaborative completion for researchers who provide a methodologically sound proposal to achieve aims in the approved proposal. Proposals should be directed to raina.paul@aah.org. A data access agreement will need to be signed, and data will be available for 3 years.

Drs Paul, Auletta, and Richardson conceptualized and designed the study, drafted the initial manuscript, designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Drs Niedner, Brilli, Macias, Balamuth, Depinet, Larsen, Huskins, Scott, and Schaffer conceptualized and designed the study, drafted the initial manuscript, and reviewed and revised the manuscript; Ms Lucasiewicz, Ms Riggs, and Ms DeSouza designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Drs Silver and Hueschen, Ms Campbell, and Ms Wathen critically reviewed the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Children’s Hospital Association and the Improving Pediatric Sepsis Outcomes Collaborative. Dr Scott’s institution is currently receiving career development salary support from the Agency for Healthcare Research and Quality.

     
  • CHA

    Children’s Hospital Association

  •  
  • ED

    emergency department

  •  
  • EHR

    electronic health record

  •  
  • IPSO

    Improving Pediatric Sepsis Outcomes

  •  
  • IV

    intravenous

  •  
  • KDD

    key driver diagram

  •  
  • PRISM

    Pediatric Risk of Mortality Score

  •  
  • PSSC

    Pediatric Severe Sepsis Collaborative

  •  
  • QI

    quality improvement

  •  
  • SPC

    statistical process control

  •  
  • T0

    time 0

1
Black
RE
,
Cousens
S
,
Johnson
HL
, et al.;
Child Health Epidemiology Reference Group of WHO and UNICEF
.
Global, regional, and national causes of child mortality in 2008: a systematic analysis
.
Lancet
.
2010
;
375
(
9730
):
1969
1987
2
Hartman
ME
,
Linde-Zwirble
WT
,
Angus
DC
,
Watson
RS
.
Trends in the epidemiology of pediatric severe sepsis
.
Pediatr Crit Care Med
.
2013
;
14
(
7
):
686
693
3
Balamuth
F
,
Weiss
SL
,
Neuman
MI
, et al
.
Pediatric severe sepsis in U.S. children’s hospitals
.
Pediatr Crit Care Med
.
2014
;
15
(
9
):
798
805
4
Gaieski
DF
,
Edwards
JM
,
Kallan
MJ
,
Carr
BG
.
Benchmarking the incidence and mortality of severe sepsis in the United States
.
Crit Care Med
.
2013
;
41
(
5
):
1167
1174
5
Oliveira
CF
,
Nogueira de Sá
FR
,
Oliveira
DS
, et al
.
Time- and fluid-sensitive resuscitation for hemodynamic support of children in septic shock: barriers to the implementation of the American College of Critical Care Medicine/Pediatric Advanced Life Support Guidelines in a pediatric intensive care unit in a developing world
.
Pediatr Emerg Care
.
2008
;
24
(
12
):
810
815
6
National Institute for Health and Care Excellence
.
Sepsis: Recognition, Diagnosis and Early Management
.
London, United Kingdom
:
National Institute for Health and Care Excellence
;
2016
7
Davis
AL
,
Carcillo
JA
,
Aneja
RK
, et al
.
American College of Critical Care Medicine clinical practice parameters for hemodynamic support of pediatric and neonatal septic shock. [published correction appears in Crit Care Med. 2017;45(9):e993]
.
Crit Care Med
.
2017
;
45
(
6
):
1061
1093
8
Weiss
SL
,
Peters
MJ
,
Alhazzani
W
, et al
.
Surviving sepsis campaign international guidelines for the management of septic shock and sepsis-associated organ dysfunction in children
.
Pediatr Crit Care Med
.
2020
;
21
(
2
):
e52
e106
9
Cruz
AT
,
Perry
AM
,
Williams
EA
,
Graf
JM
,
Wuestner
ER
,
Patel
B
.
Implementation of goal-directed therapy for children with suspected sepsis in the emergency department
.
Pediatrics
.
2011
;
127
(
3
).
10
Lane
RD
,
Funai
T
,
Reeder
R
,
Larsen
GY
.
High reliability pediatric septic shock quality improvement initiative and decreasing mortality
.
Pediatrics
.
2016
;
138
(
4
):
e20154153
11
Paul
R
,
Melendez
E
,
Stack
A
,
Capraro
A
,
Monuteaux
M
,
Neuman
MI
.
Improving adherence to PALS septic shock guidelines
.
Pediatrics
.
2014
;
133
(
5
).
12
Paul
R
,
Neuman
MI
,
Monuteaux
MC
,
Melendez
E
.
Adherence to PALS sepsis guidelines and hospital length of stay
.
Pediatrics
.
2012
;
130
(
2
).
13
Evans
IVR
,
Phillips
GS
,
Alpern
ER
, et al
.
Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis
.
JAMA
.
2018
;
320
(
4
):
358
367
14
Paul
R
,
Melendez
E
,
Wathen
B
, et al
.
A quality improvement collaborative for pediatric sepsis: lessons learned
.
Pediatr Qual Saf
.
2017
;
3
(
1
):
e051
15
Donabedian
A
.
The quality of care. How can it be assessed?
JAMA
.
1988
;
260
(
12
):
1743
1748
16
Delaney
AE
,
Dadlez
NM
,
Marshall
AC
.
Alternative approach to pediatric cardiac quality assessment for low-volume centers
.
Congenit Heart Dis
.
2019
;
14
(
4
):
665
670
17
Kampstra
NA
,
Zipfel
N
,
van der Nat
PB
,
Westert
GP
,
van der Wees
PJ
,
Groenewoud
AS
.
Health outcomes measurement and organizational readiness support quality improvement: a systematic review
.
BMC Health Serv Res
.
2018
;
18
(
1
):
1005
18
Epstein
AM
.
Pay for performance at the tipping point
.
N Engl J Med
.
2007
;
356
(
5
):
515
517
19
Fisher
ES
,
Shortell
SM
.
Accountable care organizations: accountable for what, to whom, and how
.
JAMA
.
2010
;
304
(
15
):
1715
1716
20
Porter
ME
.
What is value in health care?
N Engl J Med
.
2010
;
363
(
26
):
2477
2481
21
Nowels
D
,
Kamerow
DB
.
New “core quality measures”: only a beginning
.
J Am Board Fam Med
.
2017
;
30
(
1
):
4
7
22
Ahmed
S
,
Siegel
CA
,
Melmed
GY
.
Implementing quality measures for inflammatory bowel disease
.
Curr Gastroenterol Rep
.
2015
;
17
(
4
):
14
23
Fifolt
M
,
Preskitt
J
,
Rucks
A
,
Corvey
K
,
Benton
EC
.
Promoting continuous quality improvement in the Alabama child health improvement alliance through Q-sort methodology and learning collaboratives
.
Qual Manag Health Care
.
2017
;
26
(
1
):
33
39
24
Ku
BC
,
Chamberlain
JM
,
Shaw
KN
.
Quality improvement and safety in pediatric emergency medicine
.
Pediatr Clin North Am
.
2018
;
65
(
6
):
1269
1281
25
Crandall
WV
,
Boyle
BM
,
Colletti
RB
,
Margolis
PA
,
Kappelman
MD
.
Development of process and outcome measures for improvement: lessons learned in a quality improvement collaborative for pediatric inflammatory bowel disease
.
Inflamm Bowel Dis
.
2011
;
17
(
10
):
2184
2191
26
Goldstein
B
,
Giroir
B
,
Randolph
A
;
International Consensus Conference on Pediatric Sepsis
.
International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics
.
Pediatr Crit Care Med
.
2005
;
6
(
1
):
2
8
27
Scott
HF
,
Brilli
RJ
,
Paul
R
, et al.;
Improving Pediatric Sepsis Outcomes (IPSO) Collaborative Investigators.
.
Evaluating pediatric sepsis definitions designed for electronic health record extraction and multicenter quality improvement
.
Crit Care Med
.
2020
;
48
(
10
):
e916
e926
28
Larsen
GY
,
Brilli
R
,
Macias
CG
, et al.;
Improving Pediatric Sepsis Outcomes Collaborative Investigators
.
Development of a quality improvement learning collaborative to improve pediatric sepsis outcomes
.
Pediatrics
.
2021
;
147
(
1
):
e20201434
29
James
B
.
Information system concepts for quality measurement
.
Med Care
.
2003
;
41
(
suppl 1
):
I71
I79
30
Berwick
DM
,
James
B
,
Coye
MJ
.
Connections between quality measurement and improvement
.
Med Care
.
2003
;
41
(
suppl 1
):
I30
I38
31
Lloyd
R
.
Intro to Driver Diagrams
.
Cambridge, MA
:
Institute for Healthcare Improvement
.
Available at: https://www.youtube.com/watch?v=A2491BJcyXA. Accessed June 1, 2020
32
Boulkedid
R
,
Abdoul
H
,
Loustau
M
,
Sibony
O
,
Alberti
C
.
Using and reporting the Delphi method for selecting healthcare quality indicators: a systematic review
.
PLoS One
.
2011
;
6
(
6
):
e20476
33
Glance
LG
,
Maddox
KJ
,
Johnson
K
, et al
.
National quality forum guidelines for evaluating the scientific acceptability of risk-adjusted clinical outcome measures: a report from the national quality forum scientific methods panel
.
Ann Surg
.
2020
;
271
(
6
):
1048
1055
34
National Quality Forum
.
National Quality Forum issue brief: strengthening pediatric quality measurement and reporting
.
J Healthc Qual
.
2008
;
30
(
3
):
51
55
35
Seymour
CW
,
Rea
TD
,
Kahn
JM
,
Walkey
AJ
,
Yealy
DM
,
Angus
DC
.
Severe sepsis in pre-hospital emergency care: analysis of incidence, care, and outcome
.
Am J Respir Crit Care Med
.
2012
;
186
(
12
):
1264
1271
36
Provost
LP
,
Murray
SK
.
The Health Care Data Guide
, 1st ed.
San Francisco, CA
:
Josey-Bass
;
2011
37
Balamuth
F
,
Weiss
SL
,
Hall
M
, et al
.
Identifying pediatric severe sepsis and septic shock: accuracy of diagnosis codes
.
J Pediatr
.
2015
;
167
(
6
):
1295
300.e4
38
Dellinger
RP
,
Levy
MM
,
Rhodes
A
, et al.;
Surviving Sepsis Campaign Guidelines Committee including the Pediatric Subgroup
.
Surviving sepsis campaign: international guidelines for management of severe sepsis and septic shock: 2012
.
Crit Care Med
.
2013
;
41
(
2
):
580
637
39
Centers for Disease Control and Prevention
.
State policy approaches to sepsis prevention and early recognition.
Available at: https://www.cdc.gov/hai/pdfs/sepsis/VS-Sepsis-Policy-FINAL.pdf. Accessed May 1, 2020
40
Pollack
MM
,
Holubkov
R
,
Funai
T
, et al.;
Eunice Kennedy Shriver National Institute of Child Health and Human Development Collaborative Pediatric Critical Care Research Network
.
The pediatric risk of mortality score: update 2015
.
Pediatr Crit Care Med
.
2016
;
17
(
1
):
2
9

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data