A 56 US hospital collaborative, Improving Pediatric Sepsis Outcomes, has developed variables, metrics and a data analysis plan to track quality improvement (QI)–based patient outcomes over time. Improving Pediatric Sepsis Outcomes expands on previous pediatric sepsis QI efforts by improving electronic data capture and uniformity across sites.
An expert panel developed metrics and corresponding variables to assess improvements across the care delivery spectrum, including the emergency department, acute care units, hematology and oncology, and the ICU. Outcome, process, and balancing measures were represented. Variables and statistical process control charts were mapped to each metric, elucidating progress over time and informing plan-do-study-act cycles. Electronic health record (EHR) abstraction feasibility was prioritized. Time 0 was defined as time of earliest sepsis recognition (determined electronically), or as a clinically derived time 0 (manually abstracted), identifying earliest physiologic onset of sepsis.
Twenty-four evidence-based metrics reflected timely and appropriate interventions for a uniformly defined sepsis cohort. Metrics mapped to statistical process control charts with 44 final variables; 40 could be abstracted automatically from multiple EHRs. Variables, including high-risk conditions and bedside huddle time, were challenging to abstract (reported in <80% of encounters). Size or type of hospital, method of data abstraction, and previous QI collaboration participation did not influence hospitals’ abilities to contribute data. To date, 90% of data have been submitted, representing 200 007 sepsis episodes.
A comprehensive data dictionary was developed for the largest pediatric sepsis QI collaborative, optimizing automation and ensuring sustainable reporting. These approaches can be used in other large-scale sepsis QI projects in which researchers seek to leverage EHR data abstraction.
Institutions have focused on small scale quality improvement efforts for pediatric sepsis. These efforts have revealed that defining the sepsis cohort uniformly and standardizing time 0 for a sepsis event is challenging. The optimal variables and metrics are not fully known.
We have created a uniform sepsis cohort for quality improvement efforts across a large multi-institution collaborative with a standardized sepsis time 0. Optimal variables and metrics have been created and automated for centralized reporting and have demonstrated abstraction sustainability over time.
Severe sepsis is a leading cause of mortality for children worldwide, accounting for >8 million deaths annually.1 From 2004 to 2012, the prevalence of pediatric severe sepsis in the United States increased from 3.7% to 4.4%, with 176 000 hospitalizations and an 8.2% mortality.2–4 Despite the disease burden, significant gaps in ideal sepsis care exist.5 Ideal care has been recommended through various national and international organizations, including the National Institute for Health and Care excellence guidelines, the American College of Critical Care Medicine guidelines, and the Surviving Sepsis Campaign guidelines.6–8 Barriers in tertiary pediatric emergency department (ED) settings have resulted in poor adherence to rapid administration of intravenous (IV) fluids, vasoactive agents, and antibiotics.9–12 A recent New York state regulatory initiative demonstrated that, despite improvement in several sepsis-related measures across hospitals, only 25% of patient encounters had adherence to completion of a 1-hour sepsis treatment bundle.13 Time 0 (T0) definitions and the sepsis cohort were not standardized across hospitals, making it difficult to truly compare performance or delineate the optimal metrics for improving outcomes. A key component of ensuring delivery of quality care includes development of appropriate measures and variables that capture the key processes within a system.
In several previous quality improvement (QI) collaboratives, researchers have focused on pediatric sepsis care; however, a common challenge has been the establishment of standardized metrics across participating centers. In 2012, the Children’s Hospital Association (CHA) undertook a 12-hospital QI collaborative to understand the local burden of disease.14 A limited set of variables, corresponding metrics, and sepsis cohort definitions were not standardized, but rather were delegated to the individual institutions to determine definitions and identification strategies.
After this pilot sepsis collaborative, in 2016, the American Academy of Pediatrics launched the ED-focused Pediatric Severe Sepsis Collaborative (PSSC). This collaborative developed the first standard sepsis definitions for QI work, ensuring uniformity among the cohort, however, inconsistent reporting of 172 variables was reported (R.P., H.D., unpublished observations).
Few QI collaboratives have described their work in metric development. The Institute for Healthcare Improvement (Cambridge, MA) recommends an architecture building on that described by Donabedian.15 Collectively, these frameworks include outcome metrics (those that truly matter to the patient), process metrics (granular components of a system that directly influence the outcome metrics), and balancing metrics (those that ensure a QI intervention does not carry unintended consequences for patients and processes).
Once metrics are developed, specific variables that map to these metrics must be abstracted from the health system, predominantly from the electronic health record (EHR). Identifying factors that can affect reliable variable abstraction must be a priority for QI collaboratives. Such factors include institution size, electronic versus paper health records, EHR platform type, leadership support within the institution, and potential case prevalence at each site.16
Outcome and process metrics must be effectively linked and are increasingly part of pay-for-performance efforts and accountable care organizations.17–20 To this end, several groups have attempted to refine and create manageable sets of measures that best capture a disease process and have acceptable external validity.21–25
In 2016, CHA sponsored the Improving Pediatric Sepsis Outcomes (IPSO) QI collaborative with a goal of surmounting these challenges. This QI learning collaborative currently includes 56 hospitals and has developed comprehensive variables, metrics, and a data analysis plan to track QI-based patient outcomes over time. With IPSO, researchers aim to expand on previous pediatric sepsis QI efforts by increasing electronic data capture and uniformity across participating sites, not only in the ED setting, but also in other inpatient settings. Herein, we describe the process of metric and variable development for the IPSO collaborative.
Methods
Definitions
Previous sepsis definitions from Goldstein et al,26 namely The International Pediatric Sepsis Consensus Conference definitions, were developed for research purposes and require intensive chart review for accurate abstraction, often resulting in delayed reporting, an experience reported by the CHA rapid cycle as well as the PSSC.14 As a QI collaborative anticipating a significant number of patients with disparate institutional resources, we prioritized automated patient identification to ensure timely and reliable data submission and to ensure sites could respond in real time to dynamic fluctuations in metric adherence.
IPSO developed and retrospectively analyzed 3 major cohorts of patients over time: IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis (Fig 1).27 Raw variable submission was provided by individual sites, and CHA was responsible for derived measures to have granular data and to ensure consistent calculations.
IPSO definitions for IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis cohorts: patient identification and stratification. a Treatment: antibiotic and 1 bolus and second bolus or pressor (all within 6 hours) and blood culture within 72 hours. ICD-10, International Classification of Diseases, 10th Revision.
IPSO definitions for IPSO suspected infection, IPSO sepsis, and IPSO critical sepsis cohorts: patient identification and stratification. a Treatment: antibiotic and 1 bolus and second bolus or pressor (all within 6 hours) and blood culture within 72 hours. ICD-10, International Classification of Diseases, 10th Revision.
Variable and Metric Selection
Before initiation of the formal IPSO Collaborative, a steering committee and workgroups were assembled in 2015. Metric development initially included systematic review of existing evidence incorporated into an evidence table (Supplemental Information).28 This evidence table was then translated into a key driver diagram (KDD)28 and metrics were derived from this document, a methodology described by the Institute for Healthcare Improvement.29–31 In cases in which evidence was not available, consensus was obtained from subject matter experts composing the 17-person data and analytics workgroup. Representation was present from across the sepsis care continuum (EDs, inpatient units, ICUs, hematology and oncology, infectious disease, and surgery). Stakeholders included experts in QI methodology and research and clinical pediatric sepsis care, including nurses, physicians, and infection control specialists, as well as CHA personnel, including administrative liaisons, QI consultants, data specialists, and statisticians.
A modified Delphi approach was used to define the cohort and to develop and vet metrics as delineated by the National Quality Forum.32–34 This process was conducted over 9 virtual conferences and 4 in-person meetings, including 3 distinct voting rounds. An initial 41 metrics were posed to the group, ranked and reduced to 24 final metrics prioritizing those with supporting evidence, process metrics historically linked with improved outcomes, representation of the sepsis care continuum, generalizability, EHR abstraction feasibility, and categorization within process, outcome, and balancing domains.
In February 2020, the pediatric Surviving Sepsis Campaign guidelines were published, delineating that slower fluid and antibiotic administration may be appropriate for those with sepsis without organ dysfunction or shock.8 Hourly therapeutic targets for those who have septic shock have remained consistent. As such, our evidence table, recommendations, and collaborative goals changed to reflect differing targets for IPSO sepsis and IPSO noncritical and critical sepsis cohorts.
An initial outcome measure included death from any cause. Chart review at several sites determined that death attributed to sepsis represented significantly fewer cases than all-cause mortality. Therefore, 1 year into the collaborative, sites further determined which of their deaths were sepsis attributable on the basis of Goldstein definitions (as this was now retrospective review), and the data portal was altered to reflect this.
Previous local and collaborative QI work revealed that IV access difficulties did not substantially contribute to timely delivery of therapeutics and was unreliably reported.11 Therefore, this metric was not retained in last-round voting. Sites, however, were encouraged to drill down within their local process as they investigated delays in care.
We captured patients arriving from home and referring hospitals. As interventions from a referring center are difficult to glean accurately from manual chart review, these patients were excluded from final analysis. Emergency medical systems from the field may rarely initiate fluids; however, start and stop times are unreliably captured in the EHR.35 Analysis of these interventions would have been prohibitive on a large scale, given the extensive manual chart review needed.
T0
T0 for time-bound metrics was defined as time of earliest possible sepsis recognition, as determined electronically, denoted as functional T0 (Fig 2). Because there is not yet a singular biomarker to identify sepsis, EHR surrogates, such as time a sepsis screen alerted, were used. Researchers at many sites felt that retrospective clinical chart review could better define a true physiologic T0, so this was offered as an optional variable to report and denoted as “clinically derived T0.” Calculations for performance metrics, however, were uniformly based on functional T0.
Functional T0 definition as automatically abstracted from the EHR was used in an attempt to identify the earliest point of sepsis recognition. Manually abstracted and clinically derived T0 supplemented this definition and was used to attempt to identify the earliest physiologic onset of sepsis.
Functional T0 definition as automatically abstracted from the EHR was used in an attempt to identify the earliest point of sepsis recognition. Manually abstracted and clinically derived T0 supplemented this definition and was used to attempt to identify the earliest physiologic onset of sepsis.
Ongoing Measurement
Each site completed institutional regulatory requirements necessary for use of a limited data set for QI purposes, including an institutional review board approval at some sites and a participation agreement at all sites. Dates were the only protected health information submitted because they were needed to calculate several metrics, including age, unit transfer times, and outcome metric data. Several error alerts were built into the data portal to alert sites to variable discrepancies before submission (Supplemental Information). Additionally, discrepancies in incidence by month were resolved in real time to address cohort identification issues.
Continuous QI necessitates evaluation of process and outcomes over time. Therefore, statistical process control (SPC) charts were mapped to each metric and generated with a frequency dictated by the data.36 Most metrics were reported monthly, with rarer events, such as mortality metrics, reported quarterly. Percentage charts (p-charts) represented percent adherence to a certain goal; X bar charts were chosen for “time to” interventions and u-charts for rare outcomes, such as mortality.
Once sites submitted the baseline and first year of prospective data, they had access to centralized data portal reports managed by CHA as well as a third party (Prometheus Research, an IQVIA Company, New Haven, CT). Access to the reports was retained if the site maintained regular data submission. The portal provided visualization of individual and aggregate progress across the collaborative. Control charts allowed for monthly informing of plan-do-study-act cycles and benchmarking across sites.
We measured sites’ submission rate of variables to the portal. Sites were responsible for manual or electronic identification of the sepsis cohorts, followed by subsequent collection and submission of variables related to these cohorts. Although some sites abstracted data manually, others initially or eventually automated their abstraction efforts. We analyzed site characteristics that were associated with faster variable submission within the portal, including method of variable submission, size of hospital, freestanding children’s hospital versus not, and whether the site had participated in previous sepsis collaboratives. We used Wilcoxon rank sum statistics for 2 level comparisons or Kruskal-Wallis tests for >2 level comparisons.
Results
On the basis of the KDD and the iterative process described above, metrics, variables and SPC charts were developed. Twenty-four metrics reflected both timely and appropriate interventions and included process, outcome and balancing metrics (Fig 3). A comprehensive data dictionary included 44 final variables, 40 of which could be abstracted automatically from the EHR, including Epic, Cerner, and Meditech platforms and were reliably reported for most encounters (Table 1). Most variables mapped to metrics, but some were abstracted to describe the patient population and inform future guidelines, such as serum lactate and hypotension time. Until December 2019, 90% of expected data have been submitted, representing 200 007 sepsis episodes (Fig 4).
ISPO collaborative metrics with measures, associated variables, and control charts.
ISPO collaborative metrics with measures, associated variables, and control charts.
Abridged Data Dictionary With Variables, Definitions, and Percentages Reported in the IPSO Sepsis Cohort
Variable (Ordered by Reporting Frequency) . | Data Dictionary Variable Definition . | IPSO Sepsis Cohort (Reported in Descending Order), % . |
---|---|---|
Arrival time | Time patient arrived at the ED or was admitted to the hospital directly | 100.0 |
Functional T0 | See T0 diagram, Fig 1 | 99.3 |
Bolus 1 time | Time that first fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 98.4 |
First antibiotic time | Time that first IV antibiotic (antibacterial) was started, whether the antibiotic was effective or not; intramuscular or intraosseous used for sepsis also acceptable | 97.5 |
IV antibiotic days | The No. days the patient is on IV antibiotics within the 30 d after T0; use whole numbers only; if the patient is on IV antibiotics for one-half a day, count as 1 d; if the patient is on multiple antibiotics, each antibiotic counts as a day (for example, 3 d of 2 antibiotics = 6 d) | 97.1 |
Bolus 3 volume | Volume of the third bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 97.1 |
Bolus 2 time | Time that second fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 95.8 |
Vasoactive agent type | Indicates what kind of vasoactive agent was first used; the target time is within 24 h after T0 | 95.8 |
Birthdate | Patient’s date of birth | 95.5 |
First vasoactive time | Time the first vasoactive agent was used | 95.5 |
ICU d | No. days the patient was in the ICU for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include ICU days before T0; if the patient was not in the ICU, report 0 d | 95.3 |
Bolus 3 time | Time that third fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 95.3 |
Bolus 2 volume | Volume of the second bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 94.4 |
Time hematology or oncology from ED | Time of patient transfer between hematology and oncology service and the ED | 93.9 |
Wt | Patient’s wt in kilograms | 93.5 |
Time ICU from hematology or oncology | Time of patient transfer between the ICU and hematology or oncology service | 93.2 |
Positive pressure ventilation d | The No. days the patient was on ventilation for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; include days in which the patient had noninvasive ventilation, such as BIPAP or CPAP; do include ventilator days before T0 | 93.1 |
Time ICU from general care | Time of patient transfer between the ICU and acute care or general care service | 93.1 |
Vasoactive d | No. days the patient was on pressors for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include vasoactive days before T0 | 93.0 |
Lactic acid time | Time first lactic acid was obtained; do not capture lactic acid from an outside hospital, only the first one at your institution | 93.0 |
Time general care from ED | Time of patient transfer between the ED and the acute care or general care service | 93.0 |
Time ICU from ED | Time of patient transfer between the ICU and the ED | 92.9 |
T0 location | The location of the patient at the time of functional T0 | 92.8 |
Order set | Time of first use of either (1) any component of a severe sepsis order set or (2) a related infection‐specific order set that includes core severe sepsis components; for example, a pneumonia order set | 92.4 |
Bolus 1 volume | Volume of the first bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 92.1 |
Lactic acid value | Initial lactic acid value | 91.5 |
Sepsis screen | Time of initial screening process to identify possible severe sepsis in patient when screen result was positive; the initial screening process may consist of an electronic alert, a checklist, Pediatric Early Warning Scores (absolute value and/or change), bedside nursing screens, or other identification tools; the initial screening process may use paper‐based tools or electronic tools | 91.3 |
Extracorporeal membrane oxygenation | Indicates if patient was placed on extracorporeal membrane oxygenation within 30 d of T0 | 88.9 |
First hypotension time | If the patient was hypotensive, this is the time of the first documented hypotension. Use what your institution classifies as hypotensive. | 87.7 |
Central venous line time | Time of placement of central venous line in patient, if within time frame of 72 h before T0 to 72 h after T0 | 87.0 |
Outside hospital specified | Indicates if a patient arrived at your hospital from another hospital or facility in the 24 h before T0 | 86.7 |
Blood culture result positive | Indicates whether there was a positive blood culture result for the patient; blood culture results from an outside hospital or primary care provider can be used | 85.7 |
Chronically ventilated | Indicates whether the patient was chronically ventilated or used BIPAP or CPAP as baseline for any portion of the day | 84.5 |
Time surgical source control | Time of surgical procedure to control the source of the patient’s infection (including definitive management of abscess, peritonitis, line infection, bone or joint infection, empyema, or infected hardware); if the procedure was within 48 h before T0 to 48 h after T0 | 75.2 |
Huddle time | Time of sepsis team huddle to review clinical findings and determine if that patient is on a severe sepsis pathway in cases in which the huddle result was positive | 74.2 |
Mixed venous oxygen saturation time | Time of obtaining mixed venous saturation for patient, if within time frame of 72 h (72 h before T0 to 72 h after T0) | 69.2 |
Organ dysfunction | Indicate if and when organ dysfunction occurred after. Organ dysfunction would include ≥1 of the following: cardiovascular, respiratory, renal, hepatic, hematologic, or neurologic per Goldstein et al26 definitions. These data may require clinician review of the patient’s chart. | 53.0 |
High-risk condition other than not applicable | The patient’s underlying high-risk conditions documented in ED or on admission; may require manual chart review to determine malignancy, asplenia (including sickle cell), bone marrow transplant, indwelling line or catheter, solid organ transplant, global developmental delay or cerebral palsy, immunocompromise or suppression, technology dependence (gastrostomy tube, tracheostomy tube, or ventriculoperitoneal shunt) | 41.3 |
Clinically derived T0 | The date and time of the patient’s onset of physiologic sepsis, as determined by manual chart review; we recommend using the Goldstein et al26 definitions; this definition is different from functional T0 | 32.2 |
Risk score | Using your hospital’s standard risk-scoring tool, report the risk score for the patient. You may use the Pediatric Index of Mortality, PRISM, or another risk-scoring tool. | 11.4 |
Risk score method | Name of the tool or scoring method used to assess a risk of mortality score | 10.8 |
Other International Classification of Diseases, 10th Revision sepsis codes | A02.1, A20.7, A21.7, A22.7, A24.1, A26.7, A32.7, A39.2, A39.3, A39.4, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.50, A41.51, A41.52, A41.53, A41.59, A41.81, A41.89, A41.9, A42.7, A54.86, B00.7, B37.7 | N/A |
International Classification of Diseases, 10th Revision severe sepsis, septic shock | R65.20, R65.21 | N/A |
Variable (Ordered by Reporting Frequency) . | Data Dictionary Variable Definition . | IPSO Sepsis Cohort (Reported in Descending Order), % . |
---|---|---|
Arrival time | Time patient arrived at the ED or was admitted to the hospital directly | 100.0 |
Functional T0 | See T0 diagram, Fig 1 | 99.3 |
Bolus 1 time | Time that first fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 98.4 |
First antibiotic time | Time that first IV antibiotic (antibacterial) was started, whether the antibiotic was effective or not; intramuscular or intraosseous used for sepsis also acceptable | 97.5 |
IV antibiotic days | The No. days the patient is on IV antibiotics within the 30 d after T0; use whole numbers only; if the patient is on IV antibiotics for one-half a day, count as 1 d; if the patient is on multiple antibiotics, each antibiotic counts as a day (for example, 3 d of 2 antibiotics = 6 d) | 97.1 |
Bolus 3 volume | Volume of the third bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 97.1 |
Bolus 2 time | Time that second fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 95.8 |
Vasoactive agent type | Indicates what kind of vasoactive agent was first used; the target time is within 24 h after T0 | 95.8 |
Birthdate | Patient’s date of birth | 95.5 |
First vasoactive time | Time the first vasoactive agent was used | 95.5 |
ICU d | No. days the patient was in the ICU for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include ICU days before T0; if the patient was not in the ICU, report 0 d | 95.3 |
Bolus 3 time | Time that third fluid bolus began; vol <5 mL/kg should not be counted as a bolus | 95.3 |
Bolus 2 volume | Volume of the second bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 94.4 |
Time hematology or oncology from ED | Time of patient transfer between hematology and oncology service and the ED | 93.9 |
Wt | Patient’s wt in kilograms | 93.5 |
Time ICU from hematology or oncology | Time of patient transfer between the ICU and hematology or oncology service | 93.2 |
Positive pressure ventilation d | The No. days the patient was on ventilation for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; include days in which the patient had noninvasive ventilation, such as BIPAP or CPAP; do include ventilator days before T0 | 93.1 |
Time ICU from general care | Time of patient transfer between the ICU and acute care or general care service | 93.1 |
Vasoactive d | No. days the patient was on pressors for any part of the day beginning at T0 until discharge, death, or 30 d, whichever comes first; do not include vasoactive days before T0 | 93.0 |
Lactic acid time | Time first lactic acid was obtained; do not capture lactic acid from an outside hospital, only the first one at your institution | 93.0 |
Time general care from ED | Time of patient transfer between the ED and the acute care or general care service | 93.0 |
Time ICU from ED | Time of patient transfer between the ICU and the ED | 92.9 |
T0 location | The location of the patient at the time of functional T0 | 92.8 |
Order set | Time of first use of either (1) any component of a severe sepsis order set or (2) a related infection‐specific order set that includes core severe sepsis components; for example, a pneumonia order set | 92.4 |
Bolus 1 volume | Volume of the first bolus in milliliters; vol <5 mL/kg should not be counted as a bolus | 92.1 |
Lactic acid value | Initial lactic acid value | 91.5 |
Sepsis screen | Time of initial screening process to identify possible severe sepsis in patient when screen result was positive; the initial screening process may consist of an electronic alert, a checklist, Pediatric Early Warning Scores (absolute value and/or change), bedside nursing screens, or other identification tools; the initial screening process may use paper‐based tools or electronic tools | 91.3 |
Extracorporeal membrane oxygenation | Indicates if patient was placed on extracorporeal membrane oxygenation within 30 d of T0 | 88.9 |
First hypotension time | If the patient was hypotensive, this is the time of the first documented hypotension. Use what your institution classifies as hypotensive. | 87.7 |
Central venous line time | Time of placement of central venous line in patient, if within time frame of 72 h before T0 to 72 h after T0 | 87.0 |
Outside hospital specified | Indicates if a patient arrived at your hospital from another hospital or facility in the 24 h before T0 | 86.7 |
Blood culture result positive | Indicates whether there was a positive blood culture result for the patient; blood culture results from an outside hospital or primary care provider can be used | 85.7 |
Chronically ventilated | Indicates whether the patient was chronically ventilated or used BIPAP or CPAP as baseline for any portion of the day | 84.5 |
Time surgical source control | Time of surgical procedure to control the source of the patient’s infection (including definitive management of abscess, peritonitis, line infection, bone or joint infection, empyema, or infected hardware); if the procedure was within 48 h before T0 to 48 h after T0 | 75.2 |
Huddle time | Time of sepsis team huddle to review clinical findings and determine if that patient is on a severe sepsis pathway in cases in which the huddle result was positive | 74.2 |
Mixed venous oxygen saturation time | Time of obtaining mixed venous saturation for patient, if within time frame of 72 h (72 h before T0 to 72 h after T0) | 69.2 |
Organ dysfunction | Indicate if and when organ dysfunction occurred after. Organ dysfunction would include ≥1 of the following: cardiovascular, respiratory, renal, hepatic, hematologic, or neurologic per Goldstein et al26 definitions. These data may require clinician review of the patient’s chart. | 53.0 |
High-risk condition other than not applicable | The patient’s underlying high-risk conditions documented in ED or on admission; may require manual chart review to determine malignancy, asplenia (including sickle cell), bone marrow transplant, indwelling line or catheter, solid organ transplant, global developmental delay or cerebral palsy, immunocompromise or suppression, technology dependence (gastrostomy tube, tracheostomy tube, or ventriculoperitoneal shunt) | 41.3 |
Clinically derived T0 | The date and time of the patient’s onset of physiologic sepsis, as determined by manual chart review; we recommend using the Goldstein et al26 definitions; this definition is different from functional T0 | 32.2 |
Risk score | Using your hospital’s standard risk-scoring tool, report the risk score for the patient. You may use the Pediatric Index of Mortality, PRISM, or another risk-scoring tool. | 11.4 |
Risk score method | Name of the tool or scoring method used to assess a risk of mortality score | 10.8 |
Other International Classification of Diseases, 10th Revision sepsis codes | A02.1, A20.7, A21.7, A22.7, A24.1, A26.7, A32.7, A39.2, A39.3, A39.4, A40.0, A40.1, A40.3, A40.8, A40.9, A41.01, A41.02, A41.1, A41.2, A41.3, A41.4, A41.50, A41.51, A41.52, A41.53, A41.59, A41.81, A41.89, A41.9, A42.7, A54.86, B00.7, B37.7 | N/A |
International Classification of Diseases, 10th Revision severe sepsis, septic shock | R65.20, R65.21 | N/A |
BIPAP, bilevel positive airway pressure; CPAP, continuous positive airway pressure; N/A, not applicable.
A, Data submission over time for primary sepsis cohorts. B, Data submission over time for primary sepsis cohorts and subcohorts.
A, Data submission over time for primary sepsis cohorts. B, Data submission over time for primary sepsis cohorts and subcohorts.
The data submission curve revealed a slow submission initially, a steady ramp-up phase, then a plateau. We observed a 6 to 9-month delay from the onset of the data portal opening to first submission of data and an 18-month delay before >80% of the data were submitted (Fig 4). Hospital size, electronic versus manual data abstraction, and previous sepsis QI work before IPSO were not associated with increased median time to >80% of data submission (Table 2). At collaborative onset, the percentage of sites reporting good to excellent personnel support in the following domains was as follows: clinical informatics (78%), data analyst or statistician (71%), QI consultant (82%), data entry personnel (100%), and project manager (3%). With the onset of COVID-19 in the Unites States, sites maintained similar levels of commitment, although clinical informatics full support dropped to 56%.
Characteristics of Hospital Sites With Rate of Data Submission (48 Sites Analyzed)
Site Characteristic . | No. Sites . | Median No. Months to >80% Data Submission . | 25th Percentile . | 75th Percentile . | P . |
---|---|---|---|---|---|
Hospital bed count | .765 | ||||
<100 | 9 | 15 | 9 | 20 | |
100–199 | 12 | 16 | 12 | 19.5 | |
200–299 | 10 | 15 | 12 | 19 | |
300+ | 17 | 16 | 15 | 18 | |
Previous QI collaborative participation | .404 | ||||
Yes | 18 | 16.5 | 15 | 20 | |
No | 30 | 15 | 9 | 19 | |
Date capture process | .578 | ||||
Electronic | 24 | 16 | 15 | 19.5 | |
Manual initially, then electronic | 8 | 13 | 10 | 20 | |
Manual | 16 | 16 | 11 | 21.5 | |
Hospital type | .655 | ||||
Freestanding children’s hospital | 23 | 16 | 13 | 20 | |
Not freestanding | 25 | 15 | 11 | 19 |
Site Characteristic . | No. Sites . | Median No. Months to >80% Data Submission . | 25th Percentile . | 75th Percentile . | P . |
---|---|---|---|---|---|
Hospital bed count | .765 | ||||
<100 | 9 | 15 | 9 | 20 | |
100–199 | 12 | 16 | 12 | 19.5 | |
200–299 | 10 | 15 | 12 | 19 | |
300+ | 17 | 16 | 15 | 18 | |
Previous QI collaborative participation | .404 | ||||
Yes | 18 | 16.5 | 15 | 20 | |
No | 30 | 15 | 9 | 19 | |
Date capture process | .578 | ||||
Electronic | 24 | 16 | 15 | 19.5 | |
Manual initially, then electronic | 8 | 13 | 10 | 20 | |
Manual | 16 | 16 | 11 | 21.5 | |
Hospital type | .655 | ||||
Freestanding children’s hospital | 23 | 16 | 13 | 20 | |
Not freestanding | 25 | 15 | 11 | 19 |
“Sepsis-attributable death” was a midcollaborative added variable. The PSSC had previously revealed that a midcollaborative change in the data portal resulted in a 1-year delay in reporting of modified variables (R.P., H.D., unpublished observations). Here, 95% of sites were able to provide this variable within 5 months.
32% of patients had a clinically derived T0 reported. Clinically derived T0 was the same as functional T0 in 7675 episodes and was earlier in 2369 episodes, with an overall median difference of 0 minutes (interquartile range: 0–19).
Discussion
We have developed a set of measures with related variables and SPC charts to better understand the spectrum of pediatric sepsis care and track improvement using a QI learning collaborative model across 56 hospitals. We found that centers could reliably and consistently provide data for 24 established metrics to define >200 000 sepsis episodes in pediatric patients.
Sepsis Definitions
Identification of a sepsis patient cohort is a challenge, given the unique difficulties in defining sepsis. Unlike medical conditions that adhere to a narrow clinical phenotype, such as asthma, sepsis is heterogeneous and often subtle in initial presentation, making recognition and retrospective identification difficult, leading to unreliable coding.37 Furthermore, there is significant overlap with other conditions, making automated identification problematic.
Many sites struggled with whether to exclude patients who met the strict definition of IPSO sepsis but on chart review did not appear to have sepsis (eg, patients with transient hypotension due to seizure or those receiving multiple boluses as pretreatment of chemotherapy). The collaborative was aware that the IPSO sepsis definition was designed to be sensitive to mitigate missed case identification, yet also understood that specificity was able to be captured within the IPSO critical sepsis definition. Therefore, sites were instructed to remain consistent in their identification methods over time, with emphasis placed on adherence to the a priori, broader identification criteria.
Creating a Data Pipeline
A myriad of reasons resulted in delayed variable reporting. We noted that sites that used manual reporting had earlier submission of data, with the first site to submit data relying on manual abstraction alone. However, once sites that automated their process established their data pipeline, they were able to submit data in a complete and timely fashion thereafter. Sites spent considerable time investing in data analytic resources to create the coding structure for automated abstraction. Once created, clinical review ensured validation of the code.
Coding for abstraction of many sepsis variables, such as chronic ventilation, organ dysfunction, and sepsis screen time, was time-consuming and onerous. Bedside huddle time was reported as particularly problematic to abstract, given it is an often-performed but undocumented process and thereby difficult to capture in an automated fashion. Chart review was often needed to capture a huddle because natural language processing was not used in abstraction. Sites that are beginning their efforts should focus on creating discrete flowsheet rows within their EHR for processes such as huddle time.
Most sites had a primary sepsis QI champion responsible for early and sustainable data submission. When primary champions left the institution or were reassigned to other roles, a temporary lag in data submission ensued. Attention should be paid to developing infrastructure that can function despite primary champions transitioning.
Several nonfreestanding sites reported that they were mirroring their efforts on previous adult-based sepsis QI projects. Doing so proved to be problematic and a less sustainable strategy, given disparate therapeutic recommendations for pediatric and adult sepsis and the increased prevalence of sepsis in adults.38 For example, adult sepsis QI efforts abstract time to 30 mL/kg of IV fluids and place a larger emphasis on certain variables, such as lactate.
Previous Sepsis QI Work
It was hypothesized that sites that had participated in previous sepsis QI collaboratives or had completed significant institutional sepsis QI work would be able to submit data sooner than those that had not. New York State passed Rory’s Regulations in 2013 to ensure that all hospitals caring for pediatric patients had sepsis QI protocols and submitted data to the state.39 Because many of the variables reported were similar to those reported for IPSO, these sites could potentially report data earlier because of personnel, data infrastructure, and executive support already in place. Unfortunately, this hypothesis was unfounded. Sites reported that the definitions and variables needed for IPSO submission were substantially different from those needed for previous work and thus time to submission could not be expedited. Having a robust registry already in place at times hindered the ability to change one’s workflow and to modify alignment of data abstraction with the IPSO collaborative.
Problematic Variable Abstraction
Historically, identifying and automating T0 for sepsis onset has posed a significant challenge.11,12,14 Because of the resources required for retrospective chart review for clinically derived T0, this variable proved to be underreported. In the majority of reported cases, clinically derived T0 and functional T0 were equal, but further investigation is needed for the cases in which they differed.
Outcome variables can be sensitive to individual patient factors, such as severity of illness and idiosyncratic patient physiology. We thus included variables allowing for risk stratification, such as the Pediatric Risk of Mortality Score (PRISM).40 The PRISM was being reported by PICUs before IPSO through the Virtual PICU Systems initiative (VPS, LLC, Los Angeles, CA). Sites could also optionally calculate severity scores for patients in their initial presenting hours of non-PICU care. Because of the labor-intensive nature of calculating this score, this value was also rarely reported.
Limitations
We had initially facilitated the data abstraction process by streamlining our variables, such as presence or absence of organ dysfunction. We felt collecting specific systems of organ dysfunction present for a single encounter would be too onerous. In retrospect, inclusion of this granular data would have aided in future analysis and better aligned with current research.
Conclusions
Variables, metrics and control charts to measure recognition, appropriate therapeutic interventions, and outcomes across 56 sites were created for the largest pediatric sepsis QI collaborative. Variables were amenable to automated and sustainable abstraction. Size or type of hospital, method of data abstraction, and previous QI collaboration participation did not influence hospitals’ abilities to contribute meaningful data. Sites initiating and optimizing their sepsis QI journey can benefit from these efforts in the future.
The IPSO Collaborative Investigators
The following are IPSO Collaborative investigators:
Audrey H Barnett, MSN, RN, RNC-NIC, CPHQ, Department of Quality and Safety, Children’s Memorial Hermann Hospital, Houston, Texas;
Sopnil N Bhattarai, CPHQ, Department of Performance Improvement, Children’s National Hospital, Washington, District of Columbia;
Michael T Bigham, MD, FAAP, FCCM, Critical Care Medicine, Akron Children’s Hospital, Akron, Ohio, Associate Professor, Pediatrics, Northeast Ohio Medical University, Rootstown, Ohio, Akron, Ohio;
Kristi Booker, RN, MSN, CDN, Quality Department, Children’s Hospital at University of Oklahoma Medicine, Oklahoma City, Oklahoma;
Renee M Bruhn, ME, Center for Healthcare Quality and Analytics, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania;
David G Bundy, MD, MPH, Department of Pediatrics, Medical University of South Carolina, Charleston, South Carolina;
Stephanie A. Burrus, DO, Department of Pediatrics, Children’s Mercy Kansas City, Division of Pediatric Hospital Medicine, Children’s Mercy Kansas City, Kansas City, Missouri;
Pavan Kumar Chundi, MS, James M. Anderson Center for Health Systems Excellence, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio;
Emily C Dawson, MD, Department of Pediatrics, Advocate Children’s Hospital, Section of Pediatric Critical Care and Pediatric Emergency Medicine, Oak Lawn, Illinois;
Theodore K M DeMartini, MD, Department of Pediatrics, Penn State University, Division of Pediatric Critical Care Medicine, Penn State Children’s Hospital, Hershey, Pennsylvania;
Susan J Duffy, MD, MPH, Department of Emergency Medicine and Pediatrics Alpert Medical School, Brown University, Pediatric Emergency Medicine, Hasbro Children’s Hospital, Providence, Rhode Island;
Jill B Dykstra Nykanen, RN, MSN, CPHQ, Department of Pediatrics, Orlando Health Arnold Palmer Hospital for Children, Orlando, Florida;
Julie C Fitzgerald, MD, PhD, MSCE, Department of Anesthesiology and Critical Care, University of Pennsylvania Perelman School of Medicine, Department of Anesthesiology and Critical Care, Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania;
Meg Frizzola, DO, FAAP, Department of Pediatrics, Nemours/Alfred I. duPont Hospital for Children, Department of Pediatrics, Thomas Jefferson University, Wilmington, Delaware;
Elizabeth J. Haines, DO, MAS, FACEP, Ronald O. Perelman Department of Emergency Medicine, NYU Langone Health, Hassenfeld Children’s Hospital, NYU Langone Health, New York, New York;
Hana Hakim, MD, MS, Department of Infectious Diseases, St. Jude Children’s Research Hospital, Memphis, Tennessee;
Lauren M Hess, MD, Department of Pediatrics, Baylor College of Medicine, Section of Hospital Medicine, Texas Children’s Hospital, Houston, Texas;
Sarah B Kandil, MD, Department of Pediatrics, Yale University, School of Medicine, Section of Critical Care Medicine, Yale New Haven Children’s Hospital, New Haven, Connecticut;
Kimberly D Kato, RN, BSN, MS, Clinical Effectiveness and Improvement, Ann and Robert H. Lurie Children’s Hospital of Chicago; Chicago, Illinois;
Daniel P Kelly, MD, Department of Pediatrics, Division of Medical Critical Care, Boston Children’s Hospital, Department of Pediatrics, Harvard Medical School, Boston, Massachusetts;
Raed M. Khoury, MA, MPH, CIC, MT (ASCP), CLS (CA), ARM, CPHQ, CHSP, CJCP, CSSBB, FAPIC, Quality & Patient Safety and Medical Affairs, Madera, California;
V. Matt Laurich, MD, Department of Pediatrics, University of Connecticut, Division of Pediatric Emergency Medicine, Connecticut Children’s, Hartford, Connecticut;
Stephanie M. Lavin, MSN, RN, CPN, Clinical Collaborative, Cook Children’s Medical Center, Ft Worth, Texas;
Jennifer L Liedel, MD, Department of Pediatrics, Nemours Children’s Hospital, Divisions of Critical Care and Neonatology, Nemours Children’s Hospital, Orlando, Florida;
Jeremy M Loberger, MD, Department of Pediatrics, Division of Pediatric Critical Care Medicine, University of Alabama at Birmingham, Birmingham, Alabama;
Justin M Lockwood, MD, MSCS, Department of Pediatrics, Section of Hospital Medicine, University of Colorado School of Medicine, Aurora, Colorado;
Merrick R Lopez, MD, FAAP, Department of Pediatrics, Loma Linda University Children’s Hospital, Section of Pediatric Critical Care, Loma Linda University Children’s Hospital, Loma Linda, California;
Erica A Michiels, MD, Department of Emergency Medicine, Helen DeVos Children’s Hospital, Department of Emergency Medicine, Michigan State University, Grand Rapids, Michigan;
Kristina J Murphy, DO, Department of Pediatric Critical Care, Cohen Children’s Medical Center, New Hyde Park, New York;
Gregory P Priebe, MD, Department of Anesthesiology, Critical Care and Pain Medicine, Boston Children’s Hospital, Harvard Medical School, Boston, Massachusetts;
Sandhya Ramachandran, MPH, QI Services, Nationwide Children’s Hospital, Columbus, Ohio;
Faisal Razzaqi, MD, Valley Children’s Hospital, Madera, California;
Johanna R Rosen, MD, Department of Pediatrics, University of Pittsburgh School of Medicine, Division of Emergency Medicine, UPMC Children’s Hospital of Pittsburgh, Pittsburgh, Pennsylvania;
Lori E Rutman, MD, MPH, Department of Pediatrics, University of Washington, Division of Emergency Medicine, Seattle Children’s Hospital, Seattle, Washington;
Amanda M Sebring, MD, Department of Pediatrics, Levine Children’s Hospital, Section of Pediatric Critical Care, Levine Children’s Hospital, Charlotte, North Carolina;
Jonathan A Silverman, MD, MPH, Department of Emergency Medicine, Virginia Commonwealth University School of Medicine, Division of Pediatric Emergency Medicine, Children’s Hospital of Richmond, Richmond, Virginia;
Erika L Stalets, MD, MS, Department of Pediatrics, University of Cincinnati College of Medicine, Division of Critical Care Medicine, Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio;
Nathan E Thompson, MD, PharmD, MS, Department of Pediatrics, Medical College of Wisconsin, Section of Critical Care Medicine, Children’s Wisconsin, Milwaukee, Wisconsin;
Zebulon J Timmons, MD, FAAP, Department of Pediatrics, Phoenix Children’s Hospital, Section of Pediatric Emergency Medicine, Phoenix Children’s Hospital, Phoenix, Arizona;
Jennifer J Wilkes, MD MSCE, Department of Pediatrics, University of Washington, Division of Cancer and Blood Disorders, Seattle, Washington; and
Jennifer K Workman, MD, MSCI, Department of Pediatrics, University of Utah, Division of Pediatric Critical Care, Primary Children’s Hospital, Salt Lake City, Utah.
Individual participant data that underlie the results reported in this article, after deidentification, including the study protocol and statistical analytic plan, will be available beginning 9 months and ending 36 months after article publication and collaborative completion for researchers who provide a methodologically sound proposal to achieve aims in the approved proposal. Proposals should be directed to [email protected]. A data access agreement will need to be signed, and data will be available for 3 years.
Drs Paul, Auletta, and Richardson conceptualized and designed the study, drafted the initial manuscript, designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Drs Niedner, Brilli, Macias, Balamuth, Depinet, Larsen, Huskins, Scott, and Schaffer conceptualized and designed the study, drafted the initial manuscript, and reviewed and revised the manuscript; Ms Lucasiewicz, Ms Riggs, and Ms DeSouza designed the data collection instruments, collected data, conducted the initial analyses, and reviewed and revised the manuscript; Drs Silver and Hueschen, Ms Campbell, and Ms Wathen critically reviewed the manuscript for important intellectual content; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
FUNDING: Supported by the Children’s Hospital Association and the Improving Pediatric Sepsis Outcomes Collaborative. Dr Scott’s institution is currently receiving career development salary support from the Agency for Healthcare Research and Quality.
References
Competing Interests
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
Comments