Pediatric Outcomes After Regulatory Mandates for Sepsis Care

We performed a retrospective cohort study of children hospitalized with sepsis in New York and 4 control states: Florida, Maryland, Massachusetts, and New Jersey. We used a quasi-experimental comparative interrupted time-series approach, examining longitudinal trends in sepsis outcomes before and after the New York sepsis regulations, comparing trends in New York with trends in control states.^15,16  This approach is conceptually similar to difference-in-differences analysis and provides stronger inference than using data from New York alone.¹⁷ 

We obtained patient-level sepsis hospitalization data from the Agency for Healthcare Research and Quality’s Healthcare Cost and Utilization Project State Inpatient Databases. These data were augmented with hospital-level characteristics by using data sets from the Centers for Medicare and Medicaid Services’ Healthcare Cost Reporting Information System (HCRIS), the American Hospital Association Annual Survey of Hospitals, and the Children’s Hospital Association membership database.

All analyses were prespecified, and a statistical analysis plan was prepublished on Open Science Framework on December 13, 2018 (see the Supplemental Information).¹⁸  Deviations from this plan are noted as post hoc, and a rationale for deviations is provided in the Supplemental Information. The University of Pittsburgh Human Research Protection Office reviewed and approved this study.

All hospital admissions for patients <18 years old with sepsis were initially eligible. We identified sepsis using International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) codes. We used 2 sets of codes: the explicit ICD-9-CM codes for sepsis (785.52 and 995.92) and the ICD-9-CM codes for infection and organ failure.¹⁹  This approach is widely used in studies of pediatric sepsis epidemiology and captures a larger patient population with sepsis than the use of explicit sepsis codes alone.²  We excluded neonates born during the hospital encounter because they are not subject to the regulations. Additionally, we excluded patients missing data on key covariates.

We excluded hospitals not categorized as short-stay acute care hospitals, as defined in the HCRIS, and hospitals with no pediatric sepsis cases. Additionally, to create a more homogeneous sample between New York and control states, we stratified hospitals on the basis of key characteristics (hospital type, academic status, and the size of the regional population) and excluded hospital types that were not shared across New York and control states in both the pre- and postregulation period (see the Supplemental Information for more information on this process).

The primary outcome was in-hospital mortality by 30 days after the date of admission. We also examined 4 secondary outcomes: ICU admission rates; hospital length of stay; central venous catheter insertion rates, defined as traditional central venous catheters or peripherally inserted central catheters; and Clostridium difficile infection rates. These outcomes represent the intended effect of Rory’s Regulations (ie, lower sepsis-related mortality) as well as potential unintended effects (ie, increased resource use, increased use of invasive procedures, and overuse of antibiotics). Outcome variables were either obtained directly from the administrative record or defined by using validated ICD-9-CM codes (Supplemental Table 5).

Patient-level variables for risk-adjustment were based on previous studies of pediatric sepsis²⁰  and included age, sex, race, admission source (ie, emergency department or transfer from an acute care hospital), organ failures present on admission,²¹  the presence of pediatric complex chronic conditions,²²  and the season of admission.²³ 

Hospital-level variables for risk-adjustment included hospital type, hospital academic status, and the population of the hospital’s geographic area. To define hospital type, we categorized hospitals into 4 mutually exclusive groups: designated children’s hospital based on Children’s Hospital Association membership²⁴ ; non–children’s hospital with high pediatric volume (for which we selected a cutoff of 1200 annual pediatric admissions, which was roughly equivalent to the smallest children’s hospital in our sample); non–children’s hospital with medium pediatric volume (for which we selected cutoffs of <1200 but >50 annual pediatric admissions); or non–children’s hospital with low pediatric volume (for which we selected a cutoff of ≤50 annual pediatric admissions, ie, <1 pediatric admission per week on average). Academic status was defined on the basis of the presence or absence of trainees by using data in the HCRIS. Geographic area population was defined by using regional population data based on the American Hospital Association Annual Survey of Hospitals: small (metropolitan statistical area [MSA] population <100 000 or non-MSA), medium (MSA population 100 000–1 million), or large (MSA population >1 million).

We examined hospital characteristics in New York versus control states using χ² tests. We compared patient characteristics between New York and control states before and after the regulations by visually examining summary statistics; we did not perform statistical tests for these comparisons because our large sample sizes were likely to yield comparisons that were significant.

To examine the effects of the regulations, we performed a comparative interrupted time-series analysis in which we fit patient-level linear regression models with robust SEs.¹⁶  For each model, the dependent variable was the outcome of interest (described previously), and the independent variables were an indicator for the postregulation period, time (modeled as quarters), and state (New York versus control) and an interaction term for the indicators for state and postregulation implementation. We defined the preregulation period as January 1, 2011, to March 31, 2013, and the postregulation period as April 1, 2013, to September 30, 2015. Additional covariates included the complete set of patient- and hospital-level covariates defined previously. Detailed model specifications can be found in the Supplemental Information.

This model allowed us to estimate changes in outcomes over time in New York and control states before and after the regulations. Under this model, the primary test of the effect of the regulations was the differential change in slope between New York and control states (ie, the temporal trends in outcome could change with the regulations in New York and control states, but the change in New York would need to be significantly greater than the change in control states for the regulations to be considered to have a positive effect).

We examined the robustness of our findings by performing prespecified sensitivity analyses on our primary outcome. First, we repeated the analysis using 2 alternative ICD-9-CM coding schemas for sepsis: a broader definition by using additional codes for organ failure²⁵  and a narrower definition by using only the explicit codes for sepsis and septic shock.²  Second, we repeated the analysis, excluding hospitals that participated in the Greater New York Hospital Association (GNYHA) Strengthening Treatment and Outcomes for Patients-Sepsis (STOP-SEPSIS) initiative, a sepsis-focused regional quality improvement initiative that began in 2010 before the implementation of Rory’s Regulations.²⁶  To the degree that these hospitals had already adopted sepsis quality improvement efforts, including them might blunt any observed policy-related effects. Third, we repeated the analysis by moving the preregulation period back by 2 quarters to account for anticipatory practice changes in response to the coming policy implementation. Fourth, we performed an analysis in which the outcome variable was sepsis rates over time to examine the possibility that the regulations affected sepsis coding in a way that might bias the analysis.

We performed prespecified subgroup analyses, hypothesizing that the regulations might have had a differential impact on the basis of key patient and hospital-level factors. Subgroups of interest included patient age (categorized as <60 days, 60 days to 12 months, and 1–4, 5–9, 10–14, and 15–17 years), the number of complex chronic conditions (categorized as 0, 1, 2, and ≥3), the number of organ failures present on admission (categorized as ≤2 and >2), whether the patient was admitted through the emergency department, and hospital type, as categorized previously. The rationale for each subgroup and the prespecified hypotheses are provided in the Supplemental Information.

We conducted all analyses using StataMP version 15 (Stata Corp, College Station, TX). All tests for statistical significance were 2-tailed and evaluated at a significance level of α < .05.

After exclusions, there were 9436 pediatric patients with sepsis in 237 hospitals (Supplemental Fig 2). Hospitals in New York were more likely to be larger non–children’s hospitals as well as teaching hospitals when compared with hospitals in control states (Table 1). Patient characteristics were generally comparable between New York and control states before and after implementation of the regulations (Table 2). Most patients had at least 1 complex chronic condition, and a large majority of patients were cared for in specialized children’s hospitals or general hospitals with pediatric volumes equivalent to specialized children’s hospitals. Less than 10% of patients were cared for in general hospitals with small or medium pediatric volumes.

In the unadjusted model, annual pediatric sepsis mortality decreased over the study period in New York and control states (New York before: 14.0%; New York after: 11.5%; control states before: 14.4%; control states after: 11.2%). In the fully adjusted model, changes in mortality trends contemporaneous with the regulations were not significantly different in New York compared with control states (Table 3). There was no difference in the temporal trends between the pre- and postregulation period in New York (difference in slopes: −0.79% per quarter; 95% confidence interval (CI): −1.59% to 0.01%) or control states (difference in slopes: 0.17% per quarter; 95% CI: −0.38% to 0.72%) (Fig 1). The primary test of the regulations’ effect was the difference between these 2 changes, which was not statistically significant (differential change in slopes: −0.96% per quarter; 95% CI: 1.95% to 0.02%; P = .06). The mortality in control states remained relatively flat before and after the regulations, whereas the mortality in New York was increasing before the regulations and then started decreasing after the regulations (Fig 1). However, none of these trends were statistically significant. There were no significant differences in any of our secondary outcomes (Supplemental Table 8, Supplemental Fig 3).

In a prespecified sensitivity analysis excluding the 38 hospitals in metropolitan New York (45.8% of New York hospitals) that participated in the GNYHA STOP-SEPSIS quality initiative, we found that the regulations were associated with significant improvements in mortality trends in New York compared with control states (differential change in slopes: −2.08% per quarter; 95% CI: −3.79% to −0.37%; P = .02) (Table 3). There were no significant differences in mortality trends by using different case identification strategies or after moving the implementation period up 2 quarters to account for potential anticipatory changes. There was no differential change in the rate of pediatric sepsis after implementation of the regulations in New York compared with control states, suggesting that the regulations did not influence sepsis rates in a way that influenced our analysis (Supplemental Table 9).

The results of the prespecified subgroup analyses in which we examined risk-adjusted mortality trends stratified by New York and control states are shown in Table 4. After implementation of the regulations, New York experienced statistically significantly greater decreases in mortality among patients who were previously healthy (differential change in slopes: −1.36%; 95% CI: −2.62% to −0.09%; P = .04) and those who were not admitted through the emergency department (differential change in slopes: −2.42%; 95% CI: −4.23% to −0.61%; P = .01). There were no significant differences in mortality trends based on patient age categories, the number of organ failures present on admission, the type of hospital the child was admitted to, or the hospital’s teaching status. We did observe a clinically significant difference in patients admitted to small- and medium-sized non–children’s hospitals, although only 8% of patients were admitted to these types of hospitals, and the difference was not statistically significant (differential change in slopes: −2.37%; 95% CI: −4.91% to 0.18%; P = .07).

In an evaluation of a novel state-level health policy designed to improve pediatric sepsis outcomes, the totality of the results suggests that there was an overall decrease in risk-adjusted mortality associated with the regulations. This decrease was particularly significant in key patient subpopulations when compared with control states that did not implement the regulations, specifically previously healthy children and children directly admitted to the hospital. In addition, after excluding New York hospitals that participated in a previous regional sepsis-focused quality improvement initiative, implementation of the regulations was associated with statistically significant improvements in sepsis outcomes. Although not every analysis demonstrated a statistically significant reduction in mortality, when interpreted in context, these findings indicate a high likelihood that the regulations were associated with improved sepsis outcomes in New York State.

Given these results, New York State sepsis regulations have implications for wider sepsis regulation implementation. Given that a large majority of children with sepsis receive care at specialized children’s hospitals, many of which are already well prepared to care for a wide range of pediatric emergencies,^27–29  regulations for all hospitals to maintain pediatric sepsis readiness regardless of case volume may be too blunt an instrument to address pediatric sepsis mortality. It is notable that we observed a clinically significant, although not statistically significant, effect in non–children’s hospitals with relatively low pediatric admissions. It is also possible that effects of the regulations might have been dampened by larger efforts to improve pediatric sepsis quality in New York and other states.³⁰  Indeed, in a prespecified sensitivity analysis excluding New York hospitals that voluntarily participated in a previous regional sepsis-focused quality improvement initiative in pediatric sepsis, we did observe a statistically significant effect of the regulations.

These results should be interpreted in the context of other data published on pediatric sepsis outcomes in New York.¹⁴  These data, after the regulations, reveal completion of a sepsis bundle: obtaining a blood culture, administration of broad-spectrum antibiotics, and a fluid bolus within one hour of identification of sepsis was associated with lower in-hospital mortality.¹⁴  Our study differs from that previous study in that we evaluated the effect of the regulations mandating early sepsis care, not the effects of early sepsis care itself. Early sepsis care can improve outcomes, and policy can drive efforts to implement evidence-based sepsis care.

These results also suggest similar improvements in sepsis mortality seen in adult patients in New York after implementation of the regulations.³¹  Although the regulations include a mandate for pediatric-specific sepsis protocols, the data underlying pediatric sepsis care are far less robust than those for adult sepsis care. The lack of consensus surrounding evidence-based pediatric sepsis care may contribute to variation in care protocols and less definitive population-level differential outcomes after implementation of Rory’s Regulations. In addition, qualitative evaluation of hospitals’ operationalization of the regulations may unearth mechanisms that lead to improvement in specific populations of patients.

Our results provide insight into strategies to refine health policy for pediatric sepsis. Predominantly, our study reveals that efforts to mandate pediatric-focused sepsis quality improvement should be balanced against the current landscape, in which pediatric sepsis patients are most often admitted to large hospitals and children’s hospitals rather than small community hospitals. Policies targeted to specific types of hospitals or policies designed to direct children with suspected sepsis to children’s hospitals in the prehospital period (ie, regionalization) may be more beneficial.^32,33  Secondly, our study suggests that these policies may be most impactful in the absence of strong regional quality improvement dedicated to pediatric sepsis. When active regional quality improvement efforts are in place, government regulation may not be necessary.^30,34–36  Still, our study provides important support for regulations that can positively affect pediatric sepsis outcomes.

This study has several limitations. First, in this study, we used administrative data to identify sepsis. With this method, we may have failed to identify some patients with sepsis. However, when we used different identification strategies, analyses yielded similar results as our primary analysis. Additionally, there is the possibility that the administrative codes for sepsis were used differently over time; however, we did not find evidence of a differential change in the rates of sepsis coding during the study period in New York compared with control states (Supplemental Table 9). Second, our results could be sensitive to our modeling approach or our choice of control states. However, prepublication of our statistical analysis plan likely reduced the likelihood that our results are an artifact of our statistical methodology.³⁷  In addition, we were unable to control for sepsis quality improvement initiatives in control states implemented toward the end of the study period. Given the study method, we believe the existence of these small-scale efforts serves to strengthen the generalizability of our findings because it is likely that other states considering whether to adopt these regulations would contain some hospitals engaging in local efforts. Third, because of data limitations, we could not examine postdischarge outcomes such as long-term morbidity and mortality. These outcomes are important to patients and will be important for future evaluations.

In New York, the implementation of statewide sepsis regulations was generally associated with improved mortality trends, particularly in subpopulations of pediatric patients, compared with control states that did not implement sepsis regulations. Refinement of statewide sepsis care policies may be needed to further influence outcomes for all pediatric patients with sepsis.

 
• CI

  confidence interval

 
• GNYHA

  Greater New York Hospital Association

 
• HCRIS

  Healthcare Cost Reporting Information System

 
• ICD-9-CM

  International Classification of Diseases, Ninth Revision, Clinical Modification

 
• MSA

  metropolitan statistical area

 
• STOP-SEPSIS

  Strengthening Treatment and Outcomes for Patients-Sepsis