Comparison of Manual and Automated Sepsis Screening Tools in a Pediatric Emergency Department

This study was a retrospective cohort study of patients who presented to the ED of a quaternary care freestanding children’s hospital with ∼60 000 annual visits between June 1, 2017, and June 5, 2019. The study was approved by the hospital institutional review board with a waiver of informed consent.

A data warehouse was queried to (1) determine total number of ED visits during the study period and (2) identify encounters in which a manual sepsis screen was performed or a sepsis alert was triggered as well as the result (positive or negative) of the screening.

To identify possible cases of severe sepsis, encounters that met any of the following criteria were manually reviewed by one of the study authors: (1) an International Classification of Diseases, 10th Revision (ICD-10) code for severe sepsis or septic shock (R65.20 and R65.21) entered at any time during the ED or inpatient encounter, (2) clinician use of the ED septic shock order set, (3) any patient admitted from the ED to an ICU or intermediate care unit within 72 hours of ED disposition, or (4) patient died within 72 hours of ED disposition.

Encounters were considered to be positive for severe sepsis if the patient met the International Pediatric Sepsis Consensus Conference¹⁶  definition of severe sepsis or septic shock (Table 1) between the time of ED arrival and 24 hours after ED disposition or had an ICD-10 code for severe sepsis or septic shock entered for the ED encounter or within 24 hours of ED disposition. Patients who had a cardiac arrest before meeting criteria for severe sepsis were not considered severe sepsis cases.

Between June 1, 2017, and June 5, 2018, patients presenting to the study ED were manually screened for sepsis by using a version of the Pediatric Septic Shock Collaborative’s sepsis screening tool.¹⁸  In this screening process, when a patient with fever or suspected infection presented to the ED, the triage nurse was asked to indicate in the patient’s EHR whether vital sign values, physical examination findings, or elements of the previous medical history were consistent with an elevated risk of sepsis (Fig 1A). A patient who met ≥3 criteria was considered to have a positive sepsis screen result, which then generated an icon on the tracking board to alert the clinical team. Patients were not routinely screened a second time during the ED visits, even if their clinical condition changed.

On June 5, 2018, this process was replaced by an automated sepsis screening tool embedded in the EHR that continuously screened all patients for vital sign values and laboratory test results consistent with sepsis throughout their ED stay. The tool was based on consensus criteria and had modifications to improve the sensitivity and specificity of the algorithm.¹⁷  When alert criteria were met, the tool triggered a pop-up notification on the ED tracking screen to all clinicians assigned to the patient, notifying them of the category of the positive screen result and identifying which vital signs and laboratory studies triggered the alert (Fig 1B). For patients without an ED physician or nurse assigned to them, the alert was sent to every ED nurse who had opened the patient’s chart. The alert also generated an icon next to the patient’s name on the tracking board.

There were 3 possible alerts of increasing severity level that could be generated on the basis of the presence, type, and number of dysfunctional organ systems according to criteria set forth by the International Pediatric Sepsis Consensus Conference (Table 1).¹⁶  The lowest level alert, designated a systemic inflammatory response syndrome (SIRS) alert, was generated when a patient had 2 of the following abnormal values, 1 of which had to be temperature or white blood cell count: temperature, white blood cell count, heart rate, or respiratory rate. The midlevel alert, designated a sepsis alert, was triggered when a patient met criteria for a SIRS alert and also had 1 noncardiac organ dysfunction. The highest level alert, designated a severe sepsis alert, was triggered when the patient met criteria for a SIRS alert and had either cardiac dysfunction or 2 other organ dysfunctions. Once a patient triggered an alert, a subsequent alert would only trigger during the ED stay if it was of higher severity.

The primary outcome was the sensitivity of each screening tool for detection of sepsis between the time of ED arrival and 24 hours after ED disposition. Secondary outcomes included the specificity, PPV and negative predictive value (NPV), the proportion of all patients and patients with severe sepsis who were screened while in the ED, and the time to the first positive screen result or alert.

Study data were collected and managed by using Research Electronic Data Capture (REDCap; Vanderbilt University, Nashville, TN) tools hosted at Boston Children’s Hospital.¹⁹ 

The manual and automated sepsis screening periods were defined as June 1, 2017, to June 4, 2018, and June 5, 2018, to June 5, 2019, respectively.

To determine demographic and clinical differences in the populations who presented with sepsis during each time period, we compared means using the Student’s t test for normally distributed variables, compared medians using the Wilcoxon rank test for nonnormally distributed variables, and compared proportions using the χ² test of proportions. We calculated the sensitivity, specificity, PPV, NPV, and positive likelihood ratio and negative likelihood ratio with 95% confidence interval (CI) of the manual sepsis screen and the automated sepsis screen for the development of severe sepsis or septic shock. Calculation of test characteristics only included patients who were screened for sepsis during their ED stay.

All analyses were performed by using the software package Stata SE (version 16; Stata Corp, College Station, TX). All tests were 2 tailed, and α was set at .05.

Of the 122 221 ED encounters during the study period, the following underwent manual chart review: 2718 encounters in which the patient was directly admitted to the ICU from the ED, 238 encounters in which the patient was admitted to the floor and then subsequently required transfer to the ICU within 72 hours of ED disposition, 233 encounters in which the sepsis order set was used during the ED stay, 184 encounters in which the patient had an ICD-10 code for severe sepsis or septic shock, and 55 encounters in which the patient died during the ED stay or within 72 hours of ED disposition. In this review, we identified 275 encounters in which the patient met study criteria for severe sepsis or septic shock. After excluding 2 encounters in which the patient underwent cardiopulmonary resuscitation before meeting sepsis criteria, 273 sepsis encounters were included in the study: 137 (50.2%) during the manual screening period and 136 (49.8%) during the automated screening period. In Table 2, we compare the 2 groups. Only the proportion of encounters in which the patient had a previous medical history of cancer and an indwelling central venous line differed between the time periods.

Figure 2 reveals ED encounters with and without sepsis screening and the proportion of encounters in which the patient developed sepsis in each group.

During the period of manual screening, a sepsis screen was performed in 8910 of 61 026 (14.6%) encounters and 96 of 137 (70.1%) encounters in which the patient had sepsis. Because many patients did not undergo any sepsis screening, only 62 of 137 (45.3%) patients with severe sepsis had a positive sepsis alert. Among encounters in which the sepsis screen was performed, it had the following test characteristics: sensitivity of 64.6% (95% CI 54.2%–74.1%), specificity of 91.1% (95% CI 90.5%–91.7%), PPV of 7.3% (95% CI 6.3%–8.5%), NPV of 99.6% (95% CI 99.5%–99.7%), positive likelihood ratio of 7.2 (95% CI 6.2–8.5), and negative likelihood ratio of 0.4 (95% CI 0.3–0.5). For patients with sepsis who had a positive screen result, the median time from ED arrival to the first positive screen result was 12 minutes (interquartile range [IQR] 0–33) and 29 of 61 patients with hypotension (47.5%) had the positive screen result before the first hypotensive blood pressure measurement.

After implementation of the automated sepsis alerts, 100% of 61 195 encounters received continuous sepsis screening throughout the ED stay, with a positive alert occurring in 3136 of 61 195 (5.1%) encounters. The alert had the following test characteristics: sensitivity of 84.6% (95% CI 77.4%–90.2%), specificity of 95.1% (95% CI 94.9%–95.2%), PPV of 3.7% (95% CI 3.4%–4%), NPV of 99.9% (95% CI 99.9%–100%), positive likelihood ratio of 17.1 (95% CI 15.8–18.5), and negative likelihood ratio of 0.2 (95% CI 0.1–0.2).

For patients with sepsis, the distribution of the automated sepsis alerts was as follows (>1 alert type could generate during a single encounter): 69 of 136 (50.1%) had a SIRS alert, 42 of 136 (30.1%) had a sepsis alert, and 62 of 136 (45.6%) had a severe sepsis alert. The median time from ED arrival to the first alert was 71 minutes (IQR 0–188), and 28 of 70 (40%) patients with hypotension had an alert before the first hypotensive blood pressure measurement. Figure 3 shows a Kaplan-Meier curve comparing time from ED arrival to the first positive sepsis screen result for encounters in which the patient developed sepsis in both time periods.

In this study, an automated sepsis screening tool provided several advantages when compared with the previous manual screen. First, for a sepsis screening tool to be effective, it must be widely used because a screen that is only performed on patients who are already suspected to have sepsis has little added value when compared to clinician gestalt. In this study, after we transitioned from the manual to the automated screening tool, the proportion of encounters in which sepsis screening was performed increased dramatically from 14.6% to 100%. Additionally, although almost all patients during the manual period were screened just a single time, patients during the automated alert period underwent continuous sepsis surveillance throughout the ED stay, allowing the automated tool to better detect patients who clinically deteriorated while in the ED. At the same time, the automated screening algorithm had a higher sensitivity (84.6% vs 64.6%) and specificity (95.1% vs 91.1%) than the manual tool. As a result, the percentage of patients with severe sepsis with a positive screen result or alert increased from 45.3% to 84.6% after we switched to automated screening.

Our findings are important because pediatric sepsis screening has been widely adopted in EDs nationally, despite a dearth of data linking them to better outcomes. The Surviving Sepsis Campaign recommends systematic pediatric sepsis screening, although it notes that it is backed by a “very low quality of evidence.”¹⁰  To date, 3 states have mandated the use of pediatric sepsis screening in the ED, and several others are currently considering similar regulations.^6,20,21  However, the many available sepsis screens are heterogeneous in terms of criteria and form (manual versus automated). Neither the medical literature nor the expert opinion of consensus committees suggests which type of screen is most effective. An ideal sepsis screening tool should identify as many patients who are septic as possible while minimizing false-positives, which bring the dual risks of alarm fatigue and inappropriate resource use. In this study, the automated screening algorithm better fits this description.

There are 2 ways in which the automated sepsis screen appeared inferior to the manual screen: its lower PPV and the longer time from ED arrival to the first positive alert. The lower PPV is undoubtedly a function of disease prevalence rather than the screen itself. During the manual screening period, nurses were instructed to perform the sepsis screen only on patients with fever or concern for infection. This introduced selection bias, wherein the nursing staff was more likely to screen patients at higher risk of sepsis, and increased the prevalence of sepsis in that cohort. This is supported by the fact that the screen was performed in 70.1% of encounters in which the patient developed sepsis but only in 14.6% of encounters overall. In comparison, the automated screen was performed in 100% of patients in the ED, resulting in a lower sepsis prevalence and PPV, although the test itself was more specific than the manual screen.

Nonetheless, the low PPV introduces the risk of alarm fatigue, which has been shown to adversely affect outcomes of patients with sepsis.²²  Although this is somewhat mitigated by the low proportion of ED encounters in which an alert fired (5.1%), future iterations of the algorithm will need to be refined to improve the PPV without adversely affecting sensitivity. Alternatively, the PPV may be improved by applying a more targeted approach to the population screened, focusing on those with an increased risk of sepsis rather than all patients in the ED.

The second notable way in which the automated screening algorithm seemed inferior to the previous version was in the median time from ED arrival to the first positive sepsis alert, which increased from 12 to 71 minutes after the change from manual to automated screening. This may be because the manual tool incorporates elements such as risk factors for sepsis, which are intended to predict sepsis before onset, whereas the automated tool was designed to detect sepsis in its early stages. However, as shown in Fig 3, the greatest contributor to the longer median time to alert was the fact that the automated algorithm detected many cases of sepsis that developed after the first hour of the ED stay, compared with the manual screen, which seldom identified patients with sepsis after the first hour of care. Essentially, the manual screen done at triage either detected sepsis shortly after arrival or not at all, whereas the automated screen continued to detect sepsis cases throughout the ED stay.

Although authors of other studies have reported results of electronic sepsis screening algorithms, we are unaware of any study in which a manual screen was directly compared with an automated screen. It is interesting to note that the manual sepsis screen used in this study, based on the Pediatric Septic Shock Collaborative screening tool, was less sensitive (64.6% vs 92.1%) but more specific (91.1% vs 83.4%) than what was previously reported in the literature.¹³  However, in that study, an automated version of the tool was used, which may explain the difference in test characteristics.

This study was, in part, an effort to validate a sepsis screening tool whose derivation was previously described in the literature.¹⁷  In the previous derivation study, there was a lower prevalence of sepsis with an associated lower PPV compared to the current study (3.7% vs 9.6%). In the derivation study, sepsis was defined on the basis of provider actions consistent with treatment of sepsis (such as giving 2 fluid boluses and an IV antibiotic), whereas in this validation study, we used formal criteria for end-organ dysfunction. Additionally, the derivation tool was created by using 5 months of data from encounters across all inpatient hospital settings, whereas in the current validation study, we used 2 years of exclusively ED encounters. These differences likely drove this variation in both prevalence and PPV.

There are additional reports of automated sepsis screening in the ED that are worth noting. Balamuth et al¹⁵  demonstrated that a 2-stage sepsis alert, with an automated initial stage followed by a manual nurse screen, had a sensitivity of 96.6% in detecting patients with sepsis. Although the sensitivity of the tool in this study was not as high, the tool has the benefit of being fully automated rather than an automated and manual hybrid. Sepanski et al¹⁴  previously reported on a fully automated alert with a PPV for sepsis in the ED of 49%. However, in their study, 6% of all patients met criteria for sepsis (compared to 0.1% in our study), and higher prevalence will increase PPV at the same sensitivity and specificity. Our PPV of 3.7% is more consistent with values reported elsewhere from pediatric EDs.¹³ 

This study has several limitations. First, we used a strict definition of sepsis that included only patients who met criteria for severe sepsis or septic shock. As a result, patients with an alert who had compensated sepsis, received appropriate treatment, and never developed organ dysfunction were treated as false-positives in our study, potentially artificially lowering our calculated PPV. It is also possible that we missed cases of severe sepsis, despite our efforts to systematically identify such patients through multiple different data queries and chart review. An additional source of missed sepsis cases may be those that had infection-associated organ dysfunction but did not meet SIRS criteria because this is just one of several frameworks that are used to define sepsis in children. Third, our data do not address whether the automated algorithm was able to improve on the combination of screening plus clinician judgment, only that it was better than the manual screen alone. Likewise, we did not evaluate clinical outcomes (eg, time to antibiotics or IV fluids, development of organ dysfunction, or ICU length of stay) and therefore cannot comment on whether the improved tool performance led to better care for children with sepsis in our ED. Finally, we only looked at development of severe sepsis as an outcome and did not account for other scenarios in which an alert may have been a valuable tool in recognizing clinical deterioration, such as from respiratory failure or hemorrhage.

An automated sepsis screening algorithm embedded in the EHR had better sensitivity and specificity, dramatically increased compliance with sepsis screening, and provided continuous surveillance throughout the ED stay when compared with a manual screen.

CI

confidence interval

ED

emergency department

EHR

electronic health record

ICD-10

International Classification of Diseases, 10th Revision

IQR

interquartile range

IV

intravenous

NPV

negative predictive value

PPV

positive predictive value

SIRS

systemic inflammatory response syndrome