Seven hours into her overnight shift, Dr Carter opens the computer to complete her sixth admission of the evening. “BEEP! BEEP! BEEP!” her pager interrupts with “Sepsis Alert,” the fourth in the past 48 hours for a 12-year-old boy with metabolic disease, technology dependence, and paroxysmal sympathetic hyperactivity who was admitted 1 week ago for dehydration and increased storming. She leaves her computer to attend the mandatory bedside huddle. The result is the same as the 3 previous ones: storming caused his fever and tachycardia, not sepsis; no change to management. As Dr Carter writes a progress note documenting the huddle, her pager erupts again, requesting orders for her other patient.

Pediatric severe sepsis in the United States costs $7.3 billion annually, one-fifth of pediatric hospitalization costs.1  Over the past 2 decades, professional organizations have built awareness, developed guidelines, and driven efforts to recognize sepsis and intervene early in its course.2,3  Early initiatives began in emergency departments (EDs), although state legal mandates and national quality improvement collaboratives have led to implementation hospital-wide, including in general inpatient units.4,5 

Most guidelines frame their recommendations around sepsis response systems that incorporate clinical decision support tools (hereinafter referred to as sepsis scores) to identify suspected sepsis and trigger time-sensitive bundles of laboratory evaluation, intravenous fluid resuscitation, and broad-spectrum antibiotic administration.2,3  Sepsis response systems may increase value by improving outcomes and reducing costs associated with delayed recognition of pediatric sepsis.6,7  Conversely, they may decrease value if they overidentify sepsis, resulting in unnecessary interventions and care team dissatisfaction.810  Maintaining value in sepsis care in pediatric inpatient units presents unique challenges. In this commentary, we outline these challenges and encourage hospitals caring for children to thoughtfully consider them when implementing sepsis response systems in inpatient units.

Consensus pediatric definitions use the systemic inflammatory response syndrome (SIRS), sepsis, severe sepsis, and septic shock continuum11,12 ; however, SIRS has poor sensitivity for critical illness and high prevalence among children who are mildly ill.13  Other definitions use an intellectual construct: dysregulated host response to infection resulting in organ dysfunction.3,14  However, strategies for identifying a dysregulated immune system are nebulous with multiple criteria developed to assess for organ dysfunction (ie, Sequential [Sepsis-related] Organ Failure Assessment, Sequential [Sepsis-related] Organ Failure Assessment, PEdiatric Logistic Organ Dysfunction-2 scores).1416  These SIRS and organ dysfunction criteria differ in their reliance on laboratory values, validation in children, and generalizability across care settings,16,17  complicating efforts to use a common definition to guide clinical decision-making, research, and quality improvement.15 

Lacking accepted clinical definitions, researchers turned to intervention-based definitions that relied on provider behavior (eg, administering a sepsis bundle) rather than patient characteristics alone. Intervention-based definitions inherently overestimate sepsis score test performance because of confounding by medical intervention (positive alerts trigger interventions sufficient for the case definition regardless of whether the patient actually has sepsis) and inflate observed incidence. Balamuth et al18  found the rate of severe sepsis cases increased 1.5 times after implementation of a sepsis score when including an intervention-based definition in their analysis. Even with non–intervention-based definitions, screening with sepsis scores may trigger closer monitoring and laboratory evaluations that reveal otherwise insignificant abnormalities sufficient to meet consensus definitions of pediatric sepsis, resulting in higher observed incidence.

Sepsis scores in the ED improve sensitivity compared to physician judgment, but they also generate numerous false-positives, with reported positive predictive values between 4% and 25%.6,1821  Hospitalized children have less variable illness severities, which makes discriminating between patients with and without sepsis on the basis of subtle changes more challenging and error prone. Furthermore, repetitive screening throughout an entire hospitalization creates even more opportunities for increased testing and the discovery of mild, self-correcting abnormalities (eg, leukocytosis) that are sufficient in the setting of suspected infection for the care team to diagnose sepsis. Such overdiagnosis represents a true-positive epidemiologically, although the resultant interventions are unlikely to benefit the patient. Conversely, repetitive screening may increase false-positives, with one study finding positive predictive values 3 times lower in inpatient units compared to in the ED.21  Another study evaluating an exclusively inpatient sepsis score found that whereas only 4% of patients had possible or confirmed sepsis, 17% of all patients triggered alerts, 40% of whom alerted multiple times.22  Frequent and misdirected alerts from repetitive screening would undoubtedly contribute to alert fatigue.23 

Hospitalized children do not present de novo; an initial evaluation is completed, and their responses to interventions are observed, with multiple opportunities to reassess and individualize treatment plans on the basis of a known illness trajectory. Yet sepsis performance measures incentivize rapid treatment relative to a time zero, defined by scores agnostic to the underlying etiology (ie, 2 children with fever, tachycardia, and leukocytosis trigger identical alerts despite one having bronchiolitis and the other meningitis).24  Clinicians experience tension when seeking to reconcile recommendations for prompt and aggressive treatment with the difficulty of identifying sepsis in real-time while also pursuing other facets of the art of inpatient medicine, such as observation over time, safely doing less, and antimicrobial stewardship.25  When sepsis scores misfire, these incentives contribute to diagnostic error, overtreatment, and a feeling among some that sepsis response systems promote an inflexible, “treat first, ask questions later” approach.2527  Paradoxically, misfired sepsis alerts could delay care if care teams mistakenly anchor to a diagnosis of sepsis.

Given the difficulties in defining sepsis and developing effective sepsis scores, hospitals caring for children must use caution when implementing and evaluating sepsis response systems. The following approaches may help address the above challenges.

Sepsis scores should be developed and validated prospectively within populations not directly affected by their implementation. This can be accomplished by running the score “silently” in the electronic health record during development so that care teams do not see and act on it. After implementation, teams should monitor incidence rates unconfounded by medical intervention using recently developed pediatric sepsis surveillance definitions to assess their impact.28,29 

To address problems with repeated screening, hospitals may suppress alerts for a prespecified duration after each positive alert. This balances the need to reduce false-positives with the recognition that a patient’s condition may evolve, necessitating reevaluation of the diagnosis and plan. However, as demonstrated by Dr Carter’s patient, this may be insufficient over long hospitalizations. To improve the performance of sepsis scores for inpatient units, they should be developed by using longitudinal data from hospitalized children (incorporating patterns, eg, vital sign trends, and working diagnoses) rather than directly applying ED-derived, threshold-based scores. Newer machine-learning models may improve accuracy but have yet to be validated in practice.3032 

Another benefit of model-based approaches is the ability to report the likelihood of sepsis as a continuous probability, enabling graded rather than dichotomized responses.33  For example, low-risk patients could receive increased monitoring and supportive therapies (eg, antipyretics), with complete bundles reserved for the sickest patients with the highest likelihood of sepsis. Similar approaches to neonatal sepsis have reduced antibiotic use without adverse effects,34  and the most recent pediatric sepsis guidelines recommend 1-hour versus 3-hour time lines for antibiotic delivery on the basis of the presence or absence of shock.3  Defining acceptable risk thresholds and aligning performance measures with this approach will be important to ensure that prospective decisions made under uncertainty are not adjudicated against retrospectively assigned diagnoses lacking clinical context.35 

Sepsis response systems are complex interventions.36  Early identification and early response not only are dependent on sepsis scores alone but also incorporate clinical intuition and care team communication, both of which are bolstered by formalized tools and processes. Even highly accurate sepsis scores may fail in practice if frontline users find them untrustworthy, struggle applying them to individual patients, and fail to integrate alerts into standard workflows.37  Successful implementation acknowledges this complex interplay of interdependent components, ensures that technology supports (not overrides) clinical judgment, is mindful of challenges encountered by frontline users, and progresses iteratively while adapting to local context.35,36,38  This requires a multidisciplinary team (eg, nurses, physicians, phlebotomists, pharmacists, etc), training and feedback processes, and a functioning data infrastructure. Finally, implementation plans should, at the outset, consider processes for deimplementation if efforts reveal no improvement or harm.36,39 

Dr Carter’s experience reveals the burdens of alert fatigue and task switching associated with poorly performing sepsis scores. The care team in this case avoided the trap of automation bias and unnecessary interventions by thoughtfully considering the differential diagnosis and clinical implications of the alert before reflexively ordering tests and interventions. Overcoming these challenges requires sepsis response systems designed specifically for the patient populations and workflows found in inpatient settings. As such, sepsis scores should leverage longitudinal patient data to improve detection as opposed to treating it as a nuisance to overcome. The focus of preimplementation testing must be not only on achieving acceptable score performance but also on usability for frontline clinicians, and unbiased monitoring of postimplementation outcomes is necessary to ensure these efforts represent high-value care.

Dr Harrison conceptualized and designed the study, analyzed and interpreted the data, and drafted the initial manuscript; Dr Workman contributed to analysis and interpretation of the data and critically reviewed and revised the manuscript; Dr Bonafide contributed to analysis and interpretation of the data and critically reviewed the manuscript; Dr Lockwood conceptualized and designed the study and contributed to analysis and interpretation of the data, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.

FUNDING: Supported by the National Center for Advancing Translational Sciences of the National Institutes of Health (Award Number UL1TR002538), as well as by the Primary Children’s Hospital Foundation. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funded by the National Institutes of Health (NIH).

1
Carlton
EF
,
Barbaro
RP
,
Iwashyna
TJ
,
Prescott
HC
.
Cost of pediatric severe sepsis hospitalizations
.
JAMA Pediatr
.
2019
;
173
(
10
):
986
987
2
Davis
AL
,
Carcillo
JA
,
Aneja
RK
, et al
.
American college of critical care medicine clinical practice parameters for hemodynamic support of pediatric and neonatal septic shock
.
Crit Care Med
.
2017
;
45
(
6
):
1061
1093
3
Weiss
SL
,
Peters
MJ
,
Alhazzani
W
, et al
.
Surviving Sepsis Campaign international guidelines for the management of septic shock and sepsis-associated organ dysfunction in children
.
Pediatr Crit Care Med
.
2020
;
21
(
2
):
e52
e106
4
Sepsis Alliance
.
New Jersey now one of three states to mandate sepsis protocols
. Available at: https://www.sepsis.org/news/new-jersey-now-one-three-states-mandate-sepsis-protocols/. Accessed March 2, 2020
5
Children’s Hospital Association
.
Sepsis collaborative
. Available at: https://www.childrenshospitals.org/programs-and-services/quality-improvement-and-measurement/collaboratives/sepsis. Accessed March 2, 2020
6
Lane
RD
,
Funai
T
,
Reeder
R
,
Larsen
GY
.
High reliability pediatric septic shock quality improvement initiative and decreasing mortality
.
Pediatrics
.
2016
;
138
(
4
):
e20154153
7
Evans
IVR
,
Phillips
GS
,
Alpern
ER
, et al
.
Association between the New York sepsis care mandate and in-hospital mortality for pediatric sepsis
.
JAMA
.
2018
;
320
(
4
):
358
367
8
Guidi
JL
,
Clark
K
,
Upton
MT
, et al
.
Clinician perception of the effectiveness of an automated early warning and response system for sepsis in an academic medical center
.
Ann Am Thorac Soc
.
2015
;
12
(
10
):
1514
1519
9
Hranjec
T
,
Rosenberger
LH
,
Swenson
B
, et al
.
Aggressive versus conservative initiation of antimicrobial treatment in critically ill surgical patients with suspected intensive-care-unit-acquired infection: a quasi-experimental, before and after observational cohort study
.
Lancet Infect Dis
.
2012
;
12
(
10
):
774
780
10
Makam
AN
,
Nguyen
OK
,
Auerbach
AD
.
Diagnostic accuracy and effectiveness of automated electronic sepsis alert systems: a systematic review
.
J Hosp Med
.
2015
;
10
(
6
):
396
402
11
Goldstein
B
,
Giroir
B
,
Randolph
A
;
International Consensus Conference on Pediatric Sepsis
.
International pediatric sepsis consensus conference: definitions for sepsis and organ dysfunction in pediatrics
.
Pediatr Crit Care Med
.
2005
;
6
(
1
):
2
8
12
Kuch
BA
,
Carcillo
JA
,
Han
YY
,
Orr
RA
.
Definitions of pediatric septic shock
.
Pediatr Crit Care Med
.
2005
;
6
(
4
):
501; author reply 501
13
Scott
HF
,
Deakyne
SJ
,
Woods
JM
,
Bajaj
L
.
The prevalence and diagnostic utility of systemic inflammatory response syndrome vital signs in a pediatric emergency department
.
Acad Emerg Med
.
2015
;
22
(
4
):
381
389
14
Singer
M
,
Deutschman
CS
,
Seymour
CW
, et al
.
The third international consensus definitions for sepsis and septic shock (Sepsis-3)
.
JAMA
.
2016
;
315
(
8
):
801
810
15
Schlapbach
LJ
,
Kissoon
N
.
Defining pediatric sepsis
.
JAMA Pediatr
.
2018
;
172
(
4
):
312
314
16
Schlapbach
LJ
,
Straney
L
,
Bellomo
R
,
MacLaren
G
,
Pilcher
D
.
Prognostic accuracy of age-adapted SOFA, SIRS, PELOD-2, and qSOFA for in-hospital mortality among children with suspected infection admitted to the intensive care unit
.
Intensive Care Med
.
2018
;
44
(
2
):
179
188
17
van Nassau
SC
,
van Beek
RH
,
Driessen
GJ
,
Hazelzet
JA
,
van Wering
HM
,
Boeddha
NP
.
Translating Sepsis-3 criteria in children: prognostic accuracy of age-adjusted quick SOFA score in children visiting the emergency department with suspected bacterial infection
.
Front Pediatr
.
2018
;
6
:
266
18
Balamuth
F
,
Alpern
ER
,
Abbadessa
MK
, et al
.
Improving recognition of pediatric severe sepsis in the emergency department: contributions of a vital sign-based electronic alert and bedside clinician identification
.
Ann Emerg Med
.
2017
;
70
(
6
):
759
768.e2
19
Balamuth
F
,
Alpern
ER
,
Grundmeier
RW
, et al
.
Comparison of two sepsis recognition methods in a pediatric emergency department
.
Acad Emerg Med
.
2015
;
22
(
11
):
1298
1306
20
Cruz
AT
,
Williams
EA
,
Graf
JM
, et al
.
Test characteristics of an automated age- and temperature-adjusted tachycardia alert in pediatric septic shock
.
Pediatr Emerg Care
.
2012
;
28
(
9
):
889
894
21
Eisenberg
M
,
Madden
K
,
Christianson
JR
,
Melendez
E
,
Harper
MB
.
Performance of an automated screening algorithm for early detection of pediatric severe sepsis
.
Pediatr Crit Care Med
.
2019
;
20
(
12
):
e516
e523
22
Bradshaw
C
,
Goodman
I
,
Rosenberg
R
,
Bandera
C
,
Fierman
A
,
Rudy
B
.
Implementation of an inpatient pediatric sepsis identification pathway
.
Pediatrics
.
2016
;
137
(
3
):
e20144082
23
Nelson
B
. Preventing sepsis alert fatigue.
The Hospitalist
. January 31, 2018. Available at: https://www.the-hospitalist.org/hospitalist/article/157480/critical-care/preventing-sepsis-alert-fatigue. Accessed June 2, 2020
24
Odetola
FO
,
Freed
G
,
Shevrin
C
,
Madden
B
,
McCormick
J
,
Dombkowski
K
.
In-hospital quality-of-care measures for pediatric sepsis syndrome
.
Pediatrics
.
2017
;
140
(
2
):
e20170350
25
IDSA Sepsis Task Force
.
Infectious Diseases Society of America (IDSA) position statement: why IDSA did not endorse the Surviving Sepsis Campaign guidelines
.
Clin Infect Dis
.
2018
;
66
(
10
):
1631
1635
26
Schlapbach
LJ
,
Weiss
SL
,
Wolf
J
.
Reducing collateral damage from mandates for time to antibiotics in pediatric sepsis-primum non nocere
.
JAMA Pediatr
.
2019
;
173
(
5
):
409
410
27
Klompas
M
,
Calandra
T
,
Singer
M
.
Antibiotics for sepsis-finding the equilibrium
.
JAMA
.
2018
;
320
(
14
):
1433
1434
28
Hsu
HE
,
Abanyie
F
,
Agus
MSD
, et al
.
A national approach to pediatric sepsis surveillance
.
Pediatrics
.
2019
;
144
(
6
):
e20191790
29
Weiss
SL
,
Balamuth
F
,
Chilutti
M
, et al
.
Identification of pediatric sepsis for epidemiologic surveillance using electronic clinical data
.
Pediatr Crit Care Med
.
2020
;
21
(
2
):
113
121
30
Huff
C
.
Preventing sepsis by reimagining systems and engaging patients
.
Health Aff (Millwood)
.
2019
;
38
(
5
):
704
708
31
Le
S
,
Hoffman
J
,
Barton
C
, et al
.
Pediatric severe sepsis prediction using machine learning
.
Front Pediatr
.
2019
;
7
:
413
32
Taylor
RA
,
Pare
JR
,
Venkatesh
AK
, et al
.
Prediction of in-hospital mortality in emergency department patients with sepsis: a local big data-driven, machine learning approach
.
Acad Emerg Med
.
2016
;
23
(
3
):
269
278
33
Prescott
HC
,
Iwashyna
TJ
.
Improving sepsis treatment by embracing diagnostic uncertainty
.
Ann Am Thorac Soc
.
2019
;
16
(
4
):
426
429
34
Kuzniewicz
MW
,
Puopolo
KM
,
Fischer
A
, et al
.
A quantitative, risk-based approach to the management of neonatal early-onset sepsis
.
JAMA Pediatr
.
2017
;
171
(
4
):
365
371
35
Chartash
D
,
Sassoon
D
,
Muthu
N
.
Physicians in the era of automation: the case for clinical expertise
.
MDM Policy Pract
.
2019
;
4
(
2
):
2381468319868968
36
Craig
P
,
Dieppe
P
,
Macintyre
S
,
Michie
S
,
Nazareth
I
,
Petticrew
M
.
Developing and evaluating complex interventions: the new Medical Research Council guidance
.
BMJ
.
2008
;
337
:
a1655
37
Ginestra
JC
,
Giannini
HM
,
Schweickert
WD
, et al
.
Clinician perception of a machine learning-based early warning system designed to predict severe sepsis and septic shock
.
Crit Care Med
.
2019
;
47
(
11
):
1477
1484
38
Datta
J
,
Petticrew
M
.
Challenges to evaluating complex interventions: a content analysis of published papers
.
BMC Public Health
.
2013
;
13
:
568
39
Prasad
V
,
Ioannidis
JP
.
Evidence-based de-implementation for contradicted, unproven, and aspiring healthcare practices
.
Implement Sci
.
2014
;
9
:
1

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.