BACKGROUND AND OBJECTIVES

There is widespread unwarranted antibiotic use and large individual provider variation in antibiotic use in NICUs. Vignette-based research methodology offers a unique method of studying variation in individual provider decisions. The objective with this study was to use a vignette-based survey to identify specific areas of provider antibiotic use variation in newborns being evaluated for early onset sepsis.

METHODS

This study was undertaken as part of a statewide multicenter neonatal antibiotic stewardship quality improvement project led by a perinatal quality improvement collaborative. A web-based vignette survey was administered to identify variation in decisions to start and discontinue antibiotics in cases of early onset sepsis.

RESULTS

The largest variation was noted in 3 of the 6 vignette cases. These cases highlighted variation in (1) decisions to start antibiotics in a case describing a well-appearing newborn with risk factors and an elevated C-reactive protein, (2) decisions to start antibiotics in the case of a newborn with risk factors plus mild respiratory signs at birth, and (3) decisions to stop antibiotics in the case of the newborn with a history of sepsis risk factors and mild clinical respiratory signs that resolved after 72 hours.

CONCLUSIONS

Clinical vignette assessment identified specific areas of variation in individual provider antibiotic use decisions in cases of suspected early onset sepsis. Vignettes are a valuable method of describing individual provider variation and highlighting antibiotic stewardship improvement opportunities in NICUs.

There is wide variation and overuse of antibiotics in NICUs. Antibiotic use rates are not correlated with rates of proven infection.1  Reducing variation is a fundamental aspect of improving health care, and some of this variation is attributable to individual provider style and preference.2  Antibiotic use in early onset sepsis (EOS) is particularly susceptible to variation because of common nonspecific respiratory signs in newborns, lack of rapid and accurate diagnostic tests, and guidelines previously recommending what is now considered unnecessary antibiotic use.

Vignette-based research methods are ideal for describing variation in physician decisions.38  Vignette methodology capitalizes on the strengths and avoids the weaknesses of both survey and experimental research methods. Vignettes have been used to understand provider variation in NICU antibiotic prescribing9,10  and to understand specific antibiotic stewardship challenges faced by pediatricians.11 

Collaborative quality improvement (QI) projects are increasingly being performed to safely reduce unnecessary antibiotic use. Understanding the decisions driving variability in antibiotic use among clinicians is a vital step in reducing antibiotic use. The objective of this study was to use clinical vignette methodology to identify specific areas of variation in EOS antibiotic use.

This study was conducted during a statewide antibiotic stewardship QI collaborative in the western United States. The collaborative aimed to reduce antibiotic use by targeting EOS antibiotic use. The collaborative included 28 NICUs. The collaborative began on June 3, 2016, and ended on December 14, 2017. In this study, we used a vignette survey of individual participants that was emailed on January 19, 2017, with responses collected until April 4, 2017.

A previously published set of EOS clinical case vignettes was used as a model for development of the vignette cases.10  The specific clinical case scenarios were determined from discussions with QI team participants at in-person meetings and monthly webcasts. The vignette survey was piloted with the stewardship collaborative expert panel members. Two separate vignette series, with 3 questions in each series, were created. Vignette series A targeted whether respondents would start antibiotics in the case of a newborn being evaluated for EOS. Vignette series B targeted whether respondents would discontinue antibiotics or continue antibiotics for a treatment course after 48 hours.

Each case vignette included 1 multiple-choice question that allowed for free text answers, if desired (Table 1). The web-based vignette survey was sent to the team leader from each of the 28 participating NICUs. The team leaders were instructed to forward the survey to individuals responsible for ordering antibiotics in their NICU. Surveys were created and distributed by using Qualtrics Survey Platform (Qualtrics, Provo, UT).12  Cedars-Sinai Medical Center Office of Research Compliance and QI approved a waiver of consent for this QI study.

The quantitative (multiple choice) and qualitative (free text) survey responses were consolidated into binary results to reflect either “starting antibiotics” or “not starting antibiotics” and “discontinuing antibiotics” or “treating for 5 days or longer” for vignette sets A and B, respectively. Two of the study authors separately (K.S.E.P. and C.P.) coded all the qualitative survey responses that were free texted as the “other option.” The 2 authors obtained similar results.

A coefficient of unalikeability (CU)13  was calculated to describe the degree of variability between the final binary results. The CU describes how often answers differed from each other, with 0 representing minimum variability and 0.5 representing maximum variability. Fisher’s exact test and mixed logistics regression were used to determine if responses differed by NICU site or role of the individual respondents. We also estimated the intraclass correlation (ICC), which gives the proportion of the total variation at the site level on the basis of the null model of mixed logistics regression accounting for sites as a random effect. Analysis was computed by using SAS 9.4 (SAS Institute, Inc, Cary, NC).14 

We received surveys from at least 1 participant from 16 of the 28 participating NICUs. Respondents included neonatologists (n = 73), hospitalists (n = 15), neonatal nurse practitioners (n = 11), neonatal-perinatal medicine fellows (n = 6), and clinical nurse specialists (n = 2). The range of total number of binary responses for each question analyzed, resulting from the sum of coding of each multiple-choice and free text response, varied from 106 to 116 for each question (Tables 2 and 3).

The most variation (CU = 0.5) was noted in the decision to discontinue antibiotics in the case with “respiratory signs for 72 hours, plus risk factors” (case B3). The other 2 cases with the largest variation included decisions to start antibiotics for the cases describing a newborn who was “well appearing, with elevated CRP, and risk factors” (case A1) and a newborn with “respiratory signs and no risk factors” (case A2), both with a CU of 0.42. The 3 cases with the least variation had CUs of 0.05, 0.19, and 0.32 (Tables 2 and 3).

Analysis of responses by site suggested that 1 NICU had respondents more likely than other sites to choose to stop antibiotics in the case of “respiratory signs for 72 hours plus risk factors” (case B3), but this difference was not statistically significant (P = .09). There were no other notable differences in responses by site or role.

Individual practice variation is a barrier to reducing unnecessary antibiotic use in NICUs. Four helpful targets for antibiotic stewardship efforts emerged from this study related to perceptions of culture negative sepsis, response to elevated C-reactive protein (CRP), response to varying respiratory signs, and response to sepsis risk factors.

First, the largest variation in responses occurred with the decisions to discontinue antibiotics in the case of “respiratory signs for 72 hours plus risk factors” (case B3). Not surprisingly, the responses to this case may reflect a lack of evidence-based consensus regarding assessment of newborns with risk factors and persistent respiratory signs of unclear etiology. Discussions among individual participants during webcasts and face-to-face sessions highlighted that those choosing to continue antibiotic treatment interpreted this scenario as representing “culture negative sepsis.” Alternatively, respondents electing to discontinue antibiotics cited the negative blood culture, lack of critical illness, and resolution of clinical signs to suggest a noninfectious etiology. There continues to be controversy regarding “culture negative sepsis,” with one perspective suggesting that this diagnosis is less common and less relevant than previously thought.15  The American Association of Pediatrics clinical report on management of infants at ≥35 0/7 weeks’ gestation with suspected or proven culture positive EOS states that “when blood cultures are sterile, antibiotics should be discontinued at 36 to 48 hours, unless there is site specific evidence of infection”.16  Antibiotic stewardship efforts should address provider perceptions of culture negative sepsis to reduce variation in antibiotic use.

Second, there was notable variation in the antibiotic decisions based on an elevated CRP in a well-appearing newborn (case A1). Current evidence suggests that starting antibiotics for an elevated CRP alone in this scenario is not warranted.1618  Additionally, abnormal CBC indices alone and/or serial CRP or other inflammatory markers are not recommended to inform decisions to start or continue antibiotics in well-appearing newborns.16  Targeting management variation of well-appearing newborns with risk factors and abnormal screening laboratories may be beneficial to stewardship teams.

Third, there was variation in the decision to start antibiotics in the newborn with “respiratory signs and no sepsis risk factors” (case A2). These results suggest that obtaining consensus regarding respiratory sign thresholds prompting antibiotic use could be helpful to minimize variation in starting antibiotics in low risk newborns. Additionally, it highlights that consensus on the duration of time that is acceptable for close monitoring without antibiotics could help reduce variation.

Last, the addition of antenatal maternal risk factors (fever and prolonged rupture of membranes) with all other case variables remaining unchanged prompted a large shift in respondents choosing to start antibiotics (from 29% to 84%, respectively) (case A2 and A3, Table 2). This judgment is consistent with both clinical observations19  and risk modeling,20  which indicates that risk indicators often compound one another. Thus, standardizing provider responses to EOS risk factors, such as those incorporated in risk-based decision aids such as the neonatal sepsis calculator,21,22  may reduce variation and unnecessary treatment.

The study had several important methodologic limitations. Some vignette details could have been interpreted differently by respondents. This could have impacted variation in the results. We simplified some of the laboratory elements in the vignettes and this may create ambiguity and variability in responses. We did not determine the total proportion of eligible providers at a given site that received the survey. The results may reflect selection bias in an unknown direction because our results do not represent all participating sites and may include a low proportion of eligible respondents. Finally, the study was not powered to detect differences by role.

Clinical vignettes identified specific opportunities for antibiotic stewardship efforts related to EOS. Targeting clinicians’ antibiotic decisions related to perspectives on culture negative sepsis, abnormal screening laboratory results, threshold of tolerance for respiratory signs, and thresholds of antibiotic response to antenatal maternal risk factors may benefit antibiotic stewardship efforts in NICUs. Stakeholders implementing antibiotic stewardship should prioritize these areas and/or implement customized vignette assessments to understand individual antibiotic use variation within their local context.

The authors thank Jeff Merrill, MD, Linda Lefrak, RN, MS, CNS, and the respondents from participating NICUs for their contributions to this study.

Dr Payton conceptualized the study, developed the vignette cases, coded the survey responses, and drafted the initial manuscript; Dr Bennett and Ms Kristensen-Cabrera conceptualized and performed statistical analysis; Ms Nisbet supervised data collection; Dr Parker performed data analysis and coded the survey responses; and all authors reviewed, revised, and approved the final manuscript.

FUNDING: No external funding.

1.
Schulman
J
,
Dimand
RJ
,
Lee
HC
,
Duenas
GV
,
Bennett
MV
,
Gould
JB
.
Neonatal intensive care unit antibiotic use
.
Pediatrics
.
2015
;
135
(
5
):
826
833
2.
Balakrishnan
M
,
Raghavan
A
,
Suresh
GK
.
Eliminating undesirable variation in neonatal practice: balancing standardization and customization
.
Clin Perinatol
.
2017
;
44
(
3
):
529
540
3.
Converse
L
,
Barrett
K
,
Rich
E
,
Reschovsky
J
.
Methods of observing variations in physicians’ decisions: the opportunities of clinical vignettes
.
J Gen Intern Med
.
2015
;
30
(
suppl 3
):
S586
S594
4.
Evans
SC
,
Roberts
MC
,
Keeley
JW
, et al
.
Vignette methodologies for studying clinicians’ decision-making: validity, utility, and application in ICD-11 field studies
.
Int J Clin Health Psychol
.
2015
;
15
(
2
):
160
170
5.
Veloski
J
,
Tai
S
,
Evans
AS
,
Nash
DB
.
Clinical vignette-based surveys: a tool for assessing physician practice variation
.
Am J Med Qual
.
2005
;
20
(
3
):
151
157
6.
Lucet
JC
,
Nicolas-Chanoine
MH
,
Lefort
A
, et al
.
Do case vignettes accurately reflect antibiotic prescription?
Infect Control Hosp Epidemiol
.
2011
;
32
(
10
):
1003
1009
7.
Peabody
JW
,
Luck
J
,
Glassman
P
,
Dresselhaus
TR
,
Lee
M
.
Comparison of vignettes, standardized patients, and chart abstraction: a prospective validation study of 3 methods for measuring quality
.
JAMA
.
2000
;
283
(
13
):
1715
1722
8.
Peabody
JW
,
Luck
J
,
Glassman
P
, et al
.
Measuring the quality of physician practice by using clinical vignettes: a prospective validation study
.
Ann Intern Med
.
2004
;
141
(
10
):
771
780
9.
Patel
S
,
Landers
T
,
Larson
E
, et al
.
Clinical vignettes provide an understanding of antibiotic prescribing practices in neonatal intensive care units
.
Infect Control Hosp Epidemiol
.
2011
;
32
(
6
):
597
602
10.
van Herk
W
,
el Helou
S
,
Janota
J
, et al
.
Variation in current management of term and late-preterm neonates at risk for early-onset sepsis: an international survey and review of guidelines
.
Pediatr Infect Dis J
.
2016
;
35
(
5
):
494
500
11.
Bowes
J
,
Yasseen
AS
 III
,
Barrowman
N
, et al
.
Antimicrobial stewardship in pediatrics: focusing on the challenges clinicians face
.
BMC Pediatr
.
2014
;
14
:
212
12.
Qualtrics [software program]
.
2017
.
Provo, UT
:
Qualtrics
. .
13.
Kader
G
,
Perry
M
.
Variability for categorical variables
.
J Stat Educ
.
2007
;
15
(
2
)
14.
SAS software [computer program]
.
Version 9.4
.
Cary, NC
:
SAS Institute
15.
Cantey
JB
,
Baird
SD
.
Ending the culture of culture-negative sepsis in the neonatal ICU
.
Pediatrics
.
2017
;
140
(
4
):
e20170044
16.
Puopolo
KM
,
Benitz
WE
,
Zaoutis
TE
;
Committee on Fetus and Newborn
;
Committee on Infectious Diseases
.
Management of neonates born at ≥35 0/7 weeks’ gestation with suspected or proven early-onset bacterial sepsis
.
Pediatrics
.
2018
;
142
(
6
):
e20182894
17.
Perrone
S
,
Lotti
F
,
Longini
M
, et al
.
C reactive protein in healthy term newborns during the first 48 hours of life
.
Arch Dis Child Fetal Neonatal Ed
2018
;
103
(
2
):
F163
F166
18.
Mjelle
AB
,
Guthe
HJT
,
Reigstad
H
,
Bjørke Monsen
AL
,
Markestad
T
.
Serum concentrations of C-reactive protein in healthy term-born Norwegian infants 48-72 hours after birth
.
Acta Paediatr
.
2019
;
108
(
5
):
849
854
19.
Philip
AG
.
Neonatal sepsis resulting from possible amniotic fluid infection: risk and detection
.
Clin Pediatr (Phila)
.
1982
;
21
(
4
):
210
214
20.
Escobar
GJ
,
Puopolo
KM
,
Wi
S
, et al
.
Stratification of risk of early-onset sepsis in newborns ≥ 34 weeks’ gestation
.
Pediatrics
.
2014
;
133
(
1
):
30
36
21.
Kuzniewicz
MW
,
Walsh
EM
,
Li
S
,
Fischer
A
,
Escobar
GJ
.
Development and Implementation of an early-onset sepsis calculator to guide antibiotic management in late preterm and term neonates
.
Jt Comm J Qual Patient Saf
.
2016
;
42
(
5
):
232
239
22.
Kuzniewicz
MW
,
Puopolo
KM
,
Fischer
A
, et al
.
A quantitative, risk-based approach to the management of neonatal early-onset sepsis
.
JAMA Pediatr
.
2017
;
171
(
4
):
365
371

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.