Video Abstract

Video Abstract

Close modal
BACKGROUND AND OBJECTIVES

Short courses of antibiotic treatment are effective for pediatric community-acquired pneumonia (CAP) and skin and soft tissue infections (SSTI). We compared the effectiveness of education with performance feedback, clinical decision support (CDS), and the combination in encouraging appropriately short treatment courses by primary care clinicians.

METHODS

We designed a site-randomized, quality improvement trial within a large pediatric primary care network. Each practice was randomly assigned to 1 of 4 groups: education and feedback; CDS; both interventions (“combined group”); and control. We performed difference-in-differences analysis to compare the proportion of cases with short course treatment before and after intervention among the 4 groups.

RESULTS

For all cases of CAP and SSTI, the proportion in the control group treated with the recommended duration did not change from the baseline period (26.1% [679 of 2603]) to the intervention period (25.8% [196 of 761]; P = .9). For the education and feedback group, the proportion rose from 22.3% (428 of 1925) to 45.0% (239 of 532; P < .001); for the CDS group, from 26.6% (485 of 1824) to 52.3% (228 of 436; P < .001); and for the combined group, from 26.2% (491 of 1875) to 67.8% (314 of 463; P < .001). A difference-in-differences analysis showed that all 3 intervention groups improved performance compared with the control group (P < .001); the combined group had greater improvement than the education and feedback group or the CDS group (P < .001).

CONCLUSIONS

In this quality improvement project to encourage shorter duration treatment of CAP and SSTI, both education with performance feedback and CDS were effective in modifying clinician behavior; however, the combination of the two was substantially more effective than either strategy alone.

What’s Known on This Subject:

Shorter courses of antibiotic treatment are effective for treatment of community-acquired pneumonia and skin and soft tissue infections, but many clinicians still prescribe longer courses of treatment. It is unclear what quality improvement techniques could successfully bridge this gap.

What This Study Adds:

This site-randomized quality improvement trial demonstrated that education and feedback plus clinical decision support at the point of care improved guideline-adherent antibiotic prescribing more than either intervention alone.

Overuse of antibiotics contributes substantially to antimicrobial resistance, a significant public health problem.1,2  Furthermore, the fact that a large majority of antibiotics are prescribed in the outpatient setting suggests a need for robust outpatient antimicrobial stewardship efforts.2  One way to limit antibiotic exposure in the community, and concurrently to minimize antibiotic-associated adverse effects, is to use the shortest effective duration of treatment when prescribing antibiotics for common infections.3 

Many pediatric clinicians were traditionally taught to treat common bacterial infections with antibiotics for 10 days. However, recent evidence has demonstrated that shorter durations for 2 common pediatric infections—5 days for uncomplicated community-acquired pneumonia (CAP)4 9  and 5 to 7 days for uncomplicated skin and soft tissue infections (SSTI)10 12 —are not inferior to longer courses and confer a lower risk of antibiotic-associated adverse effects. Based on this evidence, in 2021 the American Academy of Pediatrics Committee on Infectious Diseases revised its treatment guidelines for these conditions.13 

When changes in best practice recommendations such as these are made by expert bodies, it is often challenging to prompt front-line clinicians to change long-ingrained practices.14  Two techniques that have been found effective in various antimicrobial stewardship projects include education with performance feedback and clinical decision support (CDS) delivered at the point of care.15  To our knowledge, these techniques have not been tested head-to-head or in combination in pediatric outpatient antimicrobial stewardship efforts. Therefore, to test the effectiveness of education with performance feedback and CDS on encouraging appropriate short-course antibiotic treatment of pediatric CAP and SSTI, we designed a site-randomized quality improvement trial to test the effect of each of these interventions alone and in combination. We hypothesized that each intervention would likely improve performance relative to the control group and that there may be a synergistic effect of using a combination of the 2 interventions together.

The Pediatric Physicians’ Organization at Children’s is an independent practice association of 76 privately-owned pediatric practices with ∼500 pediatric primary care clinicians (PCC; generally, physicians and pediatric nurse practitioners) affiliated with Boston Children’s Hospital. The network provides primary care to ∼400 000 children throughout Massachusetts. Practices range in size from 1 to 39 PCCs. All practices use a centrally-hosted electronic health record (Epic, Verona, WI). Our electronic health record build requires all orders, including medication prescriptions, to be linked to 1 or more International Classification of Diseases, Tenth Revision (ICD-10) diagnoses.

We created a driver diagram to establish aims and elucidate the drivers of clinical behavior not in keeping with current best practice recommendations (Fig 1).

FIGURE 1

Quality improvement project driver diagram.

FIGURE 1

Quality improvement project driver diagram.

Close modal

Seventy five practices were randomly assigned in blocks of 4 to 1 of 4 intervention groups using a random number generator: education and feedback; CDS; both education and feedback plus CDS (hereafter referred to as the “combined group”); and control (no intervention). One practice was excluded from the project because of it being an outlier on baseline data. The baseline period was defined as calendar year 2022; the intervention period was defined as the 3-month period starting with the launch of the initiative on February 15, 2023 and ending on May 14, 2023. Cases of CAP were defined as encounters for patients 6 months through 18 years of age with an ICD-10 diagnosis of J13×, J14×, J15×, J16×, and/or J18× linked to a prescription for an oral antibiotic, and cases of SSTI were defined as encounters for the same age group with an ICD-10 diagnosis of L00×, L01×, L02×, L03×, L08.89, and/or L08.9 linked to a prescription for an oral antibiotic.

PCCs within practices assigned to the education and feedback group received a personal e-mail from one of the authors (L.V.) at the outset of the project explaining that the Committee on Infectious Diseases of the American Academy of Pediatrics (the “Red Book Committee”) recommends limiting the duration of antibiotic treatment of uncomplicated CAP to 5 days and for uncomplicated SSTI to 5 to 7 days. The e-mail also shared data on the performance of the individual PCC and their practice for CAP and SSTI for the baseline period and the goals for each deemed achievable by project leaders—50% for CAP and 67% for SST (Supplemental Fig 3). An infographic was also attached to the e-mail that could be printed and displayed in the PCC’s work area (Supplemental Fig 4). One month and 2 months into the project period, each PCC in the education and feedback group received an e-mail reminding them of the recommendations and updating them on their performance since the previous e-mail.

PCCs within practices assigned to the CDS group did not receive education or any performance feedback relative to the initiative. If they prescribed an antibiotic linked to a diagnosis of CAP with a duration of greater than 5 days, or to a diagnosis of SSTI with a duration greater than 7 days, they received a pop-up advisory when they attempted to sign the prescription alerting them to the relevant recommendation (Supplemental Fig 5). The alert was a “hard stop,” meaning that the prescriber was required to respond in some way to continue their work. Options included altering the prescription to comply with the recommended duration or acknowledging the alert and sending the prescription with the originally selected duration.

PCCs within practices assigned to the Combined group received the education with performance feedback emails as described above, plus the CDS prompts described above if they attempted to sign a prescription not in compliance with the recommendations.

PCCs in practices assigned to the control group were not notified about the initiative and did not receive any interventions.

Based on the proportion of cases of adherent to the guidelines in the baseline period and the number of monthly cases of CAP and SSTI in our network, we estimated that we would need to run the trial for 3 months to have sufficient power to detect significant differences between the intervention groups and the control group at an α of .05.

The project was deemed to meet the definition of quality improvement by Boston Children’s Hospital and was therefore exempt from Institutional Review Board review.

For analysis of the outcomes, we excluded encounters that, in addition to the diagnosis of CAP or SSTI, included a common pediatric diagnosis for which a longer duration of antibiotics may be recommended; these included otitis media (ICD-10 codes H65.×, H66.×), sinusitis (J01.xx, J32.xx), and streptococcal pharyngitis (J02.2).

For the preplanned primary analysis, changes between the proportions of cases of CAP treated with antibiotics for 5 days or less plus the proportion of cases of SSTI treated with antibiotics for 7 days or less were compared between the baseline period (calendar year 2022) and the entire intervention period (Feb 15, 2023–May 14, 2023) across the 4 randomization groups through difference-in-differences analysis accounting for repeated observations using logistic regression with nested random effects terms for individual clinician and practice (SAS version 9.4, Cary, NC). Preplanned secondary analyses included analysis of compliance with the recommendation for each condition separately. Run charts were created to visualize the timing of any changes observed within the intervention period.

A comparison of characteristics of the practices randomized to each of the 4 intervention groups is shown in Table 1.

TABLE 1

Baseline Comparison of Randomization Groups

ControlEducation and FeedbackClinical Decision SupportEducation and Feedback Plus Clinical Decision Support
Number of practices 19 19 18 19 
Number of primary care clinicians 152 85 112 85 
Number of primary care clinicians per practice, median (range) 9 (1–39) 5 (1–15) 6.5 (1–26) 5 (1–28) 
Number of patients per practice, median (range) 6346 (321–31 196) 3112 (519–10 557) 4275 (598–19 157) 2610 (407–19 680) 
Practice location, number (%) Suburban: 13 (68.4); urban: 6 (31.6) Suburban: 12 (63.2); urban: 6 (36.8) Suburban: 11 (61.1); urban: 7 (38.9) Suburban: 10 (52.6); urban: 9 (47.4) 
Clinician age in years, median (range) 52 (32–77) 54 (34–68) 56 (33–83) 54 (36–83) 
Clinician sex, number (%) Female: 112 (73.7); male: 40 (26.3) Female: 69 (81.2); male: 16 (18.8) Female: 78 (69.6); male: 34 (30.4) Female: 65 (76.5); male: 20 (23.5) 
Clinician type, number (%) Physician: 120 (78.9); nurse practitioner: 32 (21.1) Physician: 72 (84.7); nurse practitioner: 13 (15.3) Physician: 95 (84.8); nurse practitioner: 17 (15.2) Physician: 69 (81.2); nurse practitioner: 16 (18.8) 
ControlEducation and FeedbackClinical Decision SupportEducation and Feedback Plus Clinical Decision Support
Number of practices 19 19 18 19 
Number of primary care clinicians 152 85 112 85 
Number of primary care clinicians per practice, median (range) 9 (1–39) 5 (1–15) 6.5 (1–26) 5 (1–28) 
Number of patients per practice, median (range) 6346 (321–31 196) 3112 (519–10 557) 4275 (598–19 157) 2610 (407–19 680) 
Practice location, number (%) Suburban: 13 (68.4); urban: 6 (31.6) Suburban: 12 (63.2); urban: 6 (36.8) Suburban: 11 (61.1); urban: 7 (38.9) Suburban: 10 (52.6); urban: 9 (47.4) 
Clinician age in years, median (range) 52 (32–77) 54 (34–68) 56 (33–83) 54 (36–83) 
Clinician sex, number (%) Female: 112 (73.7); male: 40 (26.3) Female: 69 (81.2); male: 16 (18.8) Female: 78 (69.6); male: 34 (30.4) Female: 65 (76.5); male: 20 (23.5) 
Clinician type, number (%) Physician: 120 (78.9); nurse practitioner: 32 (21.1) Physician: 72 (84.7); nurse practitioner: 13 (15.3) Physician: 95 (84.8); nurse practitioner: 17 (15.2) Physician: 69 (81.2); nurse practitioner: 16 (18.8) 

For all cases of CAP and SSTI combined and for each condition individually, the proportion of cases in the control group treated with the recommended short course did not change from the baseline period (26.1% for both conditions combined, 4.3% for CAP, 37.5% for SSTI) to the intervention period (25.8% combined, 6.2% for CAP, 36.7% for SSTI; P = .9 for both conditions combined, P = .2 for CAP, P = .7 for SSTI; Table 2). In contrast, within all 3 of the intervention groups, there were statistically significant changes from baseline to the intervention period. For the education and feedback group, the proportion of all cases treated with a recommended short course rose from 22.3% to 45.0%; from 1.7% to 18.1% for CAP; and from 34.7% to 59.6% for SSTI (P < .001 for all). For the CDS group, the proportion of all cases treated with a recommended short course rose from 26.6% to 52.3%; from 3.4% to 24.1% for CAP; and from 39.0% to 64.7% for SSTI (P < .001 for all). For the Combined group, the proportion of all cases treated with a recommended short course rose from 26.2% to 67.8%; from 3.0% to 51.1% for CAP; and from 40.9% to 79.3% for SSTI (P < .001 for all).

TABLE 2

Compliance With Short-Course Antibiotic Treatment of Baseline and Intervention Periods by Randomization Group

All CasesCommunity-acquired Pneumonia CasesSkin and Soft Tissue Infection Cases
Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)
Control 679/2603 (26.1) 196/761 (25.8) −0.3 (−3.9 to 3.2) 39/899 (4.3) 17/273 (6.2) 1.9 (−1.0 to 4.8) 640/1704 (37.5) 179/488 (36.7) −0.8 (−5.7 to 4.0) 
Education and feedback 428/1925 (22.3) 239/532 (45.0) 22.7 (18.5 to 26.9)a 12/725 (1.7) 34/188 (18.1) 16.4 (13.1 to 19.8)a 416/1200 (34.7) 205/344 (59.6) 24.9 (19.2 to 30.7)a 
Clinical decision support 485/1824 (26.6) 228/436 (52.3) 25.7 (21.0 to 30.4)a 22/636 (3.4) 32/133 (24.1) 20.6 (16.1 to 25.2)a 463/1188 (39.0) 196/303 (64.7) 25.7 (19.6 to 31.8)a 
Education and feedback plus clinical decision support 491/1875 (26.2) 314/463 (67.8) 41.6 (37.1 to 46.2)a,b,c 22/728 (3.0) 96/188 (51.1) 48.1 (43.7 to 52.4)a,b,c 469/1147 (40.9) 218/275 (79.3) 38.4 (32.1 to 44.7)a,d,e 
All CasesCommunity-acquired Pneumonia CasesSkin and Soft Tissue Infection Cases
Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)Baseline PeriodIntervention PeriodAbsolute Percent Change (95% CI)
Control 679/2603 (26.1) 196/761 (25.8) −0.3 (−3.9 to 3.2) 39/899 (4.3) 17/273 (6.2) 1.9 (−1.0 to 4.8) 640/1704 (37.5) 179/488 (36.7) −0.8 (−5.7 to 4.0) 
Education and feedback 428/1925 (22.3) 239/532 (45.0) 22.7 (18.5 to 26.9)a 12/725 (1.7) 34/188 (18.1) 16.4 (13.1 to 19.8)a 416/1200 (34.7) 205/344 (59.6) 24.9 (19.2 to 30.7)a 
Clinical decision support 485/1824 (26.6) 228/436 (52.3) 25.7 (21.0 to 30.4)a 22/636 (3.4) 32/133 (24.1) 20.6 (16.1 to 25.2)a 463/1188 (39.0) 196/303 (64.7) 25.7 (19.6 to 31.8)a 
Education and feedback plus clinical decision support 491/1875 (26.2) 314/463 (67.8) 41.6 (37.1 to 46.2)a,b,c 22/728 (3.0) 96/188 (51.1) 48.1 (43.7 to 52.4)a,b,c 469/1147 (40.9) 218/275 (79.3) 38.4 (32.1 to 44.7)a,d,e 

CI, confidence interval.

a

P < .001 compared with control group.

b

P < .001 compared with education and feedback group.

c

P < .001 compared with clinical decision support group.

d

P = .02 compared with education and feedback group.

e

P = .02 compared with clinical decision support group.

The difference-in-differences analysis for CAP and SSTI combined and for each condition analyzed separately showed that all 3 intervention groups had statistically improved performance compared with the control group (P < .001). Improvement in the education and feedback group was statistically indistinguishable from the CDS group (P > .05) for the 2 conditions combined and for each condition individually. For the conditions combined and for CAP and SSTI analyzed individually, the combined group had significantly greater improvement compared with the education and feedback group and to the CDS group (P < .001 for both conditions combined and for CAP; P = .02 for SSTI).

Run charts for CAP and SSTI stratified by the 4 randomization groups are shown in Fig 2. For CAP, the combined group showed the most rapid improvement and met the prespecified goal of ≥50% in the second and third months of the intervention period. For SSTI, the combined group again showed the most rapid improvement and met the prespecified goal of ≥67% in all 3 months of the intervention period; the education and feedback and the CDS groups both met the prespecified goal by the third month of the project.

FIGURE 2

Run charts showing compliance with short-course antibiotic duration for community-acquired pneumonia and skin and soft tissue infections by randomization group.

FIGURE 2

Run charts showing compliance with short-course antibiotic duration for community-acquired pneumonia and skin and soft tissue infections by randomization group.

Close modal

In this site-randomized quality improvement trial intended to encourage recommended shorter courses of antibiotic treatment of CAP and SSTI in children, we found that education with performance feedback via e-mail and CDS delivered at the point of care were equally effective in changing clinician behavior. The combination of the 2 techniques, however, was substantially more effective than either approach alone. The effect sizes were large, with 23% to 26% absolute improvement in the groups assigned to education with performance feedback alone or to CDS alone, and with 42% absolute improvement in the group assigned to the combination of the 2 intervention strategies. If the magnitude of improvement observed in the latter group were extrapolated to a full year of encounters across our primary-care network, patients would be treated with ∼13 000 fewer cumulative days of antibiotics or ∼30 fewer days per 1000 patients per year.

Why did the combined intervention have greater efficacy than either approach alone? Presumably when deploying education with performance feedback alone, clinicians may simply miss the communication given the overwhelming amount of e-mails that clinicians currently receive,16  and/or they may be prone to forget the recommendation in the hustle of busy clinical practice. With CDS alone, clinicians may ignore alerts given “alert fatigue”; alternatively, perhaps the CDS as designed was not seen as sufficiently convincing or authoritative.17 19  We hypothesize that the combined intervention had a larger effect than either intervention alone because the initial education with performance feedback coming from a known credible source served a “priming” function and the CDS served as a reinforcing reminder, if needed, at the point of care.

When considering these types of interventions, a practical consideration is the effort and staff involved in each. In our experience, education with performance feedback requires substantially more time and effort (and therefore cost) compared with CDS, which requires initial set-up but minimal ongoing work. Further efforts may focus on whether the design of CDS could be optimized to produce the same magnitude of effect we observed in the combined group. Based on our experience with this project, in the future we would consider presenting the CDS messages as “signed” by a network clinical leader to make clear to front-line clinicians that the CDS alert came from a credible and trusted source. Alternatively, perhaps initial education plus CDS—without individual performance feedback, which we found to be the most labor-intensive part of our quality improvement intervention—could achieve an effect size similar to that observed in the combined intervention group with less overall effort.

A question remains as to how durable the effects observed in this project will be. Previous outpatient antibiotic stewardship efforts using education with performance feedback have shown that the effect wanes over time.20,21  Also, it is generally not feasible to continue education and feedback efforts indefinitely. CDS, on the other hand, can be sustained indefinitely and is designed specifically to target those who are not following the best practice recommendation, since clinicians who are already adhering to the particular best practice will not see the alert very often. One potential problem with CDS, however, is that when cases fall outside the scope of the recommendation (eg, in this case, sicker patients or those with complex underlying health conditions who may appropriately merit longer durations of antibiotic treatment), clinicians who do typically follow the best practice recommendation in uncomplicated cases may be bothered by unnecessary or inappropriate alerts, or perhaps even prompted to change to an inappropriate treatment given the clinical circumstances. Depending on the frequency of such cases, inappropriate CDS alerts may be experienced as a burden and cause frustration among clinicians who are generally following the best practice. Worse, CDS that does not adequately consider patient complexity or specific elements of a case could contribute to substandard care, although we think this possibility is unlikely in the current project given the substantial data on the safety of short courses of treatment of the conditions studied.4 7,10 12,22  We feel that further work regarding the optimal design of CDS for outpatient antimicrobial stewardship—both in terms of crafting an effective message and limiting inappropriate firing—would be a fruitful area for research.

We feel that the major strength of this project is its randomized design which minimizes the chance of unrecognized confounding that can occur in nonrandomized interventions, and we feel should be more widely used in quality improvement work when appropriate. A possible limitation of this work may be its generalizability. Although the PCCs involved in the project represent a diverse statewide sample, they are all part of a network affiliated with an academic children’s hospital and are accustomed to participating in quality improvement projects.

In this site-randomized quality improvement trial intended to encourage shorter durations of antibiotic treatment of CAP and SSTI in children, both education with performance feedback and CDS at the point of care were highly effective in changing clinician behavior; however, the combination of the two was significantly more effective than either strategy alone.

Dr Vernacchio conceptualized the study design, supervised the data analysis, and drafted the initial manuscript; Drs Hatoun, O’Donnell, and Herigon conceptualized the study design; Ms Patane analyzed the data; and all authors critically reviewed and revised the manuscript, approved the final manuscript as submitted, and agree to be accountable for all aspects of the work.

The trial is registered at www.clinicaltrials.gov (identifier NCT06087809).

FUNDING: No external funding.

CONFLICT OF INTEREST DISCLOSURES: The authors have no potential conflicts of interest to disclose.

CAP

community-acquired pneumonia

CDS

clinical decision support

ICD-10

International Classification of Diseases, Tenth Revision

PCC

primary care clinician

SSTI

skin and soft tissue infections

1
Centers for Disease Control and Prevention
.
Antibiotic Resistance Threats in the United States
.
US Department of Health and Human Services
;
2019
2
Gerber
JS
,
Jackson
MA
,
Tamma
PD
,
Zaoutis
TE
;
AAP Committee on Infectious Diseases and Pediatric Infectious Diseases Society
.
Policy statement: antibiotic stewardship in pediatrics
.
J Pediatric Infect Dis Soc
.
2021
;
10
(
5
):
641
649
3
Spellberg
B
,
Rice
LB
.
Duration of antibiotic therapy: shorter is better
.
Ann Intern Med
.
2019
;
171
(
3
):
210
211
4
Shapiro
DJ
,
Hall
M
,
Lipsett
SC
, et al
.
Short- versus prolonged-duration antibiotics for outpatient pneumonia in children
.
J Pediatr
.
2021
;
234
:
205
211.e1
5
Williams
DJ
,
Creech
CB
,
Walter
EB
, et al
;
The DMID 14-0079 Study Team
.
Short- vs standard-course outpatient antibiotic therapy for community-acquired pneumonia in children: the SCOUT-CAP randomized clinical trial
.
JAMA Pediatr
.
2022
;
176
(
3
):
253
261
6
Bielicki
JA
,
Stöhr
W
,
Barratt
S
, et al
;
PERUKI, GAPRUKI, and the CAP-IT Trial Group
.
Effect of amoxicillin dose and treatment duration on the need for antibiotic re-treatment in children with community-acquired pneumonia: the CAP-IT randomized clinical trial
.
JAMA
.
2021
;
326
(
17
):
1713
1724
7
Pernica
JM
,
Harman
S
,
Kam
AJ
, et al
.
Short-course antimicrobial therapy for pediatric community-acquired pneumonia: the SAFER randomized clinical trial
.
JAMA Pediatr
.
2021
;
175
(
5
):
475
482
8
Li
Q
,
Zhou
Q
,
Florez
ID
, et al
.
Short-course vs long-course antibiotic therapy for children with nonsevere community-acquired pneumonia: a systematic review and meta-analysis
.
JAMA Pediatr
.
2022
;
176
(
12
):
1199
1207
9
Kuitunen
I
,
Jääskeläinen
J
,
Korppi
M
,
Renko
M
.
Antibiotic treatment duration for community-acquired pneumonia in outpatient children in high-income countries-a systematic review and meta-analysis
.
Clin Infect Dis
.
2023
;
76
(
3
):
e1123
e1128
10
Cross
ELA
,
Jordan
H
,
Godfrey
R
, et al
.
Route and duration of antibiotic therapy in acute cellulitis: a systematic review and meta-analysis of the effectiveness and harms of antibiotic treatment
.
J Infect
.
2020
;
81
(
4
):
521
531
11
Stevens
DL
,
Bisno
AL
,
Chambers
HF
, et al
.
Practice guidelines for the diagnosis and management of skin and soft tissue infections: 2014 update by the Infectious Diseases Society of America
.
Clin Infect Dis
.
2014
;
59
(
2
):
147
159
12
Williams
OM
,
Brindle
R
.
Antibiotic route and duration of therapy for cellulitis: data extracted from a multi-center clinical trial
.
Int J Antimicrob Agents
.
2020
;
56
(
3
):
106076
13
Kimberlin
D
,
Barnett
E
,
Lynfield
R
,
Sawyer
M
. Systems-based treatment table. In:
Red Book: 2021–2024 Report of the Committee on Infectious Diseases
.
Committee on Infectious Diseases
;
2021
14
Westfall
JM
,
Mold
J
,
Fagnan
L
.
Practice-based research--“Blue Highways” on the NIH roadmap
.
JAMA
.
2007
;
297
(
4
):
403
406
15
Richards
AR
,
Linder
JA
.
Behavioral economics and ambulatory antibiotic stewardship: a narrative review
.
Clin Ther
.
2021
;
43
(
10
):
1654
1667
16
Paul
IM
,
Levi
BH
.
Metastasis of e-mail at an academic medical center
.
JAMA Pediatr
.
2014
;
168
(
3
):
290
291
17
McGreevey
JD
III
,
Mallozzi
CP
,
Perkins
RM
,
Shelov
E
,
Schreiber
R
.
Reducing alert burden in electronic health records: state of the art recommendations from four health systems
.
Appl Clin Inform
.
2020
;
11
(
1
):
1
12
18
Ancker
JS
,
Edwards
A
,
Nosal
S
,
Hauser
D
,
Mauer
E
,
Kaushal
R
;
with the HITEC Investigators
.
Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system
.
BMC Med Inform Decis Mak
.
2017
;
17
(
1
):
36
19
Sittig
DF
,
Wright
A
,
Osheroff
JA
, et al
.
Grand challenges in clinical decision support
.
J Biomed Inform
.
2008
;
41
(
2
):
387
392
20
Gerber
JS
,
Prasad
PA
,
Fiks
AG
, et al
.
Durability of benefits of an outpatient antimicrobial stewardship intervention after discontinuation of audit and feedback
.
JAMA
.
2014
;
312
(
23
):
2569
2570
21
Linder
JA
,
Meeker
D
,
Fox
CR
, et al
.
Effects of behavioral interventions on inappropriate antibiotic prescribing in primary care 12 months after stopping interventions
.
JAMA
.
2017
;
318
(
14
):
1391
1392
22
Barratt
S
,
Bielicki
JA
,
Dunn
D
, et al
.
Amoxicillin duration and dose for community-acquired pneumonia in children: the CAP-IT factorial non-inferiority RCT
.
Health Technol Assess
.
2021
;
25
(
60
):
1
72

Supplementary data