Acute otitis media (AOM) is the most-common indication for antibiotics in children. Delayed antibiotic prescribing for AOM can significantly reduce unnecessary antibiotic use and is recommended by the American Academy of Pediatrics for select children. We sought to improve delayed prescribing for AOM across 8 outpatient pediatric practices in Colorado.
Through a collaborative initiative with American Academy of Pediatrics and the Centers for Disease Control and Prevention, we implemented an economical 6-month antimicrobial stewardship intervention that included education, audit and feedback, online resources, and content expertise. Practices used The Model for Improvement and plan-do-study-act cycles to improve delayed antibiotic prescribing. Generalized estimating equations were used to generate relative risk ratios (RRRs) for outcomes at the intervention end and 3- and 6-months postintervention. Practice surveys were evaluated.
In total, 69 clinicians at 8 practice sites implemented 27 plan-do-study-act cycles. Practices varied by size (range: 6–37 providers), payer type, and geographic setting. The rate of delayed antibiotic prescribing increased from 2% at baseline to 21% at intervention end (RRR: 8.96; 95% confidence interval [CI]: 4.68–17.17). Five practices submitted postintervention data. The rate of delayed prescribing at 3 months and 6 months postintervention remained significantly higher than baseline (3 months postintervention, RRR: 8.46; 95% CI: 4.18–17.11; 6 months postintervention, RRR: 6.69; 95% CI: 3.53–12.65) and did not differ from intervention end (3 months postintervention, RRR: 1.12; 95% CI: 0.62–2.05; 6-months postintervention, RRR: 0.89; 95% CI: 0.53–1.49).
Baseline rate of delayed prescribing was low. A low-cost intervention resulted in a significant and sustained increase in delayed antibiotic prescribing across a diversity of settings.
Acute otitis media (AOM) is the most-common indication for antibiotic prescribing in children in the United States, resulting in 8.7 million antibiotic prescriptions annually.1,2 Although antibiotics can be beneficial for some children with AOM, an estimated 85% of infections self-resolve and do not benefit from antibiotic treatment.3,4 Nevertheless, >95% of children with AOM in North America receive antibiotics.5 The overuse of antibiotics has significant consequences, including adverse drug events,6 increased risk for Clostridioides difficile7 and antibiotic-resistant infections,6 and changes in the microbiome that may place children at risk for future autoimmune diseases.7,8
A delayed antibiotic prescription is a prescription given to the caregiver to fill in the event that the child’s symptoms worsen or fail to improve after 48 to 72 hours. Previous studies have revealed that delayed antibiotic prescribing can significantly reduce antibiotic exposure for AOM while maintaining patient and caregiver satisfaction when compared with prescribing an immediate antibiotic.9,10 Thus, the American Academy of Pediatrics (AAP) recommends the use of delayed antibiotic prescribing in select patients with AOM, including those 6 months and older with mild to moderate unilateral AOM.11 Despite these recommendations, little is known regarding in-practice rates of delayed antibiotic prescribing or the effectiveness of quality improvement (QI) interventions in increasing delayed antibiotic prescribing. We aimed to determine rates of delayed antibiotic prescribing for AOM across 8 diverse Colorado outpatient pediatric practices and evaluated whether an antibiotic stewardship intervention improved delayed antibiotic prescribing rates.
From January to June 2019, the AAP Chapter Quality Network (CQN), Colorado Department of Public Health and Environment, and Centers for Disease Control and Prevention (CDC) collaborated with AAP Colorado Chapter to recruit 8 Colorado outpatient pediatric practices for participation in an antibiotic stewardship QI initiative that addressed pharyngitis and AOM prescribing. All interested practices were allowed to participate. One component of this initiative aimed to improve delayed antibiotic prescribing rates for AOM (see Supplemental Information for all initiative measures).
During the 6-month intervention period, AAP’s CQN provided (1) 3 educational sessions (a full day in-person learning session and 2 90-minute virtual sessions covering the CDC Core Elements of Outpatient Antibiotic Stewardship and the Institute for Health Improvement Model of Improvement QI frameworks),12,13 (2) 4 monthly 60-minute webinars to review data and discuss plan-do-study-act (PDSA) cycles, (3) subject matter expert (SME) support (infectious disease, QI, and data analysis including run chart tracking), (4) access to an online community collaborative Web site, and (5) continuing medical education (CME) and maintenance of certification (MOC) credit (Table 1). Practice teams were responsible for (1) providing a QI team including, at a minimum, a physician practice champion, an office administrator, and a nurse or medical assistant; (2) attendance at educational sessions; (3) development and execution of monthly QI tests of change (PDSA cycles); (4) submission of monthly provider-level data on project measures; and (5) completion of surveys.
Physician practice champions were responsible for implementation and oversight of the intervention at their practice site. Any number of additional clinicians and staff could participate, including accessing the educational sessions and the collaborative Web site (Table 2). Practices were provided a key-driver diagram with guidance on QI strategies, including suggested interventions to promote antibiotic stewardship, and were allowed to modify or prioritize strategies on the basis of practice needs. Therefore, interventions were not standardized across the 8 practices, although practices shared PDSA cycles and successes and failures with each other (see Supplemental Information). Practices received education on effective communication to reduce antibiotic use by using the Dialogue Around Respiratory Treatment method.14 All practices were provided templates for delayed prescribing-focused patient education material (handouts and posters).
Although the project was grant funded, interventions were designed to be low-cost and easily replicable in low-resource environments. This included using online learning communities and virtual learning sessions that could be recorded for future use, providing a manual system for data entry to generate run charts, and providing American Board of Pediatrics Part 4 MOC credit as an incentive for providers.
After conclusion of the 6-month intervention period, monthly webinars and audit and feedback were discontinued. SME support and access to the online community collaborative Web site continued. Practices were invited to a 6-month sustainability session to assess maintenance of improvements over time. Participating practices were asked to submit data at 3 months and 6 months postintervention to assess sustainability of the intervention.
Use of delayed antibiotic prescriptions for patients with AOM, defined as the percent of antibiotic prescriptions written for patients ≥6 months old with an International Classification of Diseases, 10th Revision, Clinical Modification (ICD-10-CM) code for AOM that were written as a delayed prescription, was considered a process measure in the intervention (see Supplemental Table 6 for eligible ICD-10-CM codes). Documentation of parent education for delayed antibiotic prescriptions was also used as a process measure. Standard definitions and criteria for immediate versus delayed antibiotic prescriptions were provided to practices during the first learning session (Supplemental Information).
Data Tracking and Statistical Analysis
Delayed antibiotic prescription data were collected from each practice for a 2-month period immediately preceding the intervention (baseline) and for each 1-month period during the 6-month intervention. During the optional sustainability period, delayed antibiotic prescribing data were collected during the third and sixth month postintervention.
Clinicians obtained delayed antibiotic prescribing rates through electronic health record (EHR) query or manual chart review. Clinicians using EHR query submitted data on all eligible visits. Clinicians using manual chart review could, alternatively, submit data from 10 randomly selected visits when the number of eligible patient visits in the data collection month exceeded 10. Of the 621 clinician-level monthly rates submitted, 3 were based on a random sample.
Run charts were used to illustrate delayed antibiotic prescribing rates over time. Robust generalized estimating equations with a Poisson distribution and log link function were used to generate relative risk ratios (RRRs) comparing delayed antibiotic prescribing rates at each intervention month relative to baseline, as well as postintervention rates relative to baseline and intervention end. Analyses, conducted at the clinician level, accounted for within-subject dependency. The model treated number of delayed antibiotic prescriptions as the outcome, with an offset variable equal to the log number of eligible prescriptions. Statistical tests were conducted in SAS 9.4 (SAS Institute, Inc, Cary, NC). P values <.05 were considered statistically significant.
Surveys and Interviews
Surveys of practice champions were conducted at intervention months 2 and 4 and postintervention month 6. The primary purpose of the surveys was to evaluate implementation outcomes including project engagement and barriers to change and sustainability. Surveys assessed (1) satisfaction with program components, (2) each practice’s current state in testing or implementation of delayed antibiotic prescribing, and (3) confidence in completing future QI interventions. Telephone-based semistructured interviews (between SMEs and practice champions) conducted at 1 month postintervention assessed barriers to sustainability.15
The project was exempted from institutional review board (IRB) by the AAP IRB, which served as the IRB of record for the project.
Eight practice teams, including 69 clinicians, participated in the 6-month QI intervention and reported data on a monthly basis. Practice characteristics are shown in Table 2. The project included 6 community-based practices, 1 federally qualified health center (FQHC), and 1 hospital-affiliated practice. Practices were from a variety of geographic locations including suburban (3 practices), urban (3 practices), and mountain or rural locations (2 practices). Five practice teams, including 44 clinicians, participated in the 6-month sustainability period (conducted immediately postintervention) and reported data 3 months and 6 months postintervention.
Delayed Antibiotic Prescription Use
In total, 27 delayed antibiotic prescription-focused PDSA cycles were conducted by the 8-team collaborative (see Supplemental Information for specific PDSA cycles). All practices (8) developed patient education handouts and 3 practices developed educational posters during the intervention period. All practices (8) used EHR codes to denote dissemination of patient education on delayed prescribing. Delivery of delayed antibiotic prescriptions was tracked through use of EHR prescription fields (2 practices) or by abstraction from provider notes (6 practices). Clinic-level data were reviewed for all practices at monthly webinars; however, 6 practices also shared provider-level data with their practitioners.
All practices increased delayed antibiotic prescribing during the intervention period. A run chart of delayed antibiotic prescribing for patients with AOM aggregated across participating practices is presented in Fig 1 (see Supplemental Information for annotated practice-level run charts).
The percentage of antibiotic prescriptions for AOM written as a delayed prescription increased from 2% at baseline to 21% at intervention end (RRR: 8.96; 95% confidence interval [CI]: 4.68–17.17; Table 3). Among the practices reporting sustainability period data (n = 5), the rate of delayed antibiotic prescribing at 3 and 6 months postintervention (27% and 22%, respectively) remained significantly higher than baseline (3 months postproject, RRR: 8.46; 95% CI: 4.18–17.11; 6 months postproject, RRR: 6.69; 95% CI: 3.53–12.65; Table 2) and did not differ from intervention end (3 months postintervention, RRR: 1.12; 95% CI: 0.62–2.05; 6 months postintervention, RRR: 0.89; 95% CI: 0.53–1.49).
Surveys assessing each practice’s current state in testing or implementation of delayed antibiotic prescribing at intervention months 2 and 4 and postintervention month 6 reveal that 2 (25%) practices were testing and 6 (75%) were implementing delayed prescribing at intervention month 2. Similarly, 2 (25%) practices were testing and 6 (75%) were implementing delayed prescribing at intervention month 4. At postintervention month 6, no practices (0%) reported testing delayed prescribing, 7 practices (88%) reported implementing delayed prescribing, and 1 practice (13%) reported discontinuing delayed prescribing (citing evidence that 50 of 54 [93%] delayed prescriptions had been filled during the project period). For patients with mild-moderate AOM, this practice used observation with a patient follow-up plan instead of delayed prescribing.
Surveys conducted at intervention end showed that participants valued (on a 4-point scale: 1 = not valuable; 4 = very valuable) (1) participating in the learning network (3.26), (2) monthly learning webinars (3.11), (3) monthly data review (3.30), (4) access to monthly data reports (3.37), (5) access to clinical expertise (3.50), (6) access to project management database (3.11), and (7) ability to earn QI CME credit (3.11), part 2 MOC credit (3.53), and part 4 MOC credit (3.53) (see Supplemental Table for additional survey results).
Surveys of team leaders 6 months postintervention revealed that 100% of leaders reported improved confidence in the use of QI methods as a result of project participation. Improvements in confidence using PDSA cycles to implement practice changes was rated as 3.13 (5-point scale: 1 = not at all, 5 = very much).
A low-cost QI intervention that included education, availability of SMEs, access to an online community, and audit and feedback significantly increased delayed antibiotic prescribing for AOM, and results were sustained 6 months after the end of the intervention.
In this study, we found low baseline rates (2%) of delayed antibiotic prescription use for children ≥6 months with AOM. This is comparable to delayed prescribing rates reported from a Virginia cohort of pediatric practices in a previous similar CDC and AAP collaborative project (C. Norlin, K.E. Fleming-Dutra, J. Monti, et al, unpublished observations). Similarly, a recent report revealed that <5% of prescriptions for AOM within an FQHC system were delayed prescriptions.16 Although these results may not be generalizable to all practices, we suspect that despite AAP guidelines recommending use of delayed antibiotic prescriptions, overall delayed prescribing is low.
An economical intervention that included educational webinars, access to a system for providers to enter data for audit and feedback, access to an online community, and monthly data review resulted in a significant improvement in delayed antibiotic prescribing rates. Each practice developed a unique QI strategy to improve delayed prescribing rates, and the foci of practices’ PDSA testing varied during the intervention period (Supplemental Information). Many practices implemented PDSA cycles focused on education of providers and parents, whereas others focused on system changes, such as developing EHR tools. Practices varied substantially in the timing and approach used to implement specific interventions; thus, statistical analyses could not be conducted to compare the effectiveness of specific interventions. However, on the basis of practice-specific run charts of prescribing rates, no one methodology seemed most efficacious.
Although other stewardship interventions have shown prescribing rates trended toward baseline after removal of audit and feedback, the participating practices maintained delayed prescribing rates 6 months postintervention despite removal of monthly audit and feedback.17,18 On the basis of other stewardship interventions, audit and feedback is still likely an important component of intervention to improve and sustain delayed prescribing. However, as suggested with our data, it was less critical for this intervention. It is possible that by teaching practices QI methods, such as how to use PDSA cycles, they were able to better maintain prescribing patterns by using other improvement processes without needing continued formal audit and feedback reporting.
Unfortunately, there are considerable challenges with tracking delayed antibiotic prescribing and fill rates. This has severely limited our ability to understand delayed prescribing as a clinical and research community. We found that only 1 practice’s EHR allowed them to reliably differentiate between immediate and delayed prescriptions electronically.15 Some clinics found success in using smart text within the prescription or checking a “file to pharmacy” box; however, these depended on the provider manually changing the prescription and did not permit for electronic data abstraction. Consequently, manual chart review was typically required to generate audit and feedback reports, reducing the sustainability of the intervention.15 As a result, some practices have opted to provide audit and feedback less frequently or use a point-prevalence approach moving forward.19
Results of previous studies have indicated that roughly one-third of delayed prescriptions for AOM are filled.9 We were only able to evaluate fill rates in 1 practice; however, this practice found that 93% of prescriptions were filled (Results–Surveys). It is notable that this practice was an FQHC, which caused us to question whether fill rates may vary by geographic region, clinical setting, patient demographics, or patient symptoms at presentation. To reduce unnecessary prescribing, this practice ultimately opted to discontinue use of delayed antibiotic prescriptions and instead use observation with phone follow-up for patients ≥6 months with mild-moderate AOM. Observation has also been shown to be effective in reducing antibiotic use, although we were unable to assess its efficacy in this study.9,10
Overall, all project components were well-received and highly rated by participants. Encouragingly, 100% of practice champions were confident in their ability to develop and implement QI interventions in the future. Practices seemed to find both antibiotic prescribing education and QI education (on PDSA cycles) useful. Although this project was grant funded, most program components, including education sessions and webinars, virtual learning community, system for manually entering data to create run charts, and providing MOC, are low or no cost and have been incorporated into an accessible no- or low-cost bundled toolkit for practices to use in the future.20 This is likely an important method for antibiotic stewardship to be integrated into small to mid-sized outpatient practices, particularly in resource-limited settings.
This study has several strengths in implementing and assessing clinical practice changes, including a diverse group of community-based practices with experiences that are likely relatable and relevant to other pediatricians in community-based settings; expertise from the AAP, CDC, and state public health department (Colorado Department of Public Health and Environment); and the ability to collect quantitative and qualitative data. Because these data were collected within pediatric clinics in a single state, results may not be generalizable to other geographic regions or clinical settings. However, the data presented here are similar to what was found in a cohort of Virginia practices (C. Norlin, K.E. Fleming-Dutra, J. Monti, et al, unpublished observations). It is important to note that practices that participated in the sustainability phase of the study were more likely to be urban or suburban and it is unclear how generalizable the sustainability results are to rural practices.
In terms of impacting receipt of unnecessary antibiotics, we were limited in our ability to determine appropriateness of prescriptions, fill rates for most practices, and rates of observation. Importantly, we were not able to determine if the reported increase in delayed antibiotic prescribing was attributable to a change in patient management or increased documentation of delayed antibiotic use and education. Differences in how prescribing data were captured after practices implemented EHR changes could also have influenced perceived prescribing patterns. For the 2 practices that evaluated prescribing on the basis of actual antibiotic prescription text fields that were sent to the pharmacy, it is likely that data more accurately reflected true prescribing rates. During interviews conducted at the end of the intervention all practices, except 1 (the practice that changed to observation), subjectively reported they felt that delayed prescribing had increased as a result of the intervention. Future researchers should focus on evaluating delayed prescribing fill rates across diverse settings and populations. On the basis of previous studies and practice interviews, observation rates were likely low.15 In addition, because AOM is a seasonal illness, we recommend that future researchers evaluate sustainability for an additional year to assess for prescribing variations by season. Unfortunately, the emergence of coronavirus disease limited our ability to evaluate sustainability over a longer period of time. We learned that overcoming practice-level data acquisition limitations is critical to the success of interventions for delayed antibiotic prescribing. Importantly, although AAP provided use of a database, practices manually entered data. We were unable to validate the data through direct chart review. This process was laborious and, alternative methods for audit and feedback for small to mid-sized practices need to be considered. Finally, we did not have an adequate number of data collection intervals to create statistical process control charts with control limits and thus relied on run charts and statistical testing for data interpretation.
Despite national recommendations to use delayed antibiotic prescribing for AOM, we know little about how often they are used by providers and filled by patients, particularly in diverse populations. EHR challenges with identifying delayed prescriptions and limited data sets with combined EHR and prescription fill data have restricted our ability to evaluate delayed prescribing for AOM, but these studies are important to inform future AOM and antibiotic stewardship recommendations. We found success in improving and sustaining delayed prescribing for AOM using a bundled antibiotic stewardship approach. This approach is likely replicable at other practices, including those in resource-limited settings. At the conclusion of the project, the AAP CQN developed a free, publicly available Change Package to assist practices with improving antibiotic prescribing for children in outpatient settings.20 Although additional resources will be needed to assist practices in improving outpatient antibiotic prescribing, the Change Package offers detailed, practical advice for improving prescribing for children.
We recognize Brighton Pediatrics, Children’s Medical Clinic, Every Child Pediatrics, The Ft. Collins Youth Clinic, Lowry Pediatrics, Peak Vista Pediatric Health Center, Pediatric Partners of Valley View, and Western Colorado Pediatric Associates for their tremendous engagement and work during this intervention to improve care for children. We also thank Jennifer Powell and Suzanne Emmer for their leadership and contributions to this project.
Dr Frost conceptualized and designed the study, provided subject matter support, analyzed and interpreted data, drafted the initial manuscript, and critically reviewed and revised the manuscript; Dr Monti analyzed and interpreted data, drafted the initial manuscript, and critically reviewed and revised the manuscript; Dr Andersen coordinated the data collection, analyzed and interpreted data, drafted the initial manuscript, and critically reviewed and revised the manuscript; Dr Norlin conceptualized and designed the study, provided subject matter support, analyzed and interpreted data, and reviewed and critically revised the manuscript; Ms Bizune provided subject matter support and reviewed and critically revised the manuscript; Dr Fleming-Dutra provided subject matter support and reviewed and critically revised the manuscript; Dr Czaja conceptualized and designed the study, provided subject matter support, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
The findings and conclusions in this report are those of the authors and do not necessarily represent the official position of the Centers for Disease Control and Prevention, the American Academy of Pediatrics, or the Colorado Department of Public Health and Environment.
FUNDING: Supported by the Centers for Disease Control and Prevention (CDC) through a Cooperative Agreement (6NU38OT000292-01-01) with the Chapter Quality Network, a program of the American Academy of Pediatrics. H.F. received salary support from the Eunice Kennedy Shriver National Institute of Child Health & Human Development of the National Institutes of Health under Award Number K23HD099925. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funded by the National Institutes of Health (NIH).
American Academy of Pediatrics
acute otitis media
Centers for Disease Control and Prevention
continuing medical education
Chapter Quality Network
federally qualified health clinic
International Classification of Diseases, 10th Revision, Clinical Modification
institutional review board
maintenance of certification
relative risk ratio
subject matter expert
POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.