A Bundled, Practice-Based Intervention to Increase HPV Vaccination

Annually, approximately 14 million Americans are infected with human papillomavirus (HPV), leading to 36 500 new cancers, 3800 cervical cancer deaths, and $9.6 billion in medical costs (2020 US dollars).^1–3 The Advisory Committee on Immunization Practices (ACIP) and American Academy of Pediatrics (AAP) recommend beginning HPV vaccination at age 11 to 12 years (or between ages 9 and 12 years).^4,5 Despite HPV vaccines’ high efficacy in preventing HPV-related cancers and Healthy People 2030 goals of more than 80% coverage,⁶ in 2021, before this study was conducted, only 77% of US adolescents ages 13 to 17 years had initiated and 62% had completed the series.⁷ 

Three groups of barriers limit coverage: parent/patient (eg, vaccine hesitancy or infrequent well-child care [WCC] visits among adolescents),^8–10 practice (eg, suboptimal vaccine communication by clinicians and staff or missed vaccination opportunities at office visits),^8,11,12 and health system (eg, lack of feedback to clinicians about HPV vaccination rates).^11,13 The Guide to Community Preventive Services and experts recommend bundled interventions to address barriers simultaneously.^4,14,15 Using the Immunization Barriers Model¹⁶ that categorizes immunization barriers into 3 groups (those facing patients, health care providers, and systems) as a foundation, we pre-planned a series of individual interventions that addressed each group of barriers and then a bundled intervention to address multiple barriers. This study evaluated the bundled intervention.

One evidence-based intervention addressing patient vaccine confidence or hesitancy provided communication training to clinicians about HPV vaccination.^15,17 Several randomized controlled trials (RCTs) of clinician communication training have noted improvements in HPV vaccination.^18–20 In the first of 3 sequential trials, we performed a clustered RCT with 48 practices (24 intervention and 24 control) from the AAP Pediatric Research in Office Settings (PROS) network.¹⁸ Online clinician communication training (plus weekly text messaging) reduced (improved) missed opportunity (MO) rates for HPV vaccine initiation during WCC visits by 6.8 percentage points above usual care but had little effect on MOs at acute/chronic visits.¹⁸ 

In a second RCT, addressing system-related barriers of not providing vaccination data to providers, we added performance feedback, another recommended intervention,^14,15 to intervention practices 7 months following the communication intervention.²¹ During acute/chronic visits for subsequent (ie, second or third) doses, performance feedback reduced MOs for HPV vaccination by 3.4 percentage points compared with the usual care but had no benefit during WCC visits.²¹ 

A third recommended intervention, addressing provider/practice barriers of MOs for vaccination, was to reduce MOs for vaccination by using electronic health record (EHR) or nurse/medical assistant (MA) prompts for clinicians.^14,15,22 In a third RCT, training on using clinician prompts (EHR or nurse/MA-delivered prompts) delivered to intervention group practices improved MO rates for the initial HPV vaccination dose at WCC visits by 4.5 percentage points.^23,24 

The fourth pre-planned intervention, the current study, provided a combination of the above 3 interventions as a bundled intervention. Clinicians and health systems might prefer multiple interventions to simultaneously address multiple barriers. However, although bundled interventions might theoretically have a larger impact than individual interventions, they may also be challenging to implement with fidelity.

For the current study, conducted through the AAP PROS network, we estimated the effect of a bundled intervention—communication training plus performance feedback plus clinician prompts—delivered over a 6-month period. We hypothesized that this bundled intervention would reduce MOs during WCC and acute/chronic visits. We took advantage of a unique opportunity to provide the bundled intervention to the previous usual care control group practices from the abovementioned RCTs, allowing us to evaluate the impact of a bundled intervention with the context of having evaluated the individual components in practices from the same sampling frame. We provided the bundled intervention to the prior control group practices and evaluated the bundled intervention with a pre-post study design (primary analysis). We also compared MO rates before and during the bundled intervention with MO rates in a retrospectively selected contemporaneous comparison group of 48 practices to assess for the potential impact of secular trends.

We conducted a single-arm pre-post intervention study of practices previously in the usual care control group of the multiphase RCT (Figure 1A).^18,21,23 The baseline period comprised 12 months (January 12, 2021, to January 11, 2022) before the initiation of the bundled intervention. The 6-month bundled intervention period (February 23, 2022, to August 9, 2022) followed a pre-planned 1-month ramp-up period (not analyzed) when participating clinicians were sent online training modules for the bundled intervention.

We compared outcomes between the 24 practices in the single-arm bundled intervention study with 48 practices selected retrospectively from the AAP Comparative Effective Research through Collaborative Electronic Reporting Consortium (CER²; details to follow²⁵; Figure 1B).

For our prior sequential RCTs,^18,21,.23 we had approached 113 primary care practices from the PROS network and recruited practices using flyers at PROS meetings and via EHR vendors; 48 practices agreed to participate (Figure 1A). All independent practices used 2 national EHR vendors (Physician’s Computer Company or Office Practicum); practices from 2 health systems used 3 EHRs (Cerner, Epic, or eClinicalWorks). The practices for this pre-post study comprised the 24 prior usual care control group practices (hereafter “bundled intervention practices”).

Clinicians from the 24 bundled intervention practices provided written informed consent (hereafter “participating clinicians”) to implement the interventions; they could receive 25 Part IV Maintenance of Certification (MOC) credits. The preplanned primary analysis comprised all visits for all age-eligible patients for all clinicians regardless of clinician participation (“intent-to-treat”); secondary analyses involved visits for participating clinicians.

We did not seek informed assent/consent from adolescents/parents per protocols approved by the institutional review board (IRB) because adolescents are offered universally recommended vaccination during office visits. All adolescents aged 11 to 17 years with visits to their practice were included.

We retrospectively identified 48 additional practices beyond those in the prior RCT, from the AAP PROS CER² database, to assess secular trends during the 4.5 years before the bundled intervention and 6-month bundled intervention time period (Figure 1B). The CER² database is an AAP-owned EHR data repository. It is comprised of EHR data, limited to items constituting Health Insurance Portability and Accountability Act limited datasets, submitted by participating pediatric primary care practices.²⁵ The data were from 48 practices in the CER² repository that had not been part of the RCT, had 5 years of lookback data, and used 1 of the 5 EHRs used by bundled intervention practices. The study team had no direct contact with these practices and identified practices before assessing HPV vaccination. We assessed all visits by adolescents aged 11 to 17 years for all clinicians at these 48 practices for the 5-year period.

The bundled intervention included 3 interventions, each identical to the single interventions in the 3 sequential RCTs.^18,21,23 Interactive online modules (Table 1) trained clinicians how to implement each intervention.

We sent clinicians three 20- to 30-minute online modules developed using Articulate Rise software (https://articulate.com/360/rise). Clinicians could view the modules by computer, tablet, or smartphone. Module 1 described the HPV disease burden, the vaccination’s importance, an effective presumptive recommendation, and how to answer common parent questions. Module 2 focused on involving the entire office in HPV vaccine communication and answering additional questions. Module 3 addressed vaccine hesitancy using motivational interviewing techniques. All modules included resources for clinicians and patients. Clinicians were asked to share module 1 with relevant office staff.

We sent clinicians a 15-minute training module on interpreting 3 upcoming performance feedback reports (using the R ‘shiny’ package, version 1.0.5; The R Project for Statistical Computing). We sent feedback reports via text or email (following clinician preference) during the 1-month ramp-up period and 2 and 4 months into the 6-month intervention period. Reports showed captured HPV vaccination opportunities: the proportion of vaccine-eligible visits involving HPV vaccination (ie, the converse of MO rates). The reported rates were stratified by initial and subsequent HPV vaccination, visit type (WCC or acute/chronic), and patient sex and age (11 to 12 years or 13 to 17 years). Feedback reports showed the results for each clinician, their practice, and all bundled intervention practices.

We sent two 15-minute training modules about clinician prompts for the HPV vaccine. The first module described the importance of and suggested workflows for using EHR and nurse/MA clinician prompts. The second module asked clinicians to integrate nurse/MA prompts into workflows, gave scenarios, and emphasized the value of prompts for subsequent HPV vaccine doses (visits when hesitancy is uncommon).

We sent participating clinicians weekly “quick tips” throughout the 6 months, brief text messages (if desired, emails) reinforcing messages from the modules.

The primary outcome was the MO rate for HPV vaccination. An MO was defined as a visit when an HPV vaccine was due per EHR vaccination data but not given. Reasons for MOs were not available from EHRs. We followed ACIP guidelines for eligible HPV vaccination (https://www.cdc.gov/vaccines/vpd/hpv/hcp/recommendations.html) and used evaluation and management Current Procedural Terminology codes to categorize visits into WCC or acute/chronic visits (excluding nurse-only visits). The denominator comprised all office visits (irrespective of clinician participation) by HPV vaccine–eligible adolescents aged 11 to 17 years. The numerator comprised all HPV vaccine–eligible visits during which a vaccination was not given (ie, MOs). We assessed MOs by initial and subsequent HPV vaccination and by visit type (all, WCC, or acute/chronic). For secondary analyses of the pre-post study, we assessed MOs among visits to participating clinicians.

We assessed adolescent sex and baseline HPV vaccination because both have been related to the impact of vaccination interventions. Using available data in the EHRs, we assessed race and ethnicity, social constructs, to place the bundled intervention practices into national context regarding demographic and social features. Data on race and ethnicity were often missing, as occurs with EHR analyses. EHR data were collected as part of routine clinical care.

We descriptively report the MO rate in the baseline and bundle periods using frequencies and percentages overall and separately by visit type and dose. To estimate the effect of the intervention on the MO rate, we used visit-level marginal logistic regression models fit by survey methodology for multilevel data, with practice as the primary sampling unit to reflect the treatment assignment level and account for within-practice correlation of observations. This approach, equivalent to generalized estimating equations with independence working correlation, accounts for variation in outcome rates across practices, a source of variation our design anticipated.

We hypothesized that the MO rate for HPV vaccination would be lower (1) in the bundle period than in the baseline period for total and WCC visits because an improvement was previously observed with the communications skills and prompts interventions^18,23 and (2) at acute/chronic care visits because an improvement was observed during the performance feedback intervention.²¹ We stratified the analysis by visit type: total visits, WCC visits, and acute or chronic visits.

We conducted our analyses among eligible visits to all clinicians (primary outcome) and separately among eligible visits to participating clinicians. All analyses excluded nurse-only visits. We examined intervention effects overall and stratified by visit type and dose. Models included binary MO as the outcome and period (pre-intervention or intervention) as the predictor. Models in the overall sample adjusted for visit type (WCC or acute/chronic), dose (initial or subsequent), and their 2-way interaction. We marginalized over model coefficient estimates to obtain risk differences and 95% CIs for MOs in the bundle compared with the baseline period.²⁶ 

Confidence bounds reflect the post hoc statistical power and precision of the reported estimates.²⁷ We used R (version 4.3.3) for data management and Stata (version 18.0) for analysis.

Estimates of MO rates and their differences comparing bundle practices with the 48 comparison practices over time are derived from grouped logistic regression with main effects for bundle and period and an interaction term, followed by marginal standardization. Confidence bounds reflect the clustering of observations within practices over time.

This study was approved by IRBs at the University of California, Los Angeles, and the AAP. The Children’s Hospital of Philadelphia’s IRB determined the study to be exempt.

Of the 24 practices, 22 completed the intervention: a solo practitioner departed and 1 practice withdrew because of COVID-19 pandemic issues. We analyzed the EHR data for all 24 practices because the EHR data were available for the 2 practices that withdrew. Among the 183 clinicians, 112 participating clinicians (61.2%) conducted 66.2% of all visits during the 12-month baseline period. Module completion rates ranged from 83% to 93% for 6 modules (Table 1). Table 2 shows baseline characteristics of practices, providers, adolescents, and office visits at which adolescents were eligible for HPV vaccination.

Table 3 shows the percentage of office visits with a missed opportunity for HPV vaccination; for initial and subsequent HPV vaccinations, stratified by visit type (all, WCC, acute/chronic); and the results of the secondary analyses of MOs by clinician participation (all or participating). HPV vaccination MOs occurred at approximately half of vaccine-eligible WCC visits and more than 90% of other vaccine-eligible visits. During WCC visits, MO rates for HPV vaccination were lower (improved) during the intervention vs baseline for both initial HPV vaccination (−4.8 percentage points; 95% CI, −7.2% to −2.4%) and subsequent HPV vaccination (−2.2 percentage points; 95% CI, −4.4% to −0.0%). Similar findings were noted at visits to participating and nonparticipating clinicians. During acute/chronic care visits, MO rates for HPV vaccination changed little between the baseline and intervention periods; confidence bounds ruled out improvements as small as 0.4% points for initial visits and 0.8% for subsequent visits (Table 3).

Because the above analysis compared a 12-month baseline with a 6-month intervention period, we also performed 2 sensitivity analyses (Supplemental Table 2) for initial HPV vaccine dose at WCC visits for all clinicians: (1) a calendar-matched analysis of the relevant 6-months before the bundle with the months during the bundle (eg, March 2021 matched with March 2022) and (2) an analysis of the 6-month period just before the bundle vs the 6-month bundle period. Both sensitivity analyses had similar findings to those in Table 3, confirming our main analysis.

We examined changes in MOs for initial HPV vaccination at WCC visits by practice (Figure 2): 11 practices improved by at least 5 percentage points, 4 practices improved 2.0 to 4.9 percentage points; 5 practices improved 0.1 to 1.9 percentage points, and 4 practices worsened (increased MOs) 0.1 to 1.9 percentage points. Baseline MO rates varied greatly across practices, but improvement occurred for practices with a large range of baseline MO rates.

On multivariable analysis, across all visits and all children aged 11 to 17 years, the intervention was associated with a 1.8 percentage point absolute risk reduction in MO rates (95% CI, −2.9 to −0.7), adjusted for dose (first or subsequent), visit type (WCC or other), and their 2-way interaction, for all providers and 2.0 percentage points (95% CI, −3.3 to −0.6), with the same adjustments for participating providers.

Characteristics of adolescents, HPV vaccination rates, and visits at which adolescents were eligible for HPV vaccine were all similar between bundle and comparison group practices (Supplemental Table 1) except for adolescent race (the bundled group had more missing values).

Figure 3 shows trends over 5 years in MO rates for HPV vaccination during WCC visits in the 24 bundled intervention and 48 comparison practices. MO rates were quite similar throughout the 4.5-year pre-bundle period, appeared somewhat higher (worse) for intervention practices than comparison practices for the 12 months just before the intervention, and appeared lower (better) in intervention vs comparison group practices during the intervention. The change from prebundle to bundle periods was associated with a 6.6 percentage point–lower MO rate for vaccine initiation at WCC visits among the bundled intervention practices than comparison practices (−6.6%; 95% CI, −9.3% to −3.8%) (Figure 3 and Table 4). MOs for subsequent doses at WCC visits also overlapped in the 4.5-year prebundle period for bundle intervention and comparison practices, but the confidence bounds of the estimates rule out all but small relative improvement (1.0%; 95% CI, −3.6%% to 1.7%) (Table 4).

This pre-post study found that a bundled intervention—communication training for clinicians, performance feedback for clinicians, and training on EHR and staff prompts—appeared to improve HPV vaccination at WCC visits for both initial and subsequent vaccine doses. The magnitude of the effect was similar to the effects of 2 individual interventions (communication training¹⁸ and prompts²³) in our prior RCTs. MO rates improved in 20 of 24 practices. Practice characteristics were not associated with pre-post changes in MO rates, demonstrating a broad effect. In our secondary study, although MO rates for HPV vaccination at WCC were similar in level and trajectory between bundled intervention and contemporaneous comparison practices during the 4.5 years before the intervention, MO rates were lower (better) during the intervention among bundled intervention practices vs comparison practices, suggesting that the improvements among intervention practices were not due to secular trends. Overall, a bundled intervention for HPV vaccination appears feasible and appears to improve HPV vaccination at WCC visits for initial and subsequent vaccinations.

Other studies have reported on HPV vaccination interventions. The most promising involve clinician communication training,^17,19,20 which may improve initial HPV vaccination by addressing parental vaccine confidence; this discussion usually occurs during WCC visits. The current study adds to the literature. First, this bundled intervention improved initial HPV vaccination rates at WCC visits by about the same amount as our prior study using communication training alone¹⁸ and as reported by Dempsey et al¹⁹; both studies occurred before the pandemic and rise in vaccine hesitancy. The current evidence shows communication training to be the most effective intervention studied for adolescent HPV vaccination, so it should be a foundational element of any multicomponent intervention. Second, this bundled intervention suggested improvements in subsequent HPV vaccination at WCC visits, which was not noted in our prior RCTs of individual interventions (ie, communication training,¹⁸ performance feedback,²¹ or clinician prompts²³). This suggests a possible added benefit of our multicomponent intervention.

We did not find improvements in MO rates during acute/chronic visits, despite having observed improvements at these visits in our prior RCT assessing performance feedback.²¹ Numerous studies have documented challenges in vaccinating at acute/chronic visits,^28–31 and we have noted that vaccinating at acute/chronic visits is related to longer spacing between subsequent WCC visits.³² Our study was implemented during an unusually busy respiratory season in 2022 and the COVID-19 Omicron variant,³³ making it challenging to implement changes at acute care visits, even for subsequent vaccinations for which parental vaccine hesitancy is uncommon. Given these real-world challenges, we feel HPV vaccination efforts should focus primarily on WCC visits, plus perhaps chronic care visits when more time may be available, and nursing visits for influenza vaccination, which focus on vaccination.³⁴ 

Because the magnitude of the effect in this bundled intervention is similar to the effect in the communication-only manuscript, a bundled intervention may not be more effective than a communication intervention alone. Health systems and practices seeking to increase HPV vaccination could be justified in implementing the communication intervention alone. However, it is important to consider the context: the communication training intervention was conducted before the pandemic, and the bundled intervention was conducted after the pandemic when vaccine hesitancy became greater.

We noted a 2.2% reduction in MOs for subsequent HPV vaccinations at WCC visits for all clinicians (borderline statistically significant) and a 2.1% reduction for participating clinicians (not statistically significant). The smaller sample size among participating clinicians resulted in wider CIs. One explanation is that we asked participating clinicians to discuss all 3 interventions at practice meetings, to distribute 1 of the communication modules (answering common questions) to all staff, to share feedback reports displaying MOs at the level of the practice, and apply prompts to the entire practice. Thus, the possible impact on subsequent vaccinations, albeit small, may have spread from participating clinicians to the entire practices. We feel this strategy of sharing findings and discussing at practice meetings may be helpful to those planning practice-improvement interventions.

One study strength is that we tested the combination of 3 interventions that were the same as the individual interventions in 3 prior RCTs in practices from the same sampling frame. This allowed us to assess the added value of bundled interventions. Additional study strengths are the large number of practices across the United States, inclusion of a comparison group of practices to account for secular trends, EHR analyses to document HPV vaccination MOs, and assessment of practice factors potentially related to HPV vaccination.

One study limitation is the pre-post design (ie, despite the addition of comparison practices well matched to those in the intervention, unmeasured secular trends or time-varying factors may have accounted for some pre-post findings). Second, we could not assess fidelity to the bundled intervention. We documented high module completion, feedback report receipt, and self-reported prompt use, but we could not assess actual communication with parents or clinician use of feedback reports or prompts. Third, we could not compare the effectiveness of this bundled intervention with the findings of the prior single-intervention trials because of differences in study designs. Fourth (external validity), PROS network practices voluntarily participated in research (although the CER² practices did not need to give consent).

We conclude that a bundled intervention of communication training, performance feedback on MOs, and clinician prompts appeared to improve HPV vaccination at WCC visits.

ACIP

Advisory Committee on Immunization Practices

CER²

Comparative Effective Research through Collaborative Electronic Reporting Consortium

EHR

electronic health record

HPV

human papillomavirus

IRB

institutional review board

MA

medical assistant

MO

missed opportunity

PROS

Pediatric Research in Office Settings

RCT

randomized controlled trial

WCC

well-child care.