Video Abstract

Video Abstract

Close modal
OBJECTIVES:

To evaluate the effectiveness of a stepped-wedge randomized trial of Development of Systems and Education for Human Papillomavirus Vaccination (DOSE HPV), a multilevel intervention.

METHODS:

DOSE HPV is a 7-session program that includes interprofessional provider education, communication training, data feedback, and tailored systems change. Five primary care pediatric and/or family medicine practices completed interventions between 2016 and 2018; all chose to initiate vaccination at ages 9 to 10. We compared vaccination rates in the preintervention, intervention, and postintervention periods among 9- to 17-year-olds using random-effects generalized linear regression models appropriate for stepped-wedge design, accounting for calendar time and clustering of patients by providers and clinic. Outcomes included (1) the likelihood that eligible patients would receive vaccination during clinic visits; (2) the likelihood that adolescents would complete the series by age 13; and (3) the cumulative effect on population-level vaccine initiation and completion rates. Postintervention periods ranged from 6 to 18 months.

RESULTS:

In the intervention and postintervention periods, the adjusted likelihood of vaccination at an eligible visit increased by >10 percentage points for ages 9 to 10 and 11 to 12, and completion of the vaccine series by age 13 increased by 4 percentage points (P < .001 for all comparisons). Population-level vaccine initiation coverage increased from 75% (preintervention) to 84% (intervention) to 90% (postintervention), and completion increased from 60% (preintervention) to 63% (intervention) to 69% (postintervention).

CONCLUSIONS:

Multilevel interventions that include provider education, data feedback, tailored systems changes, and early initiation of the human papillomavirus vaccine series may improve vaccine series initiation and completion beyond the conclusion of the intervention period.

What’s Known on This Subject:

Multilevel interventions targeting providers appear to increase human papillomavirus vaccine series initiation rates. Limited research exists on the effects of multilevel interventions on series completion, the sustainability of increases, and the impact of lowering the routine age of vaccination.

What This Study Adds:

Multilevel interventions targeting the interprofessional team that include provider education, data feedback, tailored systems changes, and early initiation of the human papillomavirus vaccine series may be successful in improving vaccine series initiation and completion beyond the conclusion of the intervention period.

Data from the United States and other countries demonstrate that human papillomavirus (HPV) vaccination profoundly reduces vaccine-type oncogenic HPV infections, genital warts, and precancers of the cervix, with emerging data revealing reduction in HPV-related cancers.13  The Centers for Disease Control and Prevention (CDC) recommend routine HPV vaccine initiation between the ages of 11 and 12 years. Vaccination can begin at age 9 and is also recommended for all adolescents aged 13 to 26 years who were not vaccinated on time.46  Initiating vaccination at an earlier age is preferable to vaccinating in older adolescence because vaccination is most effective before sexual debut,7  and no waning of protection is observed.8  HPV vaccination rates remain below national goals of 80% of eligible adolescents completing the vaccine series,9  however, because 68.1% of adolescents aged 13 to 17 years initiated and 51.1% completed the vaccine series in 2018.9,10  Only 39.9% of adolescents completed the series on time, defined as before their 13th birthday.11 

Research has identified barriers to HPV vaccination,12  including lack of effective provider recommendation1316  and lack of reminder and recall systems, leading to missed opportunities for both series initiation and completion.17,18  Multilevel interventions show promise for raising HPV vaccination rates by using techniques including provider education and communication training, data assessment and feedback, and quality improvement methods.1926  Increases in vaccination rates varied by study type and setting, and postintervention follow-up did not demonstrate sustainability after intervention completion.22 

In 2011 and 2012, we piloted Development of Systems and Education for Human Papillomavirus Vaccination (DOSE HPV), a multilevel performance improvement continuing medical education (PI CME) intervention that includes provider education and communication training, data assessment and feedback, and tailored systems interventions to improve HPV vaccination rates among socioeconomically vulnerable populations.23,27  PI CME is a learning format consisting of longitudinal practice improvement, for which physicians may earn continuing medical education credit. Participation requirements for PI CME align with maintenance of certification part IV requirements for board-certified pediatric, internal medicine, and family medicine physicians. For nurses, nursing professional development (ie, contact hours) is provided for participation in live meetings. The pilot study involved 2 pediatric practices, used a pre- and postevaluation design, and was conducted when a 3-dose schedule was the national recommendation. To better estimate intervention effectiveness and provide short-term sustainability data, we performed a stepped-wedge cluster randomized trial of the DOSE HPV intervention at 5 additional sites. We chose the stepped-wedge design as the best study design to separate secular trends from intervention effects as well as incentivize participation by clinical sites because all sites received the intervention. We report on the intervention’s effects on HPV vaccine series initiation and completion up through 18 months after intervention completion.

Interventions took place in Massachusetts in the Boston metropolitan area and included 5 clinical sites serving primarily low-income and minority patients. One site was a safety-net hospital; the others were independent federally qualified health centers. Sites were selected before the intervention period on the basis of 2014 site-specific data indicating HPV vaccination rates below national targets.9  The DOSE HPV intervention is a provider-focused program that included 7 sessions, 1 hour each, performed at approximately monthly intervals over 6 to 8 months between 2016 and 2018 (Fig 1). Sessions were held during regular staff meetings to maximize attendance, and food was served. Intervention sessions were interprofessional, including clinic leadership, primary care providers, and nursing staff. The intervention educates providers and activates them to make tailored systems changes within their health care settings. The intervention included 3 core components: interprofessional education and communication training for health care providers, data assessment and feedback, and creation of an action plan that included both provider- and systems-level changes. All sessions were facilitated by the intervention team, and sessions at which site-specific action plans were developed were led by clinical champions and participating providers.

FIGURE 1

Intervention time line and content with adolescent vaccination by calendar month. In this figure, we describe the time line of each intervention, as well as the percentage of adolescents with visits per calendar month who initiate the HPV vaccine series before or at the index visit. The granular representation of data details the individual action plan components at each site, and also notes changes in vaccination rates as they relate to the timing of the DOSE HPV intervention as well as release of the national 2-dose recommendations. EHR, electronic health record; PCP, primary care provider. a National recommendations for 2-dose schedule were published on December 16, 2016, in the Morbidity and Mortality Weekly Report.b Denotes action plan components that were discontinued. Recalls for patients overdue for vaccine were discontinued because of high staff effort and low patient response. Standing orders were discontinued because of staff turnover. EHR prompt was discontinued at site 5 because of PCP workflow issues. All other action plan components were continued by clinical sites in the postintervention period. c Sites 4 and 5 transitioned to a different electronic medical record system during the preintervention period, leading to a 1-month delay in preintervention data collection in site 4 and an 8-month delay in site 5.

FIGURE 1

Intervention time line and content with adolescent vaccination by calendar month. In this figure, we describe the time line of each intervention, as well as the percentage of adolescents with visits per calendar month who initiate the HPV vaccine series before or at the index visit. The granular representation of data details the individual action plan components at each site, and also notes changes in vaccination rates as they relate to the timing of the DOSE HPV intervention as well as release of the national 2-dose recommendations. EHR, electronic health record; PCP, primary care provider. a National recommendations for 2-dose schedule were published on December 16, 2016, in the Morbidity and Mortality Weekly Report.b Denotes action plan components that were discontinued. Recalls for patients overdue for vaccine were discontinued because of high staff effort and low patient response. Standing orders were discontinued because of staff turnover. EHR prompt was discontinued at site 5 because of PCP workflow issues. All other action plan components were continued by clinical sites in the postintervention period. c Sites 4 and 5 transitioned to a different electronic medical record system during the preintervention period, leading to a 1-month delay in preintervention data collection in site 4 and an 8-month delay in site 5.

Close modal

In session 1, the study team provided health center–specific preintervention data on HPV vaccine initiation and completion rates. For session 2, a physician expert provided education on HPV-associated cancers, effectiveness of vaccination, and evidence-based interventions. Sessions 3 and 4 were focused on motivational interviewing training.28  During session 5, the study team used quality improvement methods to collaboratively identify barriers to HPV vaccination and create site-developed action plans, including systems- and provider-level changes to improve performance. Sessions 6 and 7 were facilitated discussions reviewing progress on implementing action plans, defining near-term action steps, and specifying task ownership to address obstacles and accelerate progress. Although the 5 sites developed action plans specific to their unique practice environments, all decided to routinely recommend HPV vaccination before age 11 (Fig 1). The data review motivated this change because clinic-level vaccination rates revealed that, although most patients received vaccination at well-child visits, a substantial proportion of their primary care populations remained unvaccinated primarily because of attrition in well-child care among older adolescents.

The trial used a stepped-wedge cluster randomized trial design: the study team randomized clinical sites (clusters) by generated number, and interventions occurred in order of their random number, staggered by ∼6 months during the 3 year study period.29  Interventions began on April 7, 2016, and ended on March 15, 2018. Electronic medical record data were used to evaluate the intervention’s effect on HPV vaccination rates. The study population included adolescents aged 9 to 17 years with an assigned primary care provider and ≥1 visit to the primary care site at any time during the 3-year study period. The study period was divided into preintervention, intervention, and postintervention periods for each site. The intention for the stepped-wedge design was for all preintervention periods to begin on the same date (October 1, 2015), 6 months before the intervention start date at clinical site 1. However, because of changeover of electronic medical record systems, preintervention periods began later at sites 4 and 5 (November 1, 2015, and June 1, 2016, respectively; Fig 1). The intervention periods began on the date of the first session and ended on the date of the last session at each site. The postintervention periods began 1 day after the last intervention session at each site and ended on September 30, 2018, for all sites. Differential pre- and postintervention periods in the stepped-wedge design serve to control for secular trends and allow comparisons within sites over time as well as between sites.29 

All analyses accounted for the national change from a 3-dose to a 2-dose schedule; dose eligibility mirrored recommendations on the date of the clinic visit. Before December 16, 2016, all adolescents required 3 doses to complete the series; visit dates ≥2 months after the first dose and ≥4 months after the second dose were considered eligible. After December 2016, the 3-dose schedule was applied only to adolescents who initiated the series on or after their 15th birthday. Adolescents who initiated vaccination before their 15th birthday were considered complete after 2 doses, with visit dates ≥5 months after receipt of dose 1 considered the minimum interval conferring eligibility for dose 2, consistent with CDC recommendations. Analyses considering intervals of ≥6 months and ≥12 months for dose 2 eligibility yielded similar results.

Our primary outcome was the likelihood that a patient who was due for an HPV vaccine dose would receive vaccination at an eligible clinic visit. All visits to the primary care clinical site by an adolescent who was eligible to receive dose 1, 2, or 3 on the date of the clinic visit were eligible, including nurse-only and problem-focused visits. Visits to specialty departments or emergency departments were not eligible. We performed subset analyses focusing on 9- to 10-year-olds, 11- to 12-year-olds, and 13- to 17-year-olds to examine intervention effects on early, on-time, and catch-up vaccination. Our secondary outcome was rate of “on-time” completion, defined as completion before the 13th birthday, for children aged 12 and 13 years on the visit date. We also examined population-level vaccine initiation and completion rates estimated for patients who visited a study site at any time during the preintervention, intervention, or postintervention periods.

We used longitudinal generalized linear models to estimate the effect of the intervention on the primary and secondary outcomes.29,30  Our main covariate of interest was a categorical indicator of the 3 study periods: preintervention (0), intervention (1), and postintervention (2). For the primary outcome analysis, the unit of observation was a patient outpatient visit, and we included all eligible patient visits during the study period. We regressed the indicator of receipt of an eligible vaccination (0 or 1) on the period indicator as well as potential confounding variables, including child age, sex, race and ethnicity; English as primary language; insurance coverage type; and indicator of receipt of tetanus or meningitis vaccination during the outpatient visit. We included dummy indicators of calendar time by quarter to capture secular temporal trends. We used a patient-level random-effects specification to capture the clustering of all visits and individual unobserved patient effects; because each patient is affiliated with a provider and clinic, patient-level random effects also capture systematic variation by provider and clinic. We obtained heteroscedasticity-consistent robust SEs.31  Similar regression models were estimated for the secondary outcomes by using the appropriate eligible cohort noted above. As a check on the robustness of our findings, we estimated the same regression models with a patient-level fixed-effects specification instead of random effects; with fixed effects, only within-patient variation across visits is used to obtain estimates, avoiding potential confounding from unobserved factors that may be correlated with patient-level variables.31  Fixed-effects models yielded similar results; random-effects models are reported in the text and tables. The Boston University Internal Review Board approved all intervention and evaluation protocols.

A total of 16 136 individuals were included in our analyses (Table 1). Patient demographics reflected the populations served by the clinical sites in the preintervention, intervention, and postintervention periods. Patients had a mean age of 12 years (range: 9–17 years), were equally divided between male and female individuals, and were racially diverse (12.0% to 15.3% white, 19.6% to 29.8% black, 37.1% to 55.0% Hispanic or Latino, 13.4% to 17.8% other). Approximately half (42.7% to 56.0%) spoke English as their primary language, and >80% had Medicaid or other subsidized insurance. Differences in demographics between the pre- and postintervention periods relate to the stepped-wedge design. Clinical sites 1, 2, and 3 had greater proportions of Hispanic and non-Hispanic black patients and longer postintervention periods, whereas clinical sites 4 and 5 had more non-Hispanic white patients and longer preintervention periods.

TABLE 1

Demographic Characteristics of Adolescents Aged 9–17 With at Least 1 Visit to Clinic Sites During the Study Period (N = 16 136)

Preintervention PeriodIntervention PeriodPPostintervention PeriodP
Sex, n (%)      
 Male 6907 (53.5) 5370 (51.7) .0060 10 358 (50.1) <.001 
 Female 6000 (46.5) 5016 (48.3)  10 336 (49.9)  
Race, n (%)      
 Non-Hispanic white 1973 (15.3) 1111 (10.7) <.001 2468 (12.0) <.001 
 Non-Hispanic black 3843 (29.8) 2819 (27.1) <.001 4062 (19.6) <.001 
 Hispanic 4792 (37.1) 4807 (46.3) <.001 11 382 (55.0) <.001 
 Other 2299 (17.8) 1649 (15.9) <.001 2782 (13.4) <.001 
Age, mean (SD) 12.1 (2.43) 12.0 (2.31) .0105 12.2 (2.23) <.001 
English is primary language, n (%) 7228 (56.0) 5094 (49.1) <.001 8825 (42.7) <.001 
Public health insurance, n (%) 10 446 (80.9) 8900 (85.7) <.001 18 494 (89.4) <.001 
Private health insurance, n (%) 2461 (19.1) 1486 (14.3)  2200 (10.6)  
Preintervention PeriodIntervention PeriodPPostintervention PeriodP
Sex, n (%)      
 Male 6907 (53.5) 5370 (51.7) .0060 10 358 (50.1) <.001 
 Female 6000 (46.5) 5016 (48.3)  10 336 (49.9)  
Race, n (%)      
 Non-Hispanic white 1973 (15.3) 1111 (10.7) <.001 2468 (12.0) <.001 
 Non-Hispanic black 3843 (29.8) 2819 (27.1) <.001 4062 (19.6) <.001 
 Hispanic 4792 (37.1) 4807 (46.3) <.001 11 382 (55.0) <.001 
 Other 2299 (17.8) 1649 (15.9) <.001 2782 (13.4) <.001 
Age, mean (SD) 12.1 (2.43) 12.0 (2.31) .0105 12.2 (2.23) <.001 
English is primary language, n (%) 7228 (56.0) 5094 (49.1) <.001 8825 (42.7) <.001 
Public health insurance, n (%) 10 446 (80.9) 8900 (85.7) <.001 18 494 (89.4) <.001 
Private health insurance, n (%) 2461 (19.1) 1486 (14.3)  2200 (10.6)  

P value is listed once for binary variables. P values indicate statistical significance of differences in demographic variables comparing the preintervention period to the intervention and postintervention periods.

In Fig 1, we describe the stepped-wedge design in detail by calendar month. The action plan components enacted by each site are noted, as are whether these changes were continued in the postintervention period. The timing of preintervention, intervention, and postintervention phases is delineated for each site, along with the unadjusted vaccine initiation prevalence stratified by ages 9 to 10, 11 to 12, and 13 to 17 years. The monthly vaccine initiation prevalence is the proportion of adolescents with an eligible visit at the site in that month that initiated the HPV vaccine series either before or during the visit. Initiation rates for 9- and 10-year-olds generally increased during the intervention periods, generally from <20% to >50%. More gradual improvements were noted for older age groups.

At each intervention site, we evaluated unadjusted preintervention, intervention, and postintervention population coverage of HPV vaccine series initiation and completion, defined as the proportion of all 9- to 17-year-old patients with at least 1 visit to the clinical site in the preintervention, intervention, or postintervention periods with either 1 HPV vaccine dose or a complete HPV vaccine series, respectively. For all clinics combined, population-level series initiation coverage increased from 75% (preintervention) to 84% (intervention) to 90% (postintervention), and series completion increased from 60% (preintervention) to 63% (intervention) to 69% (postintervention) (P < .001 for all comparisons). Increases occurred at all clinical sites, with initiation rates exceeding 80% for all sites by the postintervention period (Fig 2 A and B). Because HPV vaccination is routinely recommended at ages 11 to 12, this group was analyzed separately. Series initiation increased from 83% (preintervention) to 89% (intervention) to 93% (postintervention), and series completion increased from 54% (preintervention) to 53% (intervention) to 69% (postintervention) (P < .001 for all comparisons except completion in the intervention period, P = .40); individual clinic performance is detailed in Fig 2 C and D.

FIGURE 2

Impact of intervention on vaccine series initiation and completion. In this figure, we detail changes in vaccine series initiation and completion rates at all intervention sites combined (all sites) as well as individual intervention sites (sites 1–5) in the preintervention, intervention, and postintervention periods. A and B, Data on all adolescents ages 9 to 17. C and D, Data on 11- to 12-year-olds, the age for which routine vaccination is recommended. (Note that clinic 5 had only a 6-month postintervention period, limiting ability to detect change.) Total population prevalence is examined in these outcomes; therefore, patients vaccinated before the intervention were excluded from primary outcomes (likelihood of vaccination at a given visit) but were included here.

FIGURE 2

Impact of intervention on vaccine series initiation and completion. In this figure, we detail changes in vaccine series initiation and completion rates at all intervention sites combined (all sites) as well as individual intervention sites (sites 1–5) in the preintervention, intervention, and postintervention periods. A and B, Data on all adolescents ages 9 to 17. C and D, Data on 11- to 12-year-olds, the age for which routine vaccination is recommended. (Note that clinic 5 had only a 6-month postintervention period, limiting ability to detect change.) Total population prevalence is examined in these outcomes; therefore, patients vaccinated before the intervention were excluded from primary outcomes (likelihood of vaccination at a given visit) but were included here.

Close modal

Adjusted models that accounted for clustering, covariates of interest, and calendar time were used to further explore the effects of the intervention. Adjusted models indicated that the likelihood of receiving vaccination at an eligible visit for children aged 9 to 10 increased by 12.7 percentage points (intervention) and 27.3 percentage points (postintervention) compared with the preintervention period (P < .001 for all comparisons; Table 2). For ages 11 to 12, increases were 16.2 percentage points (intervention) and 17.8 percentage points (postintervention periods; P < .001 for all comparisons; Table 2). For ages 13 to 17, the likelihood of vaccination increased by 8.1 percentage points in the intervention period (P < .001; Table 2) but did not remain significantly elevated in the postintervention period. Hispanic and non-Hispanic black patients, patients whose primary language was not English, and those with subsidized insurance were more likely to receive vaccination than were non-Hispanic white, English-speaking, and privately insured patients. Receipt of tetanus and/or meningitis vaccination was strongly associated with receipt of HPV vaccination at the same visit. Over time, the likelihood of vaccination at an eligible visit continued to increase for ages 9 to 10 but decreased for older age groups. This may represent a ceiling effect, as >90% of patients aged 11 and older who attended clinic visits in the postintervention periods had already initiated the vaccine series (Fig 1). Visits appearing as “missed opportunities” in older age groups in later years may represent factors not amenable to clinical practice changes; for example, patients who either were ineligible or whose parents consistently refused vaccination. We found some indirect evidence for a ceiling effect: the proportion of eligible visits with a previous missed opportunity rose from 35% to 71% from the preintervention to the postintervention periods, and adjusted models indicated that patients with previous missed opportunities were less likely to be vaccinated at future visits (P < .001; data not shown).

TABLE 2

Change in Likelihood of Adolescent Receiving Vaccination at an Eligible Visit During Intervention and Postintervention Periods Compared With Preintervention Period

VariableAges 9–10Ages 11–12Ages 13–17
Percentage Point Change in Likelihood of VaccinationaPPercentage Point Change in Likelihood of VaccinationaPPercentage Point Change in Likelihood of VaccinationaP
Study period       
 Preintervention (referenceb), % 20.2 — 45.6 — 42.5 — 
 Intervention +12.7 <.001 +16.2 <.001 +8.1 <.001 
 Postintervention +27.3 <.001 +17.8 <.001 +1.1 .49 
Sex       
 Female (reference), % 33.4 — 57.6 — 47.8 — 
 Male +3.1 .013 +1.6 .04 −6.1 <.001 
Race       
 Non-Hispanic white (reference), % 30.2 — 52.4 — 40.9 — 
 Non-Hispanic black +2.7 .08 +5.5 <.001 +4.0 .005 
 Hispanic +8.3 <.001 +7.9 <.001 +5.7 <.001 
 Other +2.2 .18 +4.2 .01 +2.5 .11 
Primary language       
 Other (reference), % 36.3 — 59.3 — 47.9 — 
 English −2.7 .006 −2.4 .016 −6.0 <.001 
Insurance       
 Private (reference), % 32.5 — 56.0 — 42.9 — 
 Public +2.8 .03 +2.7 .048 +2.4 .05 
Received meningitis or tetanus vaccines at visit       
 No (reference), % 34.5 — 40.7 — 37.3 — 
 Yes +14.5 <.001 +48.0 <.001 +40.0 <.001 
Calendar time       
 2015 fourth quarter (reference), % 21.2 — 80.4 — 61.8 — 
 2016 first quarter 0.0 .93 −2.2 .23 −6.0 .001 
 2016 second quarter −2.6 .03 −6.8 .001 −4.1 .03 
 2016 third quarter +10.8 <.001 −5.6 .007 −3.3 .1 
 2016 fourth quarter +13.9 <.001 −18.1 <.001 −8.4 <.001 
 2017 first quarter +17.6 <.001 −21.2 <.001 −14.0 <.001 
 2017 second quarter +18.5 <.001 −23.3 <.001 −18.7 <.001 
 2017 third quarter +27.3 <.001 −22.8 <.001 −18.0 <.001 
 2017 fourth quarter +20.5 <.001 −32.8 <.001 −21.7 <.001 
 2018 first quarter +21.9 <.001 −37.0 <.001 −28.2 <.001 
 2018 second quarter +17.7 <.001 −41.8 <.001 −31.7 <.001 
 2018 third quarter +19.8 <.001 −41.8 <.001 −33.1 <.001 
VariableAges 9–10Ages 11–12Ages 13–17
Percentage Point Change in Likelihood of VaccinationaPPercentage Point Change in Likelihood of VaccinationaPPercentage Point Change in Likelihood of VaccinationaP
Study period       
 Preintervention (referenceb), % 20.2 — 45.6 — 42.5 — 
 Intervention +12.7 <.001 +16.2 <.001 +8.1 <.001 
 Postintervention +27.3 <.001 +17.8 <.001 +1.1 .49 
Sex       
 Female (reference), % 33.4 — 57.6 — 47.8 — 
 Male +3.1 .013 +1.6 .04 −6.1 <.001 
Race       
 Non-Hispanic white (reference), % 30.2 — 52.4 — 40.9 — 
 Non-Hispanic black +2.7 .08 +5.5 <.001 +4.0 .005 
 Hispanic +8.3 <.001 +7.9 <.001 +5.7 <.001 
 Other +2.2 .18 +4.2 .01 +2.5 .11 
Primary language       
 Other (reference), % 36.3 — 59.3 — 47.9 — 
 English −2.7 .006 −2.4 .016 −6.0 <.001 
Insurance       
 Private (reference), % 32.5 — 56.0 — 42.9 — 
 Public +2.8 .03 +2.7 .048 +2.4 .05 
Received meningitis or tetanus vaccines at visit       
 No (reference), % 34.5 — 40.7 — 37.3 — 
 Yes +14.5 <.001 +48.0 <.001 +40.0 <.001 
Calendar time       
 2015 fourth quarter (reference), % 21.2 — 80.4 — 61.8 — 
 2016 first quarter 0.0 .93 −2.2 .23 −6.0 .001 
 2016 second quarter −2.6 .03 −6.8 .001 −4.1 .03 
 2016 third quarter +10.8 <.001 −5.6 .007 −3.3 .1 
 2016 fourth quarter +13.9 <.001 −18.1 <.001 −8.4 <.001 
 2017 first quarter +17.6 <.001 −21.2 <.001 −14.0 <.001 
 2017 second quarter +18.5 <.001 −23.3 <.001 −18.7 <.001 
 2017 third quarter +27.3 <.001 −22.8 <.001 −18.0 <.001 
 2017 fourth quarter +20.5 <.001 −32.8 <.001 −21.7 <.001 
 2018 first quarter +21.9 <.001 −37.0 <.001 −28.2 <.001 
 2018 second quarter +17.7 <.001 −41.8 <.001 −31.7 <.001 
 2018 third quarter +19.8 <.001 −41.8 <.001 −33.1 <.001 

Eligible patients are those who had not completed the vaccine series and for whom a sufficient amount of time had elapsed since the previous dose to be eligible for a subsequent dose. Eligibility was adjusted for national vaccine schedule recommendations at date of visit and age at vaccine initiation. —, reference.

a

Percentage point change refers to an increase or decrease in percentage of adolescents fulfilling criteria (eg, from 20.2% completing series on time preintervention among those aged 9 to 10 years to 20.2% + 27.3%, a total of 47.5% completing the series postintervention).

b

Reference refers to the value against which other categories of that variable are compared. For example, the preintervention period is the reference value for the study period. Percentage point changes estimate the change in the intervention and postintervention periods compared with the preintervention period.

A secondary outcome was on-time completion of the HPV vaccine series, by using the CDC definition of series completion before the 13th birthday and adjusted for visit date relative to national recommendations for a 2-dose schedule. Adjusted models indicate a 4.3 and 4.7 percentage point increase in the likelihood of on-time completion in both the intervention and postintervention periods, respectively, compared with the preintervention period (P < .001; Table 3). Male sex and Hispanic ethnicity were also positively associated with on-time series completion. Over time, the likelihood of an adolescent aged 12 to 13 completing vaccination at or before their eligible clinic visit increased (P < .05 for the third quarter of 2016 onward compared with the fourth quarter of 2015; Table 3).

TABLE 3

Change in Likelihood of Adolescent Completing Vaccine Series by the 13th Birthday During Intervention and Postintervention Periods Compared With Preintervention Period

VariablePercentage Point Change in Likelihood of VaccinationaP
Study period   
 Preintervention (referenceb), % 61.1 — 
 Intervention +4.3 <.001 
 Postintervention +4.7 <.001 
Sex   
 Female (reference), % 62.7 — 
 Male +4.0 <.001 
Race   
 Non-Hispanic white (reference), % 57.4 — 
 Non-Hispanic black 0.0 .93 
 Hispanic +12.1 <.001 
 Other +1.6 .50 
Primary language   
 Other (reference), % 65.1 — 
 English +0.3 .79 
Insurance   
 Private (reference) 64.3 — 
 Public −0.6 .71 
Calendar time   
 2015 fourth quarter (reference), % 59.4 — 
 2016 first quarter +2.7 .03 
 2016 second quarter +2.0 .17 
 2016 third quarter +4.8 .001 
 2016 fourth quarter +4.6 .002 
 2017 first quarter +6.1 <.001 
 2017 second quarter +7.2 <.001 
 2017 third quarter +10.3 <.001 
 2017 fourth quarter +9.0 <.001 
 2018 first quarter +11.5 <.001 
 2018 second quarter +12.1 <.001 
 2018 third quarter +16.7 <.001 
VariablePercentage Point Change in Likelihood of VaccinationaP
Study period   
 Preintervention (referenceb), % 61.1 — 
 Intervention +4.3 <.001 
 Postintervention +4.7 <.001 
Sex   
 Female (reference), % 62.7 — 
 Male +4.0 <.001 
Race   
 Non-Hispanic white (reference), % 57.4 — 
 Non-Hispanic black 0.0 .93 
 Hispanic +12.1 <.001 
 Other +1.6 .50 
Primary language   
 Other (reference), % 65.1 — 
 English +0.3 .79 
Insurance   
 Private (reference) 64.3 — 
 Public −0.6 .71 
Calendar time   
 2015 fourth quarter (reference), % 59.4 — 
 2016 first quarter +2.7 .03 
 2016 second quarter +2.0 .17 
 2016 third quarter +4.8 .001 
 2016 fourth quarter +4.6 .002 
 2017 first quarter +6.1 <.001 
 2017 second quarter +7.2 <.001 
 2017 third quarter +10.3 <.001 
 2017 fourth quarter +9.0 <.001 
 2018 first quarter +11.5 <.001 
 2018 second quarter +12.1 <.001 
 2018 third quarter +16.7 <.001 

Eligible adolescents are those aged 12 to 13 years on visit date; definition of completion is adjusted for national vaccine schedule recommendations on visit date (eg, 3 doses required to be considered complete for visits occurring before December 16, 2016). —, not applicable.

a

Percentage point change refers to increase or decrease in percentage of adolescents fulfilling criteria (eg, from 61.1% completing series by the 13th birthday to 61.1% + 4.7%, a total of 65.8%, postintervention).

b

Reference refers to the value against which other categories of that variable are compared. For example, the preintervention period is the reference value for the study period. Percentage point changes estimate the change in the intervention and postintervention periods compared with the preintervention period.

This stepped-wedge randomized trial of DOSE HPV, a multilevel intervention, was shown to improve vaccination rates during the intervention period, and improvements were sustained during follow-up periods ranging from 6 to 18 months. Our findings contribute to the body of evidence indicating that multilevel interventions appear promising as a means of raising vaccination rates.1923  Because multilevel interventions require substantial investments of personnel and time in the short-term, demonstrating that intervention effects continue in the postintervention period is important when clinical and policy decision-makers consider upfront costs.

Maintaining gains in vaccination rates may depend on the sustainability of intervention components.32  The DOSE HPV intervention activates providers to create tailored systems changes, which may allow improvements in HPV vaccination provision to be sustained after intervention completion. Health intervention literature indicates that systems-level changes require fewer ongoing resources and can be more sustainable than provider-focused efforts, especially as participating providers may leave the practice after intervention completion.33,34  However, research also demonstrates benefits to educating providers35  as well as developing interventions in partnership with stakeholders to ensure relevance and capacity.33,34  Viewing the DOSE HPV intervention through this lens, educational sessions and communications training may have activated providers in the short-term, and allowing participants to develop tailored systems changes to address barriers may have promoted sustainability by building engagement and aligning efforts with existing clinical processes.32,36,37  System-wide efforts to routinely recommend HPV vaccination at younger ages were considered to be easy and successful changes by providers38  and appeared to be a key factor underlying population-level gains over time. Initiating vaccination at age 9 or 10 can increase the number of opportunities to complete vaccination on time.39  High rates of vaccination among 9- and 10-year-olds in the postintervention period indicated that early vaccination was acceptable to parents. Providers attributed high parental acceptance of early vaccination to a preference for fewer shots per visit as well as an attenuated connection between vaccination and sexual activity at younger ages.38  The ability of sites to implement and sustain this change is demonstrated by increases in early and on-time vaccine series initiation and completion that persisted through the postintervention periods. Researchers of other studies have also demonstrated that initiating vaccination before age 11 years can increase on-time vaccination rates; therefore, this may be a crucial component of future interventions.40,41 

This study has several limitations, including a limited geographic sampling area and implementation in clinics serving primarily low-income, minority, urban populations. Although the clinical sites were all resource constrained, they did have relatively high HPV vaccine initiation rates at baseline (75%), indicating that providers and parents were already accustomed to vaccinating. Two of our sites transitioned to a new electronic medical record system, limiting our ability to identify patients who might receive primary care at the intervention sites but who had not presented for care in >12 months; this in turn could lead to overestimation of population-level vaccination rates. Also, we could not identify immunocompromised adolescents who required a 3-dose schedule.

HPV vaccination rates continued to increase nationally during the study period, likely facilitated by the release of the 2-dose schedule in December 2016. Although the stepped-wedge design is intended to control for secular changes, fully disentangling intervention effects from contextual factors is difficult. To be able to statistically estimate the effect of the intervention from that of the 2-dose schedule, we would need to separately estimate the DOSE HPV effect among those on 3-dose and 2-dose schedules separately. Because our intervention focused primarily on younger adolescents, all of whom were affected by the 2-dose schedule switch, we cannot perform this type of stratified analysis and primarily use the stepped-wedge design to assess for secular trends. The regression models indicate that the intervention increased the likelihood of vaccine receipt and series completion after controlling for calendar time. Despite these limitations, this trial is one of the first to indicate effectiveness of a multilevel intervention leading to improvements in HPV vaccination rates 6 to 18 months beyond completion of the intervention. If long-term gains are replicated in other populations, these types of interventions could be uniquely useful tools to improve HPV vaccination rates in the United States.

Multilevel interventions that include interprofessional provider education, data feedback, and tailored systems changes that focus on initiating HPV vaccination before age 11 may be successful in improving HPV vaccine series initiation and completion beyond the conclusion of the intervention period. Replication of these findings in settings with lower baseline vaccination rates, in other geographic regions, and among rural populations and those with private insurance is an important area for future research.

We thank the providers and clinic staff who participated in the intervention.

Dr Perkins conceptualized, designed, and delivered component interventions of the study, supervised data collection, drafted the initial manuscript, and revised and reviewed the manuscript; Dr Bernstein conceptualized the study, delivered core curricular components on motivational interviewing, and reviewed the manuscript; Drs Joseph, Adams, Clark, Fenton, and Drainoni conceptualized and designed the study and critically reviewed and revised the manuscript; Dr Hanchate and Mr Legler conceptualized and designed the study methodology and statistical analyses and conducted all data analyses; Drs Schuch and Leschly oversaw implementation of interventions and critically reviewed and revised the manuscript; Ms Eun and Ms Biancarelli critically reviewed and revised the manuscript; Ms Jansen delivered component interventions of the study, performed data analysis during active intervention, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

This trial has been registered at www.clinicaltrials.gov (identifier NCT02812732).

FUNDING: Supported by an American Cancer Society Research Scholar grant (grant 128607-RSG-15-150-01-CPHPS). No commercial support was obtained.

CDC

Centers for Disease Control and Prevention

DOSE HPV

Development of Systems and Education for Human Papillomavirus Vaccination

HPV

human papillomavirus

PI CME

Performance Improvement Continuing Medical Education

1
Drolet
M
,
Bénard
É
,
Boily
MC
, et al
.
Population-level impact and herd effects following human papillomavirus vaccination programmes: a systematic review and meta-analysis
.
Lancet Infect Dis
.
2015
;
15
(
5
):
565
580
2
Luostarinen
T
,
Apter
D
,
Dillner
J
, et al
.
Vaccination protects against invasive HPV-associated cancers
.
Int J Cancer
.
2018
;
142
(
10
):
2186
2187
3
World Health Organization
.
Human papillomavirus vaccines: WHO position paper, October 2014
.
Wkly Epidemiol Rec
.
2014
;
89
(
43
):
465
491
4
Markowitz
LE
,
Dunne
EF
,
Saraiya
M
, et al;
Centers for Disease Control and Prevention (CDC)
.
Human papillomavirus vaccination: recommendations of the Advisory Committee on Immunization Practices (ACIP) [published correction appears in MMWR Recomm Rep. 2014;63(49):1182]
.
MMWR Recomm Rep
.
2014
;
63
(
RR
):
1
30
5
Meites
E
,
Kempe
A
,
Markowitz
LE
.
Use of a 2-dose schedule for human papillomavirus vaccination - updated recommendations of the Advisory Committee on Immunization Practices
.
MMWR Morb Mortal Wkly Rep
.
2016
;
65
(
49
):
1405
1408
6
Meites
E
,
Szilagyi
PG
,
Chesson
HW
,
Unger
ER
,
Romero
JR
,
Markowitz
LE
.
Human papillomavirus vaccination for adults: updated recommendations of the Advisory Committee on Immunization Practices
.
MMWR Morb Mortal Wkly Rep
.
2019
;
68
(
32
):
698
702
7
Gertig
DM
,
Brotherton
JM
,
Budd
AC
,
Drennan
K
,
Chappell
G
,
Saville
AM
.
Impact of a population-based HPV vaccination program on cervical abnormalities: a data linkage study
.
BMC Med
.
2013
;
11
:
227
8
Guevara
A
,
Cabello
R
,
Woelber
L
, et al
.
Antibody persistence and evidence of immune memory at 5years following administration of the 9-valent HPV vaccine
.
Vaccine
.
2017
;
35
(
37
):
5050
5057
9
HealthyPeople 2020
.
Immunization and infectious diseases
. Available at: https://www.healthypeople.gov/2020/topics-objectives/topic/immunization-and-infectious-diseases/objectives. Accessed July 2, 2019
10
Walker
TY
,
Elam-Evans
LD
,
Yankey
D
, et al
.
National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years - United States, 2018
.
MMWR Morb Mortal Wkly Rep
.
2019
;
68
(
33
):
718
723
11
National Committee for Quality Assurance
.
Immunizations for adolescents (IMA)
. Available at: https://www.ncqa.org/hedis/measures/immunizations-for-adolescents/. Accessed August 8, 2019
12
Holman
DM
,
Benard
V
,
Roland
KB
,
Watson
M
,
Liddon
N
,
Stokley
S
.
Barriers to human papillomavirus vaccination among US adolescents: a systematic review of the literature
.
JAMA Pediatr
.
2014
;
168
(
1
):
76
82
13
Perkins
RB
,
Clark
JA
,
Apte
G
, et al
.
Missed opportunities for HPV vaccination in adolescent girls: a qualitative study
.
Pediatrics
.
2014
;
134
(
3
). Available at: www.pediatrics.org/cgi/content/full/134/3/e666
14
Vadaparampil
ST
,
Kahn
JA
,
Salmon
D
, et al
.
Missed clinical opportunities: provider recommendations for HPV vaccination for 11–12 year old girls are limited
.
Vaccine
.
2011
;
29
(
47
):
8634
8641
15
Rosenthal
SL
,
Weiss
TW
,
Zimet
GD
,
Ma
L
,
Good
MB
,
Vichnin
MD
.
Predictors of HPV vaccine uptake among women aged 19-26: importance of a physician’s recommendation
.
Vaccine
.
2011
;
29
(
5
):
890
895
16
Albright
K
,
Barnard
J
,
O’Leary
ST
, et al
.
Noninitiation and noncompletion of HPV vaccine among English- and Spanish-speaking parents of adolescent girls: a qualitative study
.
Acad Pediatr
.
2017
;
17
(
7
):
778
784
17
Beavis
AL
,
Levinson
KL
.
Preventing cervical cancer in the United States: barriers and resolutions for HPV vaccination
.
Front Oncol
.
2016
;
6
:
19
18
Perkins
RB
,
Chigurupati
NL
,
Apte
G
, et al
.
Why don’t adolescents finish the HPV vaccine series? A qualitative study of parents and providers
.
Hum Vaccin Immunother
.
2016
;
12
(
6
):
1528
1535
19
Brewer
NT
,
Hall
ME
,
Malo
TL
,
Gilkey
MB
,
Quinn
B
,
Lathren
C
.
Announcements versus conversations to improve HPV vaccination coverage: a randomized trial
.
Pediatrics
.
2017
;
139
(
1
):
e20161764
20
Paskett
ED
,
Krok-Schoen
JL
,
Pennell
ML
, et al
.
Results of a multilevel intervention trial to increase human papillomavirus (HPV) vaccine uptake among adolescent girls
.
Cancer Epidemiol Biomarkers Prev
.
2016
;
25
(
4
):
593
602
21
Fisher-Borne
M
,
Preiss
AJ
,
Black
M
,
Roberts
K
,
Saslow
D
.
Early outcomes of a multilevel human papillomavirus vaccination pilot intervention in federally qualified health centers
.
Acad Pediatr
.
2018
;
18
(
2S
):
S79
S84
22
Gilkey
MB
,
Dayton
AM
,
Moss
JL
, et al
.
Increasing provision of adolescent vaccines in primary care: a randomized controlled trial
.
Pediatrics
.
2014
;
134
(
2
). Available at: www.pediatrics.org/cgi/content/full/134/2/e346
23
Perkins
RB
,
Zisblatt
L
,
Legler
A
,
Trucks
E
,
Hanchate
A
,
Gorin
SS
.
Effectiveness of a provider-focused intervention to improve HPV vaccination rates in boys and girls
.
Vaccine
.
2015
;
33
(
9
):
1223
1229
24
Zimet
G
,
Dixon
BE
,
Xiao
S
, et al
.
Simple and elaborated clinician reminder prompts for human papillomavirus vaccination: a randomized clinical trial
.
Acad Pediatr
.
2018
;
18
(
2S
):
S66
S71
25
Krantz
L
,
Ollberding
NJ
,
Beck
AF
,
Carol Burkhardt
M
.
Increasing HPV vaccination coverage through provider-based interventions
.
Clin Pediatr (Phila)
.
2018
;
57
(
3
):
319
326
26
Dempsey
AF
,
Pyrznawoski
J
,
Lockhart
S
, et al
.
Effect of a health care professional communication training intervention on adolescent human papillomavirus vaccination: a cluster randomized clinical trial
.
JAMA Pediatr
.
2018
;
172
(
5
):
e180016
27
National Cancer Institute
;
Research Tested Intervention Program.
DOSE HPV: development of systems and education for HPV vaccination. Available at: https://rtips.cancer.gov/rtips/programDetails.do?programId=25930477. Accessed August 1, 2019
28
Joseph
NP
,
Bernstein
J
,
Pelton
S
, et al
.
Brief client-centered motivational and behavioral intervention to promote HPV vaccination in a hard-to-reach population: a pilot randomized controlled trial
.
Clin Pediatr (Phila)
.
2016
;
55
(
9
):
851
859
29
Hemming
K
,
Haines
TP
,
Chilton
PJ
,
Girling
AJ
,
Lilford
RJ
.
The stepped wedge cluster randomised trial: rationale, design, analysis, and reporting
.
BMJ
.
2015
;
350
:
h391
30
Twisk
JWR
,
de Vente
W
.
The analysis of randomised controlled trial data with more than one follow-up measurement. A comparison between different approaches
.
Eur J Epidemiol
.
2008
;
23
(
10
):
655
660
31
Cameron
AC
,
Trivedi
PK
.
Microeconometrics: Methods and Applications
.
New York
:
Cambridge University Press
;
2005
32
Walling
EB
,
Benzoni
N
,
Dornfeld
J
, et al
.
Interventions to improve HPV vaccine uptake: a systematic review
.
Pediatrics
.
2016
;
138
(
1
):
e20153863
33
Swerissen
H
,
Crisp
BR
.
The sustainability of health promotion interventions for different levels of social organization
.
Health Promot Int
.
2004
;
19
(
1
):
123
130
34
Wiltsey Stirman
S
,
Kimberly
J
,
Cook
N
,
Calloway
A
,
Castro
F
,
Charns
M
.
The sustainability of new programs and innovations: a review of the empirical literature and recommendations for future research
.
Implement Sci
.
2012
;
7
:
17
35
Leung
SOA
,
Akinwunmi
B
,
Elias
KM
,
Feldman
S
.
Educating healthcare providers to increase human papillomavirus (HPV) vaccination rates: a qualitative systematic review
.
Vaccine X
.
2019
;
3
:
100037
36
Smulian
EA
,
Mitchell
KR
,
Stokley
S
.
Interventions to increase HPV vaccination coverage: a systematic review
.
Hum Vaccin Immunother
.
2016
;
12
(
6
):
1566
1588
37
Whelan
J
,
Love
P
,
Pettman
T
, et al
.
Cochrane update: predicting sustainability of intervention effects in public health evidence: identifying key elements to provide guidance
.
J Public Health (Oxf)
.
2014
;
36
(
2
):
347
351
38
Biancarelli
DL
,
Drainoni
M-L
,
Perkins
RB
.
Provider experience recommending HPV vaccination before age 11 years
.
J Pediatr
.
2020
;
217
:
92
97
39
Uddin
SG
,
O’Connor
KS
,
Ashman
JJ
.
Physician office visits by children for well and problem-focused care: United States, 2012
.
NCHS Data Brief
.
2016
;(
248
):
1
8
40
St Sauver
JL
,
Rutten
LJF
,
Ebbert
JO
,
Jacobson
DJ
,
McGree
ME
,
Jacobson
RM
.
Younger age at initiation of the human papillomavirus (HPV) vaccination series is associated with higher rates of on-time completion
.
Prev Med
.
2016
;
89
:
327
333
41
Goleman
MJ
,
Dolce
M
,
Morack
J
.
Quality improvement initiative to improve human papillomavirus vaccine initiation at 9 years of age
.
Acad Pediatr
.
2018
;
18
(
7
):
769
775

Competing Interests

POTENTIAL CONFLICT OF INTEREST: Dr Joseph has received grant funding from Merck; the other authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.