Video Abstract
To evaluate the effectiveness of a stepped-wedge randomized trial of Development of Systems and Education for Human Papillomavirus Vaccination (DOSE HPV), a multilevel intervention.
DOSE HPV is a 7-session program that includes interprofessional provider education, communication training, data feedback, and tailored systems change. Five primary care pediatric and/or family medicine practices completed interventions between 2016 and 2018; all chose to initiate vaccination at ages 9 to 10. We compared vaccination rates in the preintervention, intervention, and postintervention periods among 9- to 17-year-olds using random-effects generalized linear regression models appropriate for stepped-wedge design, accounting for calendar time and clustering of patients by providers and clinic. Outcomes included (1) the likelihood that eligible patients would receive vaccination during clinic visits; (2) the likelihood that adolescents would complete the series by age 13; and (3) the cumulative effect on population-level vaccine initiation and completion rates. Postintervention periods ranged from 6 to 18 months.
In the intervention and postintervention periods, the adjusted likelihood of vaccination at an eligible visit increased by >10 percentage points for ages 9 to 10 and 11 to 12, and completion of the vaccine series by age 13 increased by 4 percentage points (P < .001 for all comparisons). Population-level vaccine initiation coverage increased from 75% (preintervention) to 84% (intervention) to 90% (postintervention), and completion increased from 60% (preintervention) to 63% (intervention) to 69% (postintervention).
Multilevel interventions that include provider education, data feedback, tailored systems changes, and early initiation of the human papillomavirus vaccine series may improve vaccine series initiation and completion beyond the conclusion of the intervention period.
Multilevel interventions targeting providers appear to increase human papillomavirus vaccine series initiation rates. Limited research exists on the effects of multilevel interventions on series completion, the sustainability of increases, and the impact of lowering the routine age of vaccination.
Multilevel interventions targeting the interprofessional team that include provider education, data feedback, tailored systems changes, and early initiation of the human papillomavirus vaccine series may be successful in improving vaccine series initiation and completion beyond the conclusion of the intervention period.
Data from the United States and other countries demonstrate that human papillomavirus (HPV) vaccination profoundly reduces vaccine-type oncogenic HPV infections, genital warts, and precancers of the cervix, with emerging data revealing reduction in HPV-related cancers.1–3 The Centers for Disease Control and Prevention (CDC) recommend routine HPV vaccine initiation between the ages of 11 and 12 years. Vaccination can begin at age 9 and is also recommended for all adolescents aged 13 to 26 years who were not vaccinated on time.4–6 Initiating vaccination at an earlier age is preferable to vaccinating in older adolescence because vaccination is most effective before sexual debut,7 and no waning of protection is observed.8 HPV vaccination rates remain below national goals of 80% of eligible adolescents completing the vaccine series,9 however, because 68.1% of adolescents aged 13 to 17 years initiated and 51.1% completed the vaccine series in 2018.9,10 Only 39.9% of adolescents completed the series on time, defined as before their 13th birthday.11
Research has identified barriers to HPV vaccination,12 including lack of effective provider recommendation13–16 and lack of reminder and recall systems, leading to missed opportunities for both series initiation and completion.17,18 Multilevel interventions show promise for raising HPV vaccination rates by using techniques including provider education and communication training, data assessment and feedback, and quality improvement methods.19–26 Increases in vaccination rates varied by study type and setting, and postintervention follow-up did not demonstrate sustainability after intervention completion.22
In 2011 and 2012, we piloted Development of Systems and Education for Human Papillomavirus Vaccination (DOSE HPV), a multilevel performance improvement continuing medical education (PI CME) intervention that includes provider education and communication training, data assessment and feedback, and tailored systems interventions to improve HPV vaccination rates among socioeconomically vulnerable populations.23,27 PI CME is a learning format consisting of longitudinal practice improvement, for which physicians may earn continuing medical education credit. Participation requirements for PI CME align with maintenance of certification part IV requirements for board-certified pediatric, internal medicine, and family medicine physicians. For nurses, nursing professional development (ie, contact hours) is provided for participation in live meetings. The pilot study involved 2 pediatric practices, used a pre- and postevaluation design, and was conducted when a 3-dose schedule was the national recommendation. To better estimate intervention effectiveness and provide short-term sustainability data, we performed a stepped-wedge cluster randomized trial of the DOSE HPV intervention at 5 additional sites. We chose the stepped-wedge design as the best study design to separate secular trends from intervention effects as well as incentivize participation by clinical sites because all sites received the intervention. We report on the intervention’s effects on HPV vaccine series initiation and completion up through 18 months after intervention completion.
Methods
DOSE HPV Intervention and Setting
Interventions took place in Massachusetts in the Boston metropolitan area and included 5 clinical sites serving primarily low-income and minority patients. One site was a safety-net hospital; the others were independent federally qualified health centers. Sites were selected before the intervention period on the basis of 2014 site-specific data indicating HPV vaccination rates below national targets.9 The DOSE HPV intervention is a provider-focused program that included 7 sessions, 1 hour each, performed at approximately monthly intervals over 6 to 8 months between 2016 and 2018 (Fig 1). Sessions were held during regular staff meetings to maximize attendance, and food was served. Intervention sessions were interprofessional, including clinic leadership, primary care providers, and nursing staff. The intervention educates providers and activates them to make tailored systems changes within their health care settings. The intervention included 3 core components: interprofessional education and communication training for health care providers, data assessment and feedback, and creation of an action plan that included both provider- and systems-level changes. All sessions were facilitated by the intervention team, and sessions at which site-specific action plans were developed were led by clinical champions and participating providers.
Intervention time line and content with adolescent vaccination by calendar month. In this figure, we describe the time line of each intervention, as well as the percentage of adolescents with visits per calendar month who initiate the HPV vaccine series before or at the index visit. The granular representation of data details the individual action plan components at each site, and also notes changes in vaccination rates as they relate to the timing of the DOSE HPV intervention as well as release of the national 2-dose recommendations. EHR, electronic health record; PCP, primary care provider. a National recommendations for 2-dose schedule were published on December 16, 2016, in the Morbidity and Mortality Weekly Report. b Denotes action plan components that were discontinued. Recalls for patients overdue for vaccine were discontinued because of high staff effort and low patient response. Standing orders were discontinued because of staff turnover. EHR prompt was discontinued at site 5 because of PCP workflow issues. All other action plan components were continued by clinical sites in the postintervention period. c Sites 4 and 5 transitioned to a different electronic medical record system during the preintervention period, leading to a 1-month delay in preintervention data collection in site 4 and an 8-month delay in site 5.
Intervention time line and content with adolescent vaccination by calendar month. In this figure, we describe the time line of each intervention, as well as the percentage of adolescents with visits per calendar month who initiate the HPV vaccine series before or at the index visit. The granular representation of data details the individual action plan components at each site, and also notes changes in vaccination rates as they relate to the timing of the DOSE HPV intervention as well as release of the national 2-dose recommendations. EHR, electronic health record; PCP, primary care provider. a National recommendations for 2-dose schedule were published on December 16, 2016, in the Morbidity and Mortality Weekly Report. b Denotes action plan components that were discontinued. Recalls for patients overdue for vaccine were discontinued because of high staff effort and low patient response. Standing orders were discontinued because of staff turnover. EHR prompt was discontinued at site 5 because of PCP workflow issues. All other action plan components were continued by clinical sites in the postintervention period. c Sites 4 and 5 transitioned to a different electronic medical record system during the preintervention period, leading to a 1-month delay in preintervention data collection in site 4 and an 8-month delay in site 5.
In session 1, the study team provided health center–specific preintervention data on HPV vaccine initiation and completion rates. For session 2, a physician expert provided education on HPV-associated cancers, effectiveness of vaccination, and evidence-based interventions. Sessions 3 and 4 were focused on motivational interviewing training.28 During session 5, the study team used quality improvement methods to collaboratively identify barriers to HPV vaccination and create site-developed action plans, including systems- and provider-level changes to improve performance. Sessions 6 and 7 were facilitated discussions reviewing progress on implementing action plans, defining near-term action steps, and specifying task ownership to address obstacles and accelerate progress. Although the 5 sites developed action plans specific to their unique practice environments, all decided to routinely recommend HPV vaccination before age 11 (Fig 1). The data review motivated this change because clinic-level vaccination rates revealed that, although most patients received vaccination at well-child visits, a substantial proportion of their primary care populations remained unvaccinated primarily because of attrition in well-child care among older adolescents.
Data Collection and Analysis
The trial used a stepped-wedge cluster randomized trial design: the study team randomized clinical sites (clusters) by generated number, and interventions occurred in order of their random number, staggered by ∼6 months during the 3 year study period.29 Interventions began on April 7, 2016, and ended on March 15, 2018. Electronic medical record data were used to evaluate the intervention’s effect on HPV vaccination rates. The study population included adolescents aged 9 to 17 years with an assigned primary care provider and ≥1 visit to the primary care site at any time during the 3-year study period. The study period was divided into preintervention, intervention, and postintervention periods for each site. The intention for the stepped-wedge design was for all preintervention periods to begin on the same date (October 1, 2015), 6 months before the intervention start date at clinical site 1. However, because of changeover of electronic medical record systems, preintervention periods began later at sites 4 and 5 (November 1, 2015, and June 1, 2016, respectively; Fig 1). The intervention periods began on the date of the first session and ended on the date of the last session at each site. The postintervention periods began 1 day after the last intervention session at each site and ended on September 30, 2018, for all sites. Differential pre- and postintervention periods in the stepped-wedge design serve to control for secular trends and allow comparisons within sites over time as well as between sites.29
All analyses accounted for the national change from a 3-dose to a 2-dose schedule; dose eligibility mirrored recommendations on the date of the clinic visit. Before December 16, 2016, all adolescents required 3 doses to complete the series; visit dates ≥2 months after the first dose and ≥4 months after the second dose were considered eligible. After December 2016, the 3-dose schedule was applied only to adolescents who initiated the series on or after their 15th birthday. Adolescents who initiated vaccination before their 15th birthday were considered complete after 2 doses, with visit dates ≥5 months after receipt of dose 1 considered the minimum interval conferring eligibility for dose 2, consistent with CDC recommendations. Analyses considering intervals of ≥6 months and ≥12 months for dose 2 eligibility yielded similar results.
Our primary outcome was the likelihood that a patient who was due for an HPV vaccine dose would receive vaccination at an eligible clinic visit. All visits to the primary care clinical site by an adolescent who was eligible to receive dose 1, 2, or 3 on the date of the clinic visit were eligible, including nurse-only and problem-focused visits. Visits to specialty departments or emergency departments were not eligible. We performed subset analyses focusing on 9- to 10-year-olds, 11- to 12-year-olds, and 13- to 17-year-olds to examine intervention effects on early, on-time, and catch-up vaccination. Our secondary outcome was rate of “on-time” completion, defined as completion before the 13th birthday, for children aged 12 and 13 years on the visit date. We also examined population-level vaccine initiation and completion rates estimated for patients who visited a study site at any time during the preintervention, intervention, or postintervention periods.
We used longitudinal generalized linear models to estimate the effect of the intervention on the primary and secondary outcomes.29,30 Our main covariate of interest was a categorical indicator of the 3 study periods: preintervention (0), intervention (1), and postintervention (2). For the primary outcome analysis, the unit of observation was a patient outpatient visit, and we included all eligible patient visits during the study period. We regressed the indicator of receipt of an eligible vaccination (0 or 1) on the period indicator as well as potential confounding variables, including child age, sex, race and ethnicity; English as primary language; insurance coverage type; and indicator of receipt of tetanus or meningitis vaccination during the outpatient visit. We included dummy indicators of calendar time by quarter to capture secular temporal trends. We used a patient-level random-effects specification to capture the clustering of all visits and individual unobserved patient effects; because each patient is affiliated with a provider and clinic, patient-level random effects also capture systematic variation by provider and clinic. We obtained heteroscedasticity-consistent robust SEs.31 Similar regression models were estimated for the secondary outcomes by using the appropriate eligible cohort noted above. As a check on the robustness of our findings, we estimated the same regression models with a patient-level fixed-effects specification instead of random effects; with fixed effects, only within-patient variation across visits is used to obtain estimates, avoiding potential confounding from unobserved factors that may be correlated with patient-level variables.31 Fixed-effects models yielded similar results; random-effects models are reported in the text and tables. The Boston University Internal Review Board approved all intervention and evaluation protocols.
Results
Patient Characteristics
A total of 16 136 individuals were included in our analyses (Table 1). Patient demographics reflected the populations served by the clinical sites in the preintervention, intervention, and postintervention periods. Patients had a mean age of 12 years (range: 9–17 years), were equally divided between male and female individuals, and were racially diverse (12.0% to 15.3% white, 19.6% to 29.8% black, 37.1% to 55.0% Hispanic or Latino, 13.4% to 17.8% other). Approximately half (42.7% to 56.0%) spoke English as their primary language, and >80% had Medicaid or other subsidized insurance. Differences in demographics between the pre- and postintervention periods relate to the stepped-wedge design. Clinical sites 1, 2, and 3 had greater proportions of Hispanic and non-Hispanic black patients and longer postintervention periods, whereas clinical sites 4 and 5 had more non-Hispanic white patients and longer preintervention periods.
Demographic Characteristics of Adolescents Aged 9–17 With at Least 1 Visit to Clinic Sites During the Study Period (N = 16 136)
. | Preintervention Period . | Intervention Period . | P . | Postintervention Period . | P . |
---|---|---|---|---|---|
Sex, n (%) | |||||
Male | 6907 (53.5) | 5370 (51.7) | .0060 | 10 358 (50.1) | <.001 |
Female | 6000 (46.5) | 5016 (48.3) | 10 336 (49.9) | ||
Race, n (%) | |||||
Non-Hispanic white | 1973 (15.3) | 1111 (10.7) | <.001 | 2468 (12.0) | <.001 |
Non-Hispanic black | 3843 (29.8) | 2819 (27.1) | <.001 | 4062 (19.6) | <.001 |
Hispanic | 4792 (37.1) | 4807 (46.3) | <.001 | 11 382 (55.0) | <.001 |
Other | 2299 (17.8) | 1649 (15.9) | <.001 | 2782 (13.4) | <.001 |
Age, mean (SD) | 12.1 (2.43) | 12.0 (2.31) | .0105 | 12.2 (2.23) | <.001 |
English is primary language, n (%) | 7228 (56.0) | 5094 (49.1) | <.001 | 8825 (42.7) | <.001 |
Public health insurance, n (%) | 10 446 (80.9) | 8900 (85.7) | <.001 | 18 494 (89.4) | <.001 |
Private health insurance, n (%) | 2461 (19.1) | 1486 (14.3) | 2200 (10.6) |
. | Preintervention Period . | Intervention Period . | P . | Postintervention Period . | P . |
---|---|---|---|---|---|
Sex, n (%) | |||||
Male | 6907 (53.5) | 5370 (51.7) | .0060 | 10 358 (50.1) | <.001 |
Female | 6000 (46.5) | 5016 (48.3) | 10 336 (49.9) | ||
Race, n (%) | |||||
Non-Hispanic white | 1973 (15.3) | 1111 (10.7) | <.001 | 2468 (12.0) | <.001 |
Non-Hispanic black | 3843 (29.8) | 2819 (27.1) | <.001 | 4062 (19.6) | <.001 |
Hispanic | 4792 (37.1) | 4807 (46.3) | <.001 | 11 382 (55.0) | <.001 |
Other | 2299 (17.8) | 1649 (15.9) | <.001 | 2782 (13.4) | <.001 |
Age, mean (SD) | 12.1 (2.43) | 12.0 (2.31) | .0105 | 12.2 (2.23) | <.001 |
English is primary language, n (%) | 7228 (56.0) | 5094 (49.1) | <.001 | 8825 (42.7) | <.001 |
Public health insurance, n (%) | 10 446 (80.9) | 8900 (85.7) | <.001 | 18 494 (89.4) | <.001 |
Private health insurance, n (%) | 2461 (19.1) | 1486 (14.3) | 2200 (10.6) |
P value is listed once for binary variables. P values indicate statistical significance of differences in demographic variables comparing the preintervention period to the intervention and postintervention periods.
Vaccination Rates at Intervention Sites by Calendar Time and Study Period
In Fig 1, we describe the stepped-wedge design in detail by calendar month. The action plan components enacted by each site are noted, as are whether these changes were continued in the postintervention period. The timing of preintervention, intervention, and postintervention phases is delineated for each site, along with the unadjusted vaccine initiation prevalence stratified by ages 9 to 10, 11 to 12, and 13 to 17 years. The monthly vaccine initiation prevalence is the proportion of adolescents with an eligible visit at the site in that month that initiated the HPV vaccine series either before or during the visit. Initiation rates for 9- and 10-year-olds generally increased during the intervention periods, generally from <20% to >50%. More gradual improvements were noted for older age groups.
At each intervention site, we evaluated unadjusted preintervention, intervention, and postintervention population coverage of HPV vaccine series initiation and completion, defined as the proportion of all 9- to 17-year-old patients with at least 1 visit to the clinical site in the preintervention, intervention, or postintervention periods with either 1 HPV vaccine dose or a complete HPV vaccine series, respectively. For all clinics combined, population-level series initiation coverage increased from 75% (preintervention) to 84% (intervention) to 90% (postintervention), and series completion increased from 60% (preintervention) to 63% (intervention) to 69% (postintervention) (P < .001 for all comparisons). Increases occurred at all clinical sites, with initiation rates exceeding 80% for all sites by the postintervention period (Fig 2 A and B). Because HPV vaccination is routinely recommended at ages 11 to 12, this group was analyzed separately. Series initiation increased from 83% (preintervention) to 89% (intervention) to 93% (postintervention), and series completion increased from 54% (preintervention) to 53% (intervention) to 69% (postintervention) (P < .001 for all comparisons except completion in the intervention period, P = .40); individual clinic performance is detailed in Fig 2 C and D.
Impact of intervention on vaccine series initiation and completion. In this figure, we detail changes in vaccine series initiation and completion rates at all intervention sites combined (all sites) as well as individual intervention sites (sites 1–5) in the preintervention, intervention, and postintervention periods. A and B, Data on all adolescents ages 9 to 17. C and D, Data on 11- to 12-year-olds, the age for which routine vaccination is recommended. (Note that clinic 5 had only a 6-month postintervention period, limiting ability to detect change.) Total population prevalence is examined in these outcomes; therefore, patients vaccinated before the intervention were excluded from primary outcomes (likelihood of vaccination at a given visit) but were included here.
Impact of intervention on vaccine series initiation and completion. In this figure, we detail changes in vaccine series initiation and completion rates at all intervention sites combined (all sites) as well as individual intervention sites (sites 1–5) in the preintervention, intervention, and postintervention periods. A and B, Data on all adolescents ages 9 to 17. C and D, Data on 11- to 12-year-olds, the age for which routine vaccination is recommended. (Note that clinic 5 had only a 6-month postintervention period, limiting ability to detect change.) Total population prevalence is examined in these outcomes; therefore, patients vaccinated before the intervention were excluded from primary outcomes (likelihood of vaccination at a given visit) but were included here.
Adjusted models that accounted for clustering, covariates of interest, and calendar time were used to further explore the effects of the intervention. Adjusted models indicated that the likelihood of receiving vaccination at an eligible visit for children aged 9 to 10 increased by 12.7 percentage points (intervention) and 27.3 percentage points (postintervention) compared with the preintervention period (P < .001 for all comparisons; Table 2). For ages 11 to 12, increases were 16.2 percentage points (intervention) and 17.8 percentage points (postintervention periods; P < .001 for all comparisons; Table 2). For ages 13 to 17, the likelihood of vaccination increased by 8.1 percentage points in the intervention period (P < .001; Table 2) but did not remain significantly elevated in the postintervention period. Hispanic and non-Hispanic black patients, patients whose primary language was not English, and those with subsidized insurance were more likely to receive vaccination than were non-Hispanic white, English-speaking, and privately insured patients. Receipt of tetanus and/or meningitis vaccination was strongly associated with receipt of HPV vaccination at the same visit. Over time, the likelihood of vaccination at an eligible visit continued to increase for ages 9 to 10 but decreased for older age groups. This may represent a ceiling effect, as >90% of patients aged 11 and older who attended clinic visits in the postintervention periods had already initiated the vaccine series (Fig 1). Visits appearing as “missed opportunities” in older age groups in later years may represent factors not amenable to clinical practice changes; for example, patients who either were ineligible or whose parents consistently refused vaccination. We found some indirect evidence for a ceiling effect: the proportion of eligible visits with a previous missed opportunity rose from 35% to 71% from the preintervention to the postintervention periods, and adjusted models indicated that patients with previous missed opportunities were less likely to be vaccinated at future visits (P < .001; data not shown).
Change in Likelihood of Adolescent Receiving Vaccination at an Eligible Visit During Intervention and Postintervention Periods Compared With Preintervention Period
Variable . | Ages 9–10 . | Ages 11–12 . | Ages 13–17 . | |||
---|---|---|---|---|---|---|
Percentage Point Change in Likelihood of Vaccinationa . | P . | Percentage Point Change in Likelihood of Vaccinationa . | P . | Percentage Point Change in Likelihood of Vaccinationa . | P . | |
Study period | ||||||
Preintervention (referenceb), % | 20.2 | — | 45.6 | — | 42.5 | — |
Intervention | +12.7 | <.001 | +16.2 | <.001 | +8.1 | <.001 |
Postintervention | +27.3 | <.001 | +17.8 | <.001 | +1.1 | .49 |
Sex | ||||||
Female (reference), % | 33.4 | — | 57.6 | — | 47.8 | — |
Male | +3.1 | .013 | +1.6 | .04 | −6.1 | <.001 |
Race | ||||||
Non-Hispanic white (reference), % | 30.2 | — | 52.4 | — | 40.9 | — |
Non-Hispanic black | +2.7 | .08 | +5.5 | <.001 | +4.0 | .005 |
Hispanic | +8.3 | <.001 | +7.9 | <.001 | +5.7 | <.001 |
Other | +2.2 | .18 | +4.2 | .01 | +2.5 | .11 |
Primary language | ||||||
Other (reference), % | 36.3 | — | 59.3 | — | 47.9 | — |
English | −2.7 | .006 | −2.4 | .016 | −6.0 | <.001 |
Insurance | ||||||
Private (reference), % | 32.5 | — | 56.0 | — | 42.9 | — |
Public | +2.8 | .03 | +2.7 | .048 | +2.4 | .05 |
Received meningitis or tetanus vaccines at visit | ||||||
No (reference), % | 34.5 | — | 40.7 | — | 37.3 | — |
Yes | +14.5 | <.001 | +48.0 | <.001 | +40.0 | <.001 |
Calendar time | ||||||
2015 fourth quarter (reference), % | 21.2 | — | 80.4 | — | 61.8 | — |
2016 first quarter | 0.0 | .93 | −2.2 | .23 | −6.0 | .001 |
2016 second quarter | −2.6 | .03 | −6.8 | .001 | −4.1 | .03 |
2016 third quarter | +10.8 | <.001 | −5.6 | .007 | −3.3 | .1 |
2016 fourth quarter | +13.9 | <.001 | −18.1 | <.001 | −8.4 | <.001 |
2017 first quarter | +17.6 | <.001 | −21.2 | <.001 | −14.0 | <.001 |
2017 second quarter | +18.5 | <.001 | −23.3 | <.001 | −18.7 | <.001 |
2017 third quarter | +27.3 | <.001 | −22.8 | <.001 | −18.0 | <.001 |
2017 fourth quarter | +20.5 | <.001 | −32.8 | <.001 | −21.7 | <.001 |
2018 first quarter | +21.9 | <.001 | −37.0 | <.001 | −28.2 | <.001 |
2018 second quarter | +17.7 | <.001 | −41.8 | <.001 | −31.7 | <.001 |
2018 third quarter | +19.8 | <.001 | −41.8 | <.001 | −33.1 | <.001 |
Variable . | Ages 9–10 . | Ages 11–12 . | Ages 13–17 . | |||
---|---|---|---|---|---|---|
Percentage Point Change in Likelihood of Vaccinationa . | P . | Percentage Point Change in Likelihood of Vaccinationa . | P . | Percentage Point Change in Likelihood of Vaccinationa . | P . | |
Study period | ||||||
Preintervention (referenceb), % | 20.2 | — | 45.6 | — | 42.5 | — |
Intervention | +12.7 | <.001 | +16.2 | <.001 | +8.1 | <.001 |
Postintervention | +27.3 | <.001 | +17.8 | <.001 | +1.1 | .49 |
Sex | ||||||
Female (reference), % | 33.4 | — | 57.6 | — | 47.8 | — |
Male | +3.1 | .013 | +1.6 | .04 | −6.1 | <.001 |
Race | ||||||
Non-Hispanic white (reference), % | 30.2 | — | 52.4 | — | 40.9 | — |
Non-Hispanic black | +2.7 | .08 | +5.5 | <.001 | +4.0 | .005 |
Hispanic | +8.3 | <.001 | +7.9 | <.001 | +5.7 | <.001 |
Other | +2.2 | .18 | +4.2 | .01 | +2.5 | .11 |
Primary language | ||||||
Other (reference), % | 36.3 | — | 59.3 | — | 47.9 | — |
English | −2.7 | .006 | −2.4 | .016 | −6.0 | <.001 |
Insurance | ||||||
Private (reference), % | 32.5 | — | 56.0 | — | 42.9 | — |
Public | +2.8 | .03 | +2.7 | .048 | +2.4 | .05 |
Received meningitis or tetanus vaccines at visit | ||||||
No (reference), % | 34.5 | — | 40.7 | — | 37.3 | — |
Yes | +14.5 | <.001 | +48.0 | <.001 | +40.0 | <.001 |
Calendar time | ||||||
2015 fourth quarter (reference), % | 21.2 | — | 80.4 | — | 61.8 | — |
2016 first quarter | 0.0 | .93 | −2.2 | .23 | −6.0 | .001 |
2016 second quarter | −2.6 | .03 | −6.8 | .001 | −4.1 | .03 |
2016 third quarter | +10.8 | <.001 | −5.6 | .007 | −3.3 | .1 |
2016 fourth quarter | +13.9 | <.001 | −18.1 | <.001 | −8.4 | <.001 |
2017 first quarter | +17.6 | <.001 | −21.2 | <.001 | −14.0 | <.001 |
2017 second quarter | +18.5 | <.001 | −23.3 | <.001 | −18.7 | <.001 |
2017 third quarter | +27.3 | <.001 | −22.8 | <.001 | −18.0 | <.001 |
2017 fourth quarter | +20.5 | <.001 | −32.8 | <.001 | −21.7 | <.001 |
2018 first quarter | +21.9 | <.001 | −37.0 | <.001 | −28.2 | <.001 |
2018 second quarter | +17.7 | <.001 | −41.8 | <.001 | −31.7 | <.001 |
2018 third quarter | +19.8 | <.001 | −41.8 | <.001 | −33.1 | <.001 |
Eligible patients are those who had not completed the vaccine series and for whom a sufficient amount of time had elapsed since the previous dose to be eligible for a subsequent dose. Eligibility was adjusted for national vaccine schedule recommendations at date of visit and age at vaccine initiation. —, reference.
Percentage point change refers to an increase or decrease in percentage of adolescents fulfilling criteria (eg, from 20.2% completing series on time preintervention among those aged 9 to 10 years to 20.2% + 27.3%, a total of 47.5% completing the series postintervention).
Reference refers to the value against which other categories of that variable are compared. For example, the preintervention period is the reference value for the study period. Percentage point changes estimate the change in the intervention and postintervention periods compared with the preintervention period.
A secondary outcome was on-time completion of the HPV vaccine series, by using the CDC definition of series completion before the 13th birthday and adjusted for visit date relative to national recommendations for a 2-dose schedule. Adjusted models indicate a 4.3 and 4.7 percentage point increase in the likelihood of on-time completion in both the intervention and postintervention periods, respectively, compared with the preintervention period (P < .001; Table 3). Male sex and Hispanic ethnicity were also positively associated with on-time series completion. Over time, the likelihood of an adolescent aged 12 to 13 completing vaccination at or before their eligible clinic visit increased (P < .05 for the third quarter of 2016 onward compared with the fourth quarter of 2015; Table 3).
Change in Likelihood of Adolescent Completing Vaccine Series by the 13th Birthday During Intervention and Postintervention Periods Compared With Preintervention Period
Variable . | Percentage Point Change in Likelihood of Vaccinationa . | P . |
---|---|---|
Study period | ||
Preintervention (referenceb), % | 61.1 | — |
Intervention | +4.3 | <.001 |
Postintervention | +4.7 | <.001 |
Sex | ||
Female (reference), % | 62.7 | — |
Male | +4.0 | <.001 |
Race | ||
Non-Hispanic white (reference), % | 57.4 | — |
Non-Hispanic black | 0.0 | .93 |
Hispanic | +12.1 | <.001 |
Other | +1.6 | .50 |
Primary language | ||
Other (reference), % | 65.1 | — |
English | +0.3 | .79 |
Insurance | ||
Private (reference) | 64.3 | — |
Public | −0.6 | .71 |
Calendar time | ||
2015 fourth quarter (reference), % | 59.4 | — |
2016 first quarter | +2.7 | .03 |
2016 second quarter | +2.0 | .17 |
2016 third quarter | +4.8 | .001 |
2016 fourth quarter | +4.6 | .002 |
2017 first quarter | +6.1 | <.001 |
2017 second quarter | +7.2 | <.001 |
2017 third quarter | +10.3 | <.001 |
2017 fourth quarter | +9.0 | <.001 |
2018 first quarter | +11.5 | <.001 |
2018 second quarter | +12.1 | <.001 |
2018 third quarter | +16.7 | <.001 |
Variable . | Percentage Point Change in Likelihood of Vaccinationa . | P . |
---|---|---|
Study period | ||
Preintervention (referenceb), % | 61.1 | — |
Intervention | +4.3 | <.001 |
Postintervention | +4.7 | <.001 |
Sex | ||
Female (reference), % | 62.7 | — |
Male | +4.0 | <.001 |
Race | ||
Non-Hispanic white (reference), % | 57.4 | — |
Non-Hispanic black | 0.0 | .93 |
Hispanic | +12.1 | <.001 |
Other | +1.6 | .50 |
Primary language | ||
Other (reference), % | 65.1 | — |
English | +0.3 | .79 |
Insurance | ||
Private (reference) | 64.3 | — |
Public | −0.6 | .71 |
Calendar time | ||
2015 fourth quarter (reference), % | 59.4 | — |
2016 first quarter | +2.7 | .03 |
2016 second quarter | +2.0 | .17 |
2016 third quarter | +4.8 | .001 |
2016 fourth quarter | +4.6 | .002 |
2017 first quarter | +6.1 | <.001 |
2017 second quarter | +7.2 | <.001 |
2017 third quarter | +10.3 | <.001 |
2017 fourth quarter | +9.0 | <.001 |
2018 first quarter | +11.5 | <.001 |
2018 second quarter | +12.1 | <.001 |
2018 third quarter | +16.7 | <.001 |
Eligible adolescents are those aged 12 to 13 years on visit date; definition of completion is adjusted for national vaccine schedule recommendations on visit date (eg, 3 doses required to be considered complete for visits occurring before December 16, 2016). —, not applicable.
Percentage point change refers to increase or decrease in percentage of adolescents fulfilling criteria (eg, from 61.1% completing series by the 13th birthday to 61.1% + 4.7%, a total of 65.8%, postintervention).
Reference refers to the value against which other categories of that variable are compared. For example, the preintervention period is the reference value for the study period. Percentage point changes estimate the change in the intervention and postintervention periods compared with the preintervention period.
Discussion
This stepped-wedge randomized trial of DOSE HPV, a multilevel intervention, was shown to improve vaccination rates during the intervention period, and improvements were sustained during follow-up periods ranging from 6 to 18 months. Our findings contribute to the body of evidence indicating that multilevel interventions appear promising as a means of raising vaccination rates.19–23 Because multilevel interventions require substantial investments of personnel and time in the short-term, demonstrating that intervention effects continue in the postintervention period is important when clinical and policy decision-makers consider upfront costs.
Maintaining gains in vaccination rates may depend on the sustainability of intervention components.32 The DOSE HPV intervention activates providers to create tailored systems changes, which may allow improvements in HPV vaccination provision to be sustained after intervention completion. Health intervention literature indicates that systems-level changes require fewer ongoing resources and can be more sustainable than provider-focused efforts, especially as participating providers may leave the practice after intervention completion.33,34 However, research also demonstrates benefits to educating providers35 as well as developing interventions in partnership with stakeholders to ensure relevance and capacity.33,34 Viewing the DOSE HPV intervention through this lens, educational sessions and communications training may have activated providers in the short-term, and allowing participants to develop tailored systems changes to address barriers may have promoted sustainability by building engagement and aligning efforts with existing clinical processes.32,36,37 System-wide efforts to routinely recommend HPV vaccination at younger ages were considered to be easy and successful changes by providers38 and appeared to be a key factor underlying population-level gains over time. Initiating vaccination at age 9 or 10 can increase the number of opportunities to complete vaccination on time.39 High rates of vaccination among 9- and 10-year-olds in the postintervention period indicated that early vaccination was acceptable to parents. Providers attributed high parental acceptance of early vaccination to a preference for fewer shots per visit as well as an attenuated connection between vaccination and sexual activity at younger ages.38 The ability of sites to implement and sustain this change is demonstrated by increases in early and on-time vaccine series initiation and completion that persisted through the postintervention periods. Researchers of other studies have also demonstrated that initiating vaccination before age 11 years can increase on-time vaccination rates; therefore, this may be a crucial component of future interventions.40,41
This study has several limitations, including a limited geographic sampling area and implementation in clinics serving primarily low-income, minority, urban populations. Although the clinical sites were all resource constrained, they did have relatively high HPV vaccine initiation rates at baseline (75%), indicating that providers and parents were already accustomed to vaccinating. Two of our sites transitioned to a new electronic medical record system, limiting our ability to identify patients who might receive primary care at the intervention sites but who had not presented for care in >12 months; this in turn could lead to overestimation of population-level vaccination rates. Also, we could not identify immunocompromised adolescents who required a 3-dose schedule.
HPV vaccination rates continued to increase nationally during the study period, likely facilitated by the release of the 2-dose schedule in December 2016. Although the stepped-wedge design is intended to control for secular changes, fully disentangling intervention effects from contextual factors is difficult. To be able to statistically estimate the effect of the intervention from that of the 2-dose schedule, we would need to separately estimate the DOSE HPV effect among those on 3-dose and 2-dose schedules separately. Because our intervention focused primarily on younger adolescents, all of whom were affected by the 2-dose schedule switch, we cannot perform this type of stratified analysis and primarily use the stepped-wedge design to assess for secular trends. The regression models indicate that the intervention increased the likelihood of vaccine receipt and series completion after controlling for calendar time. Despite these limitations, this trial is one of the first to indicate effectiveness of a multilevel intervention leading to improvements in HPV vaccination rates 6 to 18 months beyond completion of the intervention. If long-term gains are replicated in other populations, these types of interventions could be uniquely useful tools to improve HPV vaccination rates in the United States.
Conclusions
Multilevel interventions that include interprofessional provider education, data feedback, and tailored systems changes that focus on initiating HPV vaccination before age 11 may be successful in improving HPV vaccine series initiation and completion beyond the conclusion of the intervention period. Replication of these findings in settings with lower baseline vaccination rates, in other geographic regions, and among rural populations and those with private insurance is an important area for future research.
Acknowledgments
We thank the providers and clinic staff who participated in the intervention.
Dr Perkins conceptualized, designed, and delivered component interventions of the study, supervised data collection, drafted the initial manuscript, and revised and reviewed the manuscript; Dr Bernstein conceptualized the study, delivered core curricular components on motivational interviewing, and reviewed the manuscript; Drs Joseph, Adams, Clark, Fenton, and Drainoni conceptualized and designed the study and critically reviewed and revised the manuscript; Dr Hanchate and Mr Legler conceptualized and designed the study methodology and statistical analyses and conducted all data analyses; Drs Schuch and Leschly oversaw implementation of interventions and critically reviewed and revised the manuscript; Ms Eun and Ms Biancarelli critically reviewed and revised the manuscript; Ms Jansen delivered component interventions of the study, performed data analysis during active intervention, and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
This trial has been registered at www.clinicaltrials.gov (identifier NCT02812732).
FUNDING: Supported by an American Cancer Society Research Scholar grant (grant 128607-RSG-15-150-01-CPHPS). No commercial support was obtained.
References
Competing Interests
POTENTIAL CONFLICT OF INTEREST: Dr Joseph has received grant funding from Merck; the other authors have indicated they have no potential conflicts of interest to disclose.
FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.
Comments