BACKGROUND AND OBJECTIVES

Nationally, 54.2% of youth are fully vaccinated for human papilloma virus (HPV) with persistent gender and racial/ethnic disparities. We used a quality improvement approach to improve completion of the HPV vaccine series by age 13 years. As a secondary aim, we examined racial/ethnic and gender differences in vaccine uptake.

METHODS

The study setting included 2 pediatric, academic, primary care practices in Massachusetts. We designed a multilevel patient-, provider-, and systems-level intervention addressing parental hesitancy, provider communication, and clinical operations. Rates of HPV series completion by age 13 were monitored using a control p chart. Bivariate and multivariate analyses evaluated vaccine completion differences on the basis of clinic size, gender, and race/ethnicity.

RESULTS

Between July 1, 2014, and September 30, 2021, control p charts showed special cause variation with HPV vaccine initiation by age 9 years, increasing from 1% to 52%, and vaccine completion by 13 years, increasing from 37% to 77%. Compared with White and Black children, Hispanic children were more likely to initiate the HPV vaccine at age 9 (adjusted odds ratio [95% confidence interval] = (1.4–2.6)] and complete the series by age 13 (adjusted odds ratio [95% confidence interval] = 2.3 (1.7–3.0).

CONCLUSIONS

A multilevel intervention was associated with sustained HPV vaccine series completion by age 13 years. Hispanic children were more likely to be vaccinated. Qualitative family input was critical to intervention design. Provider communication training addressed vaccine hesitancy. Initiation of the vaccine at age 9 and clinicwide vaccine protocols were key to sustaining improvements.

What’s Known on the Subject

Uptake of the human papilloma virus (HPV) vaccine has risen nationally but remains suboptimal. Multilevel interventions that impact patients, providers, and clinic systems show promise. There are significant racial/ethnic disparities. The impact of the coronavirus disease 2019 pandemic on HPV vaccination rates is unknown.

What this Study Adds

This study shows effects of a multilevel intervention to improve HPV vaccination completion rates for 13-year-old girls and boys, addresses sex and race/ethnicity disparities, and sustains changes over 7 years, including the first 18 months of the coronavirus disease 2019 pandemic.

Human Papillomavirus (HPV) remains the most common sexually transmitted infection in the United States.1  In 2018, the estimated incidence for HPV infection for 15 to 24-year-old youth was 7.1 million.2,3  HPV-associated cancers are higher in both non-Hispanic Black and Hispanic adults.3  The HPV vaccine was licensed in 2006 with a 3-dose schedule for females and in 2011 for males.47  A 2-dose schedule was recommended for children aged 9 to 14 years in 2016.6  Since 2006, HPV infections in adolescent females significantly decreased.710  By 2020, only 64.5% of United States youth aged 13 to 17 years and 48.4% of 13-year-olds were fully vaccinated for HPV.11  Black and Hispanic children were initially less likely to receive the HPV vaccine, but this disparity shifted by 2019 because Hispanic children had higher rates of HPV vaccine completion (58.1%) than Black children (54.3%) and White children (51.6%).11,12  Sex differences persisted with females more fully vaccinated for HPV than males (56.8% vs 51.8%).11 

Previous primary care interventions to increase HPV vaccine uptake focused on providers, parents, systems improvements, and community outreach.13,14  Educational interventions and social marketing show varied success.1517  Vaccine reminders with phone calls, letters, or texts increased receipt of vaccine.1820  Successful provider interventions included education, performance feedback, and electronic decision support.17,2025  Communication training emphasizes strong clear vaccine recommendations.2628  Clinic systems-based interventions, including provider prompts, and initiating the HPV vaccine at age 9 rather than age 11 are effective and cost-effective.4,27,2931  Multilevel interventions that build on previous research, combining improvement initiatives focused on families, providers, and clinic systems, show promise.3234 

Although HPV vaccination rates are increasing, they remain suboptimal. Few studies have applied multilevel interventions and measured improvements in HPV vaccination completion for both girls and boys over a sustained period. We sought to use a comprehensive group of quality improvement (QI) interventions to improve HPV vaccine completion for a diverse population of girls and boys. Interventions were targeted at improving family acceptance of the vaccine, provider knowledge and communication skills, and clinic systems. As a primary aim, we evaluated, over 7 years, the cumulative effects and sustainability of this intervention, aiming to improve the rate of HPV vaccine completion by age 13 years for girls and boys at 2 urban primary care practices. As a secondary aim, we examined racial, ethnic, and sex differences in vaccine uptake. We believe study findings may suggest strategies for improving uptake of all vaccines, including coronavirus disease 2019 (COVID-19).

The setting included an academic, hospital-based clinic and a community health center in Massachusetts. The hospital-based clinic serves 16 000 patients aged 0 to 21 years, 75% with Medicaid, with a staff of 29 pediatricians, 65 pediatric residents, 4 nurse practitioners, 20 nurses, 1 QI consultant (QIC), and 33 administrative staff. The community health center, nonfederally qualified, serves 6000 patients aged 0 to 25 years, 85% with Medicaid, with a staff of 6 pediatricians, 2 nurse practitioners, 12 pediatric resident physicians, 9 nurses, 1 QIC, and 15 administrative staff. Both practices have a majority Black and Hispanic patient population. They share clinical leadership, electronic health record (EHR), and have undergone practice transformation using the Model for Improvement,35  improving patient continuity, staff engagement, care teams, and QI processes.

Our objective was to test evidence-based strategies for improving HPV vaccine completion. The improvement team included 6 physicians, 1 departmental QI director, 1 nurse manager, 2 QICs, and 4 evaluators. Key drivers and improvement initiatives were identified using literature review and by conducting focus groups and semistructured interviews with nonresident providers, parents/caregivers, and adolescents at both practices.13,15,16,18,36  From March 2014 to June 2014, the team used this qualitative information to identify individual-, provider-, and systems-level barriers. Individual-level themes included family vaccine mistrust, feelings that their child’s HPV risk was low, parents’ need for more information, and reluctance to return for series completion. Provider-level themes included discomfort discussing the HPV vaccine given its association with sexual activity and lack of time. Systems-level barriers were challenges with the dosage schedule and administration of multiple vaccines at the same time, especially at age 11 when other vaccines were required.37 

Using qualitative data and literature review, the team developed process maps clarifying steps of HPV vaccination delivery, highlighting gaps in processes and vaccination barriers. The team proposed potential solutions. In a series of meetings, a final set of intervention components was identified on the basis of feasibility, acceptability, and potential impact, and a key driver diagram (Fig 1) was constructed. The institutional review board at Boston Children’s Hospital approved the study.

FIGURE 1

Key driver diagram.

FIGURE 1

Key driver diagram.

Close modal

The QICs maintained the patient-tracking registry, created run charts for staff feedback, and developed statistical control charts. The team collectively planned improvement efforts, obtained practice engagement, and tracked tests of change. Table 1 summarizes intervention components, implemented between July 1, 2014, and June 30, 2016, while the 3-dose HPV schedule was required. The clinics used the 2-dose schedule in September 2016, when the intervention entered a postintervention phase. During the pandemic phase, well-child care and vaccine access were limited as the clinics shifted to telemedicine.

TABLE 1

Summary of Clinic System, Provider and Patient Intervention Components

ObjectiveIntervention ComponentImplementation Date
Level: Clinic System 
 Increase access Monthly evening vaccine clinic July 2014 
 Decrease parent barriers Start vaccine series at age 9 July 2014 
 Decrease provider barriers Clinic-wide protocol for HPV vaccine series in recommended talking points; focus starting series at age 9 y July 2014 
 Follow-up visit improvement Clinic flow: standing orders (nurses administer second and third doses without written physician orders) September 2014 
  Scheduling: reminder cards with dosing schedule; EHR vaccine order set for subsequent doses November 2014 
  Attendance: automated patient tracking reports; nonautomated patient reminder calls; phone calls for missed appointments January 2015 
 Track progress Implement quality measure; report data to clinic regarding progress November 2014 
 Improve access for HPV vaccination Urgent care patients HPV status checked; administrative outreach for doses 2 and 3 August 2015 
 Track HPV gaps HPV added to preventive bundle report November 2015 
 CDC recommends change to 2-dose schedule Clinics revised measures for 2 dose recommendations (new goals): October 2016 
  1 HPV dose at age 9, second dose age 10 January 2017 
 Pay for performance Medicaid Accountable Care Organization includes HPV in quality measure portfolio. January 2018 
Level: provider and nurse 
 Objective Intervention component Implementation date 
 Identify barriers and assess knowledge Provider, staff and parent/adolescent focus groups May 2014 
 Educate providers decrease parent barriers Training on importance of HPV vaccine May and July 2014 
  Use of standardized recommendation language July 2014 
  Motivational Interview training for providers  
Level: patient   
 Objective Intervention component Implementation date 
 Increase knowledge and empower parents Visual reinforcement campaign with informational handouts in English and Spanish, posters in waiting and exam rooms September 2014 
ObjectiveIntervention ComponentImplementation Date
Level: Clinic System 
 Increase access Monthly evening vaccine clinic July 2014 
 Decrease parent barriers Start vaccine series at age 9 July 2014 
 Decrease provider barriers Clinic-wide protocol for HPV vaccine series in recommended talking points; focus starting series at age 9 y July 2014 
 Follow-up visit improvement Clinic flow: standing orders (nurses administer second and third doses without written physician orders) September 2014 
  Scheduling: reminder cards with dosing schedule; EHR vaccine order set for subsequent doses November 2014 
  Attendance: automated patient tracking reports; nonautomated patient reminder calls; phone calls for missed appointments January 2015 
 Track progress Implement quality measure; report data to clinic regarding progress November 2014 
 Improve access for HPV vaccination Urgent care patients HPV status checked; administrative outreach for doses 2 and 3 August 2015 
 Track HPV gaps HPV added to preventive bundle report November 2015 
 CDC recommends change to 2-dose schedule Clinics revised measures for 2 dose recommendations (new goals): October 2016 
  1 HPV dose at age 9, second dose age 10 January 2017 
 Pay for performance Medicaid Accountable Care Organization includes HPV in quality measure portfolio. January 2018 
Level: provider and nurse 
 Objective Intervention component Implementation date 
 Identify barriers and assess knowledge Provider, staff and parent/adolescent focus groups May 2014 
 Educate providers decrease parent barriers Training on importance of HPV vaccine May and July 2014 
  Use of standardized recommendation language July 2014 
  Motivational Interview training for providers  
Level: patient   
 Objective Intervention component Implementation date 
 Increase knowledge and empower parents Visual reinforcement campaign with informational handouts in English and Spanish, posters in waiting and exam rooms September 2014 

At the systems level, we developed initiatives to improve both efficiency of vaccine administration and vaccine access. A nursing protocol allowed nurses to recommend and administer the vaccine before providers entered the room. The clinics increased vaccine access by creating a monthly evening shot clinic. To decrease parent barriers related to number of vaccines typically administered at ages 11 to 12, we standardized initiating the HPV vaccine at age 9. We developed reminder-based prompting by programming EHR checkboxes for ordering subsequent doses of the vaccine, gave patients dosing schedule cards with the suggested dates of future doses, and used a population management system for tracking, reminding, and rescheduling patients who needed the vaccine. We established standing orders for nurse administration of all HPV vaccine doses.

At the provider-level, we focused on improving providers’ communication skills, including standardized HPV vaccine recommendations for provider (medical doctor or nurse practitioner) or nurse “announcement” language, such as, “Your child is due for the HPV vaccine today,” rather than “Would you like your child to receive the HPV vaccine today?”27  All providers received a 45-minute training on motivation interviewing (MI), a nonconfrontational, nonjudgmental counseling style that explores and addresses ambivalence to overcome barriers to behavior change.38  The MI training encouraged provider reflection on their own concerns about HPV vaccination while increasing their confidence for engaging in difficult discussions with parents. At the patient/parent level, we posted informational posters in English and Spanish throughout the clinic, including on the back of examination room doors (for patients and parents to read while they waited), and developed informational handouts (Appendix 1 in Supplemental Information). Residents received education from their attending physicians. Vaccine rates were reviewed quarterly on run charts and distributed to the practices through monthly QI newsletters.

Methods of Evaluation

We compiled a data set from the EHR that contained records for every clinic patient who had any clinic visit during the previous year from the start of the intervention rollout period, July 1, 2014, to September 30, 2021.The data set contained dates for every HPV vaccine given in both practices. Doses of HPV vaccine obtained elsewhere were entered into the EHR manually or through the Massachusetts Immunization Information System, which includes vaccines given in all Massachusetts immunization sites.39  The data set included patient sociodemographic characteristics (birthdate, sex, race/ethnicity, and insurance status). Parents self-reported their child’s race and ethnicity data at registration for entry into standardized EHR fields.

Quality Measure

We assessed quarterly (3-month) rates of receiving 1 dose at age 9 and completion of 2 doses of HPV at age 13 for females, males, and both sexes combined. We used a visit within the last year to define the population. The denominator for each quarter was the number of patients with a birthday during that quarter who had 1 or more primary care visits within the last year in relation to the reporting quarter, and the numerator was the number of these patients who had received at least 1 dose (or all doses) of HPV vaccine. For example, in the 13th year analysis, the denominator of this measure is the number of patients who reached age 14 years during the reporting quarter with at least 1 visit during the previous 12 months. The numerator was the number of these patients who received all doses of HPV vaccine before age 14.

In September 2015 and February 2016, the team conducted 45-minute feedback sessions with clinic leadership and staff (n = 16 across all sessions) to assess the perceived acceptability (ie, positive and negative perceptions) of each intervention component. The sessions were audiotaped and transcribed; 2 independent coders used inductive content analysis to group data into themes.40 

Analysis

We tracked the effects of our intervention over time using process (p) charts with control limits set at 3 α.41  Using ChartRunner expl Lean (version 3.0), we created charts for patients aged 9 and 13 years. We established a mean completion rate for each p-chart, illustrated as the center line. We established upper control limits and lower control limits. Special cause variation, defined by the standard control limits built into the Statistical quality Control chart software as 7 points beyond the center line, signaled an effect and a system change. All other statistical analyses used SAS version 9.4 software. χ2 tests were performed to examine relations between study time period and covariates (clinic size, race/ethnicity, sex). Logistic regression was used to determine which characteristics were associated with HPV vaccine adherence. For each outcome (1 dose at age 9 and series completion by age 13), unadjusted logistic regression models were used to examine the independent associations between study period, clinic size, sex, and race/ethnicity with HPV vaccine adherence. Additionally, for each outcome, a multivariate, logistic regression model adjusted for study period, clinic size, sex, and race/ethnicity was used to analyze the effects of these covariates on adherence.

We identified 6779 children aged 10 years and 5491 children aged 14 years who had attended any visit between July 1, 2014, and March 30, 2020 (Table 2). Nearly half of the population was female (49%), with 38% non-Hispanic Black and 45% Hispanic. Most had public insurance (70%).

TABLE 2

Population Demographics from July 2014 to March 2020

Patient Aged 9 YearsPatient Aged 13 Years
N%N%
Large clinic 5137 75.8 3968 72.3 
Small clinic 1642 24.2 1523 27.7 
Female 3277 48.3 2724 49.6 
Male 3502 51.7 2767 50.4 
Asian, non-Hispanic 135 2.0 95 1.7 
Black, non-Hispanic 2309 34.1 2060 37.5 
Hispanic 3063 45.2 2463 44.8 
White, non-Hispanic 298 4.4 225 4.1 
Other, non-Hispanic 569 8.4 387 7.1 
Multiracial, non-Hispanic 89 1.3 45 0.8 
Unknown race/ethnicity 316 4.7 216 3.9 
Patient Aged 9 YearsPatient Aged 13 Years
N%N%
Large clinic 5137 75.8 3968 72.3 
Small clinic 1642 24.2 1523 27.7 
Female 3277 48.3 2724 49.6 
Male 3502 51.7 2767 50.4 
Asian, non-Hispanic 135 2.0 95 1.7 
Black, non-Hispanic 2309 34.1 2060 37.5 
Hispanic 3063 45.2 2463 44.8 
White, non-Hispanic 298 4.4 225 4.1 
Other, non-Hispanic 569 8.4 387 7.1 
Multiracial, non-Hispanic 89 1.3 45 0.8 
Unknown race/ethnicity 316 4.7 216 3.9 

Figures 2 and 3 show p-charts for children receiving the first dose of HPV vaccine at age 9 and 2 doses of HPV vaccine by age 13 between July 1, 2014, and September 30, 2021. For all patients, receipt of the first HPV vaccine dose by age 9 increased from 1.1% to 51.8% but fell to 22.1% after March 30, 2020. Completion of the HPV vaccine series by age 13 increased from 37.4% to 77.2% between 2014 and 2021. Special cause variation was achieved by March 30, 2015, and again in September 30, 2016, signaling the postintervention phase.

FIGURE 2

Control chart for male and female receipt of HPV dose 1 at age 9 years.

FIGURE 2

Control chart for male and female receipt of HPV dose 1 at age 9 years.

Close modal
FIGURE 3

Control chart for female and male completion of HPV 2-dose series by age 13 years.

FIGURE 3

Control chart for female and male completion of HPV 2-dose series by age 13 years.

Close modal

Tables 3 and 4 summarizes bivariate analysis for 1 dose of HPV at age 9 and age 13 completion of the HPV series by clinic size, sex, and race/ethnicity at baseline and postintervention. Table 5 summarizes the multivariate analysis. In the postintervention phase, the small clinic was more successful at administering the first dose of HPV vaccine at age 9 years than the larger clinic (64.7% vs 47.5%; P < .001). This small clinic trend continued for completion of the HPV series by age 13 years (85.9% vs 73.6%; P < .001) and persisted in multivariate analysis (P < .001) at both age 9 and 13. By age 13, both bivariate and multivariate analysis showed significant sex differences at baseline, with females more likely to initiate the vaccine (58.2% vs 16.8%; P < .001). These differences disappeared in postintervention (female 77.7% versus male 76.2%; P = .38).

TABLE 3

Bivariate Analysis for Receipt of First Dose of HPV Vaccine at Age 9 Years

Baseline Period (2013 Q1–2014 Q1), n = 1114Postintervention Period (2017 Q1–2020 Q1), n = 3118Postintervention Versus BaselineDifference in Proportion to BaselineDifference in Proportion to Postintervention
Received First HPV Immunization at 9 years Old% (95% CI)% (95% CI)PP
Overall 1.1 (0.6–1.9) 51.8 (50.0–53.5) <.001 — — 
Clinic 
 Large 1.1 (0.5–2.0) 47.5 (45.4–49.5) <.001 .99 <.001 
 Small 1.1 (0.2–3.3) 64.7 (61.3–68.1) <.001 
Sex  
 Female 1.4 (0.6, 2.7) 51.9 (49.3–54.4) <.001 .29 .91 
 Male 0.7 (0.2–1.9) 51.7 (49.2–54.1) <.001 
Race/ethnicity 
 Asian American, non-Hispanic 0.0 (0.0–14.9) 56.1 (43.3–68.3) <.001 .25 <.001 
 Black, non-Hispanic 0.3 (0.0–1.5) 46.0 (42.9–49.1) <.001 
 Hispanic 1.2 (0.2–2.1) 58.5 (56.0–61.1) <.001 
 White, non-Hispanic 3.1 (0.0–7.3) 40.6 (32.3–49.0) <.001 
 Other, non-Hispanic 2.7 (0.33–9.3) 46.1 (40.2–52.1) <.001 
 Multiracial, non-Hispanic 0.0 (0.0–20.4) 54.9 (40.3–68.9) <.001 
 Unknown 2.0 (0.1–10.9) 44.8 (36.4–53.3) <.001 
Baseline Period (2013 Q1–2014 Q1), n = 1114Postintervention Period (2017 Q1–2020 Q1), n = 3118Postintervention Versus BaselineDifference in Proportion to BaselineDifference in Proportion to Postintervention
Received First HPV Immunization at 9 years Old% (95% CI)% (95% CI)PP
Overall 1.1 (0.6–1.9) 51.8 (50.0–53.5) <.001 — — 
Clinic 
 Large 1.1 (0.5–2.0) 47.5 (45.4–49.5) <.001 .99 <.001 
 Small 1.1 (0.2–3.3) 64.7 (61.3–68.1) <.001 
Sex  
 Female 1.4 (0.6, 2.7) 51.9 (49.3–54.4) <.001 .29 .91 
 Male 0.7 (0.2–1.9) 51.7 (49.2–54.1) <.001 
Race/ethnicity 
 Asian American, non-Hispanic 0.0 (0.0–14.9) 56.1 (43.3–68.3) <.001 .25 <.001 
 Black, non-Hispanic 0.3 (0.0–1.5) 46.0 (42.9–49.1) <.001 
 Hispanic 1.2 (0.2–2.1) 58.5 (56.0–61.1) <.001 
 White, non-Hispanic 3.1 (0.0–7.3) 40.6 (32.3–49.0) <.001 
 Other, non-Hispanic 2.7 (0.33–9.3) 46.1 (40.2–52.1) <.001 
 Multiracial, non-Hispanic 0.0 (0.0–20.4) 54.9 (40.3–68.9) <.001 
 Unknown 2.0 (0.1–10.9) 44.8 (36.4–53.3) <.001 
TABLE 4

Bivariate Analysis for Completion of HPV Series by Age 13 Years

Baseline period (2013 Q1–2014 Q1), n = 875Maintenance period (2017 Q1–2020 Q1), n = 2529Maintenance versus baselineDifference in proportion–baselineDifference in proportion–maintenance
HPV series complete at 13 y old% (95% CI)% (95% CI)PP
Overall 37.4 (34.2–40.6) 76.95 (75.3–78.6) <.001 — — 
Clinic 
 Large 30.5 (26.6–34.2) 73.6 (71.6–75.6) <.001 <.001 <.001 
 Small 55.4 (48.9–61.7) 85.86 (83.0–88.4) <.001   
Sex 
 Female 58.2 (53.4–62.8) 77.7 (75.3–79.9) <.001 <.001 .38 
 Male 16.8 (13.4–20.7) 76.20 (73.7–78.5) <.001 
Race/ethnicity 
 Asian American, non-Hispanic 23.1 (5.0–53.8) 73.0 (55.9–86.2) <.001 <.001 <.001 
 Black, non-Hispanic 28.4 (23.7–33.4) 73.11 (70.1–76.0) <.001 
 Hispanic 49.7 (44.7–54.8) 83.5 (81.2–85.6) <.001 
 White, non-Hispanic 25.6 (13.5–41.2) 58.9 (49.0–68.3) <.001 
 Other, non-Hispanic 22.2 (11.2–37.1) 72.5 (65.7–78.7) <.001 
 Multiracial, non-Hispanic 42.9 (9.9–81.6) 60.7 (40.6–78.5) <.001 
 Unknown 23.1 (8.97–43.65) 72.1 (62.8–80.2) <.001 
Baseline period (2013 Q1–2014 Q1), n = 875Maintenance period (2017 Q1–2020 Q1), n = 2529Maintenance versus baselineDifference in proportion–baselineDifference in proportion–maintenance
HPV series complete at 13 y old% (95% CI)% (95% CI)PP
Overall 37.4 (34.2–40.6) 76.95 (75.3–78.6) <.001 — — 
Clinic 
 Large 30.5 (26.6–34.2) 73.6 (71.6–75.6) <.001 <.001 <.001 
 Small 55.4 (48.9–61.7) 85.86 (83.0–88.4) <.001   
Sex 
 Female 58.2 (53.4–62.8) 77.7 (75.3–79.9) <.001 <.001 .38 
 Male 16.8 (13.4–20.7) 76.20 (73.7–78.5) <.001 
Race/ethnicity 
 Asian American, non-Hispanic 23.1 (5.0–53.8) 73.0 (55.9–86.2) <.001 <.001 <.001 
 Black, non-Hispanic 28.4 (23.7–33.4) 73.11 (70.1–76.0) <.001 
 Hispanic 49.7 (44.7–54.8) 83.5 (81.2–85.6) <.001 
 White, non-Hispanic 25.6 (13.5–41.2) 58.9 (49.0–68.3) <.001 
 Other, non-Hispanic 22.2 (11.2–37.1) 72.5 (65.7–78.7) <.001 
 Multiracial, non-Hispanic 42.9 (9.9–81.6) 60.7 (40.6–78.5) <.001 
 Unknown 23.1 (8.97–43.65) 72.1 (62.8–80.2) <.001 

CI, confidence interval.

TABLE 5

Multivariate Analysis for First Dose of HPV Vaccine at Age 9 Years and Completion of HPV Series by Age 13 Years

PredictorOR95% CIPOR95% CIP
UnadjustedAdjusted
First HPV immunization at 9 years old       
 Study period 
  Baseline Reference Reference 
  Intervention 45.5 25.6–80.9 <.0001 47.9 27.0–59.3 <.0001 
  Postintervention 98.5 55.6–174.8 <.0001 105.4 59.3–187.1 <.0001 
 Clinic 
  Large Reference Reference 
  Small 1.9 1.7–2.1 <.0001 1.7 1.5–2.0 <.0001 
 Sex 
  Male Reference Reference 
  Female 0.9–1.1 .92 0.9–1.2 .49 
 Race/ethnicity 
  White, non-Hispanic Reference Reference 
  Asian American, non-Hispanic 1.9 1.2–3.0 .2 1.9 1.2–3.1 .11 
  Black, non-Hispanic 1.4 1.1–1.9 .07 1.4 1.0–1.8 .32 
  Hispanic 2.3 1.7–3.0 <.0001 1.9 1.4–2.6 <.0001 
  Multiracial, non-Hispanic 1.8 1.2–3.0 .47 1.6 0.9–1.8 .73 
  Other, non-Hispanic 1.5 1.1–2.1 .61 1.3 0.9–2.0 .22 
  Unknown 1.4 1.0–2.1 .44 1.4 0.9–2.0 .55 
HPV series complete at 13 y old 
 Study period 
  Baseline Reference Reference 
  Intervention 3.1 2.6–3.6 <.0001 3.4 2.8–4.0 <.0001 
  Maintenance 5.6 4.7–6.6 <.0001 6.4 5.4–7.6 <.0001 
 Clinic 
  Large Reference Reference 
  Small 2.5 2.2–2.9 <.0001 2.1 1.8–2.5 <.0001 
 Sex 
  Male Reference Reference 
  Female 1.7 1.5–1.9 <.0001 1.8 1.6–2.0 <.0001 
 Race/ethnicity 
  White, non-Hispanic Reference Reference 
  Asian American, non-Hispanic 1.1 0.7–1.8 .23 0.6–1.7 .41 
  Black, non-Hispanic 1.4 1.1–1.9 .78 1.4 1.0–1.9 .11 
  Hispanic 2.8 2.2–3.8 <.0001 2.3 1.7–3.0 <.0001 
  Multiracial, non-Hispanic 1.1 0.6–2.1 .36 0.9 0.4–1.7 .21 
  Other, non-Hispanic 1.4 1.0–2.0 .75 1.2 0.9–1.8 .99  
  Unknown 1.4 1.0–2.0 .97 1.2 0.8–1.8 .92 
PredictorOR95% CIPOR95% CIP
UnadjustedAdjusted
First HPV immunization at 9 years old       
 Study period 
  Baseline Reference Reference 
  Intervention 45.5 25.6–80.9 <.0001 47.9 27.0–59.3 <.0001 
  Postintervention 98.5 55.6–174.8 <.0001 105.4 59.3–187.1 <.0001 
 Clinic 
  Large Reference Reference 
  Small 1.9 1.7–2.1 <.0001 1.7 1.5–2.0 <.0001 
 Sex 
  Male Reference Reference 
  Female 0.9–1.1 .92 0.9–1.2 .49 
 Race/ethnicity 
  White, non-Hispanic Reference Reference 
  Asian American, non-Hispanic 1.9 1.2–3.0 .2 1.9 1.2–3.1 .11 
  Black, non-Hispanic 1.4 1.1–1.9 .07 1.4 1.0–1.8 .32 
  Hispanic 2.3 1.7–3.0 <.0001 1.9 1.4–2.6 <.0001 
  Multiracial, non-Hispanic 1.8 1.2–3.0 .47 1.6 0.9–1.8 .73 
  Other, non-Hispanic 1.5 1.1–2.1 .61 1.3 0.9–2.0 .22 
  Unknown 1.4 1.0–2.1 .44 1.4 0.9–2.0 .55 
HPV series complete at 13 y old 
 Study period 
  Baseline Reference Reference 
  Intervention 3.1 2.6–3.6 <.0001 3.4 2.8–4.0 <.0001 
  Maintenance 5.6 4.7–6.6 <.0001 6.4 5.4–7.6 <.0001 
 Clinic 
  Large Reference Reference 
  Small 2.5 2.2–2.9 <.0001 2.1 1.8–2.5 <.0001 
 Sex 
  Male Reference Reference 
  Female 1.7 1.5–1.9 <.0001 1.8 1.6–2.0 <.0001 
 Race/ethnicity 
  White, non-Hispanic Reference Reference 
  Asian American, non-Hispanic 1.1 0.7–1.8 .23 0.6–1.7 .41 
  Black, non-Hispanic 1.4 1.1–1.9 .78 1.4 1.0–1.9 .11 
  Hispanic 2.8 2.2–3.8 <.0001 2.3 1.7–3.0 <.0001 
  Multiracial, non-Hispanic 1.1 0.6–2.1 .36 0.9 0.4–1.7 .21 
  Other, non-Hispanic 1.4 1.0–2.0 .75 1.2 0.9–1.8 .99  
  Unknown 1.4 1.0–2.0 .97 1.2 0.8–1.8 .92 

CI, confidence interval; OR, odds ratio.

During postintervention for both initiation at age 9 and series completion by age 13, a significant difference among racial/ethnic groups was observed (P < .001). Hispanic children had highest rates of initiation at 9 years (58.5%) and highest rates for completing the series by 13 years in both baseline (49.7%) and postintervention (83.5%) periods. This trend persisted in multivariate analysis adjusting for clinic size, sex, and study period (P < .001).

Staff feedback sessions (Supplemental Table 6) showed positive perceptions toward clinicwide protocols standardizing language recommending the vaccine to patients/parents and beginning the vaccine series at age 9. Staff endorsed standing orders for HPV vaccine. The evening shot clinic and phone reminders had more barriers than benefits, with neither perceived as increasing immunization rates. Providers had mixed feedback for some components with promise but needing improvement. They rated MI training positively, but received insufficient skills to use MI effectively for vaccine refusal. Providers suggested more intensive MI trainings. They noted logistical issues with the appointment reminder system. Using this feedback, we focused on nursing roles with standing orders, use of announcement language, and initiating the vaccine at age 9 in the postintervention period.

In our multilevel intervention, we met our goal to improve rates of HPV vaccination for racially and ethnically diverse girls and boys over a sustained period of time. Our approach included patient-, provider-, and systems-level changes at 2 practices, 1 large and 1 small, to improve completion of a 2-dose HPV vaccine schedule by age 13 years. During the study, the HPV vaccine completion rate at 13 years increased from 37.4% to 77.2%, exceeding Massachusetts rates at 13 to 17 years. Nationally, HPV vaccine completion at 13 years only increased from 22.1% to 48.4% from 2013 to 2020.11  During the first 18 months of the pandemic, HPV vaccine completion rates were maintained even though limitations on clinic well care negatively impacted initiation of the vaccine at age 9. Our intervention trained staff on MI techniques, and streamlined clinical systems.31,36,37,39  Empowering nurses to initiate vaccinations using standard protocols was a key component. We found that the smaller clinic with higher nurse-to-provider ratios achieved greater gains in HPV vaccine rates. Both clinics were comparable because they shared clinic leadership and providers. Uptake of the intervention may have been better at the smaller clinic with fewer staff and trainees. In qualitative feedback, providers were generally positive about the intervention, especially regarding the clinicwide components, but highlighted the need for more intensity of training.

Our multilevel study extended and combined the strengths of previous research in its inclusion of Black and Hispanic girls and boys, as well as children as young as age 9, and we examined a population rather than a convenience sample over an extended period of time into the COVID-19 pandemic. Most previous studies have not studied HPV series completion at age 13. This is important because risk for HPV infection begins in early adolescence. Multilevel studies have used provider communication training, data feedback, systems changes, family education with videos, or other education materials, nurse prompts, and decision aides.21,37,39  In the postintervention, there were no sex disparities in HPV vaccine completion, although, in the United States, males continue to be less likely to complete the HPV series.11 

Our focus on a predominantly Black and Hispanic population is important given racial/ethnic disparities in both HPV infection and rates of HPV vaccination that were prominent at the beginning of our intervention.11,12  Although these disparities narrowed for all patients, we found that Hispanic children improved the most for initiating the vaccine at age 9 and completing the series at age 13, potentially reflecting lower vaccine confidence within other racial/ethnic groups. This aligns with findings in our design phase qualitative study that Black adolescents and their parents were concerned about this vaccine being untested and new.36  Hispanic families may be more accepting of vaccination because of high cervical cancer rates in Hispanic females.3  There may also be neighborhood characteristics that we did not measure influencing HPV vaccine acceptance. This racial/ethnic variability in vaccine acceptance needs further study.

A culture of improvement with accountability, standardization, integration of tasks into daily processes, and data feedback helped sustain our interventions’ improvements.42  The practices’ QI committee continuously reviewed and communicated HPV vaccine completion rates. Qualitative feedback supported focusing on nursing protocols, standardized communication, and vaccine initiation at age 9 in the postintervention period, when vaccine completion continues to rise. Early vaccine initiation explains the sustained HPV completion rates observed during the pandemic, when care shifted significantly to telemedicine.

Limitations of the study include testing the intervention in only 2 clinics in 1 metropolitan area of the United States, lack of a randomized controlled design, and inability to account for the impact of national HPV prevention strategies through media and other modalities on vaccine acceptance. However, our improvements far exceeded national trends. We anticipate that many components could be replicated in other similar practice settings with appropriate consideration for local adaptation. We may have missed identifying doses of HPV given outside of Massachusetts or not entered into the Massachusetts Immunization Information System. Race/ ethnicity data were missing for 4.7% of cases, and it is unknown whether missingness varied by race/ethnicity. There may have been differences in study clinics other than size and staffing ratios that we did not identify. We did not measure individual nurse or provider adherence to the intervention, and did not compare pediatricians versus residents. By using a multilevel intervention, we were unable to identify which aspects were most important. Our intervention was initially designed for a 3-dose HPV schedule, and the Centers for Disease Control and Prevention recommendations changed in October 2016 to a 2-dose schedule, which could be completed at 2 annual visits. To simplify the evaluation, we chose to report only on the 2-dose series. Qualitative feedback suggested that creating nurse protocols was highly effective. Although we also obtained postintervention provider feedback, we were not able to obtain family feedback. Finally, no data were obtained on cost of training and implementation of the intervention.

In conclusion, our study demonstrated increased uptake of the HPV vaccine using a QI approach that addressed individual-, provider-, and systems-level factors. The intervention was feasible, acceptable to providers, and successful at increasing HPV immunization, especially for Hispanic children. Individual intervention initiatives can be applied to improving uptake of other vaccines, including COVID-19. Future research should investigate racial/ethnic differences in vaccine acceptability. Testing the effectiveness of our promising intervention using a clustered randomized design in other communities could facilitate examining neighborhood factors that affect implementation and vaccine adherence. A cost–benefit analysis study may help understand intervention impact on cost and revenue.

Dr Cox participated in the design of the study and intervention, helped to interpret the data, and drafted and revised the manuscript; Dr Bogart conceptualized and designed the study and intervention, helped to interpret the data, and contributed to drafting and revising the manuscript; Dr Elliott helped to design the study and interpret the data, and helped to draft and revise the manuscript; Dr Starmer helped design the intervention, helped with data analysis, and helped to draft and revise the manuscript; Ms Meleedy-Rey analyzed data, created process control charts, and helped to draft and revise the manuscript; Ms Banerjee helped design the study, coordinated the collection of data, helped to interpret the data, and helped to draft and revise the manuscript; Dr Goggin helped to design the intervention, interpret the data, and to draft and revise the final manuscript; Dr Samuels helped to implement the intervention, to interpret the data, and to draft the final manuscript; Mr Hahn analyzed the data and helped to draft and revise the manuscript; Dr Epee-Bounya helped to design the intervention, to interpret the data, and to draft the final manuscript; Dr Allende-Richter helped to implement the intervention, and helped to draft and revise the final manuscript; Ms Fu helped to design and implement the intervention, to interpret the data, and to draft and revise the final manuscript; Dr Schuster conceptualized and designed the study and intervention, helped to interpret the data, and contributed to drafting and revising the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Funded by the National Institute of Minority Health and Health Disparities (supplement to 2 R24 MD001648 MA Schuster principal investigator). Funded by the National Institutes of Health (NIH).

CONFLICT OF INTEREST DISCLAIMER: The authors have indicated they have no conflicts of interest relevant to this article to disclose.

COVID-19

coronavirus disease 2019

EHR

electronic health record

HPV

human papilloma virus

MI

motivational interviewing

p

process

QI

quality improvement

QIC

quality improvement consultant

1
Kreisel
KM
,
Spicknall
IH
,
Gargano
JW
, et al
.
Sexually transmitted infections among US women and men: prevalence and incidence estimates, 2018
.
Sex Transm Dis
.
2021
;
48
(
4
):
208
214
2
McQuillan
G
,
Kruszon-Moran
D
,
Markowitz
LE
,
Unger
ER
,
Paulose-Ram
R
.
Prevalence of HPV in adults aged 18–69: United States, 2011–2014
.
NCHS Data Brief
.
2017
(
280
):
1
8
3
Burger
EA
,
Lee
K
,
Saraiya
M
, et al
.
Racial and ethnic disparities in human papillomavirus-associated cancer burden with first-generation and second-generation human papillomavirus vaccines
.
Cancer
.
2016
;
122
(
13
):
2057
2066
4
Markowitz
LE
,
Dunne
EF
,
Saraiya
M
, et al.
Centers for Disease Control and Prevention (CDC)
.
Human papillomavirus vaccination: recommendations of the Advisory Committee on Immunization Practices (ACIP)
.
MMWR Recomm Rep
.
2014
;
63
(
RR-05 RR-2
):
1
30
5
Centers for Disease Control and Prevention (CDC)
.
FDA licensure of quadrivalent human papillomavirus vaccine (HPV4, Gardasil) for use in males and guidance from the Advisory Committee on Immunization Practices (ACIP)
.
MMWR Morb Mortal Wkly Rep
.
2010
;
59
(
20
):
630
632
6
Meites
E
,
Kempe
A
,
Markowitz
LE
.
Use of a 2-dose schedule for human papillomavirus vaccination — updated recommendations of the Advisory Committee on Immunization Practices
.
MMWR Morb Mortal Wkly Rep
.
2016
;
65
(
49
):
1405
1408
7
Saraiya
M
,
Unger
ER
,
Thompson
TD
, et al
.
US assessment of HPV types in cancers: implications for current and 9-valent HPV vaccines
.
J Natl Cancer Inst
.
2015
;
107
(
6
):
djv086
8
Oliver
SE
,
Unger
ER
,
Lewis
R
, et al
.
Prevalence of human papillomavirus among females after vaccine introduction- national health and Nutrition Examination Survey, United States, 2003–2014
.
J Infect Dis
.
2017
;
216
(
5
):
594
603
9
Drolet
M
,
Bénard
É
,
Boily
MC
, et al
.
Population-level impact and herd effects following human papillomavirus vaccination programmes: a systematic review and meta-analysis
.
Lancet Infect Dis
.
2015
;
15
(
5
):
565
580
10
Guo
F
,
Cofie
LE
,
Berenson
AB
.
Cervical cancer incidence in young US females after human papillomavirus vaccine introduction
.
Am J Prev Med
.
2018
;
55
(
2
):
197
204
11
Pingali
C
,
Yankey
D
,
Elam-Evans
LD
, et al
.
National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years–United States, 2020
.
MMWR Morb Mortal Wkly Rep
.
2021
;
70
(
35
):
1183
1190
12
Hirth
J
.
Disparities in HPV vaccination rates and HPV prevalence in the United States: a review of the literature
.
Hum Vaccin Immunother
.
2019
;
15
(
1
):
146
155
13
Niccolai
LM
,
Hansen
CE
.
Practice- and community-based interventions to increase human papillomavirus vaccine coverage: a systematic review
.
JAMA Pediatr
.
2015
;
169
(
7
):
686
692
14
Walling
EB
,
Dodd
S
,
Bobenhouse
N
,
Reis
EC
,
Sterkel
R
,
Garbutt
J
.
Implementation of strategies to improve human papillomavirus vaccine coverage: a provider survey
.
Am J Prev Med
.
2019
;
56
(
1
):
74
83
15
Cates
JR
,
Diehl
SJ
,
Crandell
JL
,
Coyne-Beasley
T
.
Intervention effects from a social marketing campaign to promote HPV vaccination in preteen boys
.
Vaccine
.
2014
;
32
(
33
):
4171
4178
16
Fu
LY
,
Bonhomme
LA
,
Cooper
SC
,
Joseph
JG
,
Zimet
GD
.
Educational interventions to increase HPV vaccination acceptance: a systematic review
.
Vaccine
.
2014
;
32
(
17
):
1901
1920
17
Walling
EB
,
Benzoni
N
,
Dornfeld
J
, et al
.
Interventions to improve HPV vaccine uptake: a systematic review
.
Pediatrics
.
2016
;
138
(
1
):
e20153863
18
Suh
CA
,
Saville
A
,
Daley
MF
, et al
.
Effectiveness and net cost of reminder/recall for adolescent immunizations
.
Pediatrics
.
2012
;
129
(
6
):
e1437
e1445
19
Jacobson Vann
JC
,
Jacobson
RM
, %
Coyne-Beasley
T
,
Asafu-Adjei
JK
,
Szilagyi
PG
.
Patient reminder and recall interventions to improve immunization rates
.
Cochrane Database Syst Rev
.
2018
;
1
:
CD003941
20
Staras
SA
,
Vadaparampil
ST
,
Livingston
MD
,
Thompson
LA
,
Sanders
AH
,
Shenkman
EA
.
Increasing human papillomavirus vaccine initiation among publicly insured Florida adolescents
.
J Adolesc Health
.
2015
;
56
(
5 Suppl
):
S40
S46
21
Dempsey
AF
,
Pyrznawoski
J
,
Lockhart
S
, et al
.
Effect of a health care professional communication training intervention on adolescent human papillomavirus vaccination a cluster randomized clinical trial
.
JAMA Pediatr
.
2018
;
172
(
5
):
e180016
22
Gilkey
MB
,
Moss
JL
,
Coyne-Beasley
T
,
Hall
ME
,
Shah
PD
,
Brewer
NT
.
Physician communication about adolescent vaccination: how is human papillomavirus vaccine different?
Prev Med
.
2015
;
77
:
181
185
23
Gilkey
MB
,
McRee
AL
.
Provider communication about HPV vaccination: a systematic review
.
Hum Vaccin Immunother
.
2016
;
12
(
6
):
1454
1468
24
Shay
LA
,
Baldwin
AS
,
Betts
AC
, et al
.
Parent–provider communication of HPV vaccine hesitancy
.
Pediatrics
.
2018
;
141
(
6
):
e20172312
25
Szilagyi
PG
,
Serwint
JR
,
Humiston
SG
, et al
.
Effect of provider prompts on adolescent immunization rates: a randomized trial
.
Acad Pediatr
.
2015
;
15
(
2
):
149
157
26
Rand
CM
,
Schaffer
SJ
,
Dhepyasuwan
N
, et al
.
Provider communication, prompts, and feedback to improve HPV vaccination rates in resident clinics
.
Pediatrics
.
2018
;
141
(
4
):
e20170498
27
Malo
TL
,
Hall
ME
,
Brewer
NT
,
Lathren
CR
,
Gilkey
MB
.
Why is announcement training more effective than conversation training for introducing HPV vaccination? A theory- based investigation
.
Implement Sci
.
2018
;
13
(
1
):
57
28
Szilagyi
PG
,
Humiston
SG
,
Stephens-Shields
AJ
, et al
.
Effect of training pediatric clinicians in human papillomavirus communication strategies on human papillomavirus vaccination rates: a cluster randomized clinical trial
.
JAMA Pediatr
.
2021
;
175
(
9
):
901
910
29
Kempe
A
,
O’Leary
ST
,
Markowitz
LE
, et al
.
HPV vaccine delivery practices by primary care physicians
.
Pediatrics
.
2019
;
144
(
4
):
e20191475
30
Attia
AC
,
Wolf
J
,
Núñez
AE
.
On surmounting the barriers to HPV vaccination: we can do better
.
Ann Med
.
2018
;
50
(
3
):
209
225
31
Spencer
JC
,
Brewer
NT
,
Trogdon
JG
, %
Weinberger
M
,
Coyne-Beasley
T
,
Wheeler
SB
.
Cost-effectiveness of interventions to increase HPV vaccine uptake
.
Pediatrics
.
2020
;
146
(
6
):
e20200395
32
Perkins
RB
,
Legler
A
,
Jansen
E
, et al
.
Improving HPV vaccination rates: a stepped-wedge randomized trial
.
Pediatrics
.
2020
;
146
(
1
):
e20192737
33
Paskett
ED
,
Krok-Schoen
JL
,
Pennell
ML
, et al
.
Results of a multilevel intervention trial to increase human papillomavirus (HPV) vaccination amoung adolescent girls
.
Cancer Epidemiol Biomarkers Prev
.
2016
;
25
(
4
):
593
602
34
Fisher-Borne
M
,
Preiss
AJ
,
Black
M
, %
Roberts
K
,
Saslow
D
.
Early outcomes of a multilevel human papillomavirus vaccination pilot intervention in federally qualified health centers
.
Acad Pediatr
.
2018
;
18
(
2S
):
S79
S84
35
Institute for Healthcare Improvement
.
How to improve
.
Available at: www.ihi.org/resources/Pages/HowtoImprove/default.aspx. Accessed April 20, 2021
36
Katz
IT
,
Bogart
LM
,
Fu
CM
, et al
.
Barriers to HPV immunization among blacks and latinos: a qualitative analysis of caregivers, adolescents, and providers
.
BMC Public Health
.
2016
;
16
(
1
):
874
37
Centers for Disease Control and Prevention
.
Immunization schedules
.
38
Miller
WR
,
Rose
GS
.
Toward a theory of motivational interviewing
.
Am Psychol
.
2009
;
64
(
6
):
527
537
39
Massachusetts Immunization Information System
.
40
Crewell
J
,
Poth
C
.
Qualitative Inquiry and Research Design: Choosing Among Five Approaches
.
Los Angeles, CA
:
Sage Publications
;
2012
41
Benneyan
JC
,
Lloyd
RC
,
Plsek
PE
.
Statistical process control as a tool for research and healthcare improvement
.
Qual Saf Heal Care
.
2003
;
12
(
6
):
458
464
42
Scoville
R
,
Little
K
,
Rakover
J
,
Luther
K
,
Mate
K
.
Sustaining improvement
.

Supplementary data