Implementing Evidence-Based Strategies to Improve HPV Vaccine Delivery

Participants were pediatricians employed by Cook Children’s Health Care System, a large, not-for-profit integrated delivery system based in Fort Worth, Texas. Cook Children’s ambulatory care clinics share a common electronic medical record (EMR) system. Most clinics in the system use a practice model in which physicians care for a defined panel of patients, allowing for provider-specific assessment of vaccination coverage. However, 7 clinics, including those that primarily serve publicly insured patients, do not use this model; in these clinics, multiple physicians can care for a patient, and provider-specific assessment is not possible. In the current study, eligible physicians practiced in ambulatory care clinics serving primarily commercially insured patients and served panels of ≥50 patients aged 12 to 14, as identified by Cook Children’s EMRs. Physicians were excluded if they practiced in clinics in which physicians did not have defined patient panels or in the clinic that piloted the QI program. We focused on 12- to 14-year-old patients to align with Cook Children’s existing quality metrics for adolescent vaccination. On the basis of our eligibility criteria, we excluded 7 clinics in which physicians did not have defined patient panels and 1 pilot clinic, leaving 25 clinics eligible for participation (Fig 1).

This evaluation used a cluster-randomized design. A Cook Children’s staff member who was unaffiliated with the project randomly assigned clinics to either the QI arm (13 clinics) or the wait-list control arm (12 clinics). Because our evaluation design used a wait-list control, it was not possible to blind training leaders or participating physicians to their assigned evaluation arm. To facilitate recruitment, Cook Children’s leadership provided a letter describing the project and encouraging participation. One author, who is a Cook Children’s physician (J.V.T.), then contacted each clinic to confirm participation and coordinate scheduling. Cook Children’s Institutional Review Board determined that the project did not constitute human subjects research.

In our QI program, we used existing materials that we adapted to the local context of Cook Children’s. First, physicians received a 1-hour, in-clinic training session delivered between February 2017 and March 2017. Training sessions were led by 1 of 5 Cook Children’s pediatricians who we identified as high performers in that they had delivered ≥1 dose of the HPV vaccine to ≥70% of their 12- to 14-year-old patients. Training content focused on the epidemiology of HPV, the need to improve HPV vaccine coverage, the vaccine’s safety profile and prevention benefits, and the importance of delivering high-quality presumptive recommendations for HPV vaccination. In the case of parental hesitancy, the training promoted communication strategies such as emphasizing cancer prevention and providing follow-up counseling as needed to encourage secondary acceptance of HPV vaccination. Training leaders used an existing PowerPoint presentation,¹⁹ which we adapted with Cook Children’s branding. Leaders also used video vignettes to demonstrate strategies for communicating with parents about HPV vaccination. These video vignettes were filmed by Cook Children’s pediatricians and videographers in local settings using scripts adapted from those developed by the Minnesota Department of Health.²⁰ Separate vignettes addressed common parent concerns about whether HPV vaccination is effective, necessary, and safe.

Trainings were primarily targeted toward physicians. However, we invited all providers and staff in participating clinics to attend. Additionally, physicians were encouraged to engage others in their clinic by sharing HPV vaccination coverage estimates, training staff on HPV vaccination guidelines, and implementing standing orders. Participants received 1 hour of continuing medical education credit for their participation, and physicians were awarded Maintenance of Certification Part 4 points for completion of the program.

In addition to training, each participating physician received individual assessment and feedback on HPV vaccination coverage at 3 time points. As previously described, each physician cared for a defined panel of patients. Using EMR data and an adapted version of an existing immunization “report card,”¹⁹ each physician’s assessments reported the percentage of 12- to 14-year-old patients in his or her panel who had initiated HPV vaccination. The report card was used to encourage physicians to set a goal to raise HPV vaccination coverage over the 6-month project period by vaccinating at least 10% of their 12- to 14-year-old patients who had not initiated HPV vaccination. At 3- and 6-month follow-ups, physicians received updated coverage estimates so that they could track their progress toward their goal. These assessments included all patients in the physician’s panel, regardless of whether they had made an office visit during the follow-up period. Physicians also received educational materials via e-mail.

Before beginning the QI project, 1 author (J.V.T.) piloted the training in 1 clinic. He refined the approach on the basis of physicians’ feedback and then prepared physician training leaders to deliver it through an in-person “train the trainer” session. Training leaders were compensated for time spent traveling to and facilitating the training sessions. At a 6-month follow-up, clinics randomly assigned to the wait-list control arm received the same QI program.

We assessed the reach of our QI program at the clinic and physician levels. For clinics, we calculated the proportion of eligible clinics in the QI arm that received training. For physicians, we calculated the proportion of those in the QI arm who received the intervention, which we defined as attending training and receiving assessment and feedback. Physicians received the initial assessment during their in-person training. The second and third assessments were sent by e-mail and counted as having been received if the e-mail transmitted successfully.

Our vaccination outcome was coverage change for HPV vaccine initiation (≥1 dose) between baseline and the 6-month follow-up. At each time point, we assessed vaccination coverage (ie, the proportion of patients vaccinated) among patients aged 12 to 14 using standardized EMR queries. The denominator consisted of all patients in a physician’s panel at that time point, regardless of whether those patients had had an office visit during the study period.

We conducted hierarchical linear modeling (HLM) to assess the QI program’s impact on HPV vaccination. We used 3-level random-intercept models to allow for observations of HPV vaccination coverage over time while accounting for the nested data structure of physicians within clinics, including correlated residuals.²¹ We tested for the effect of the QI program by including 2 dichotomous explanatory variables: time (0 = baseline or “pre”; 1 = 6-month follow-up or “post”) and treatment (0 = control; 1 = QI). The impact of the QI program was represented by the cross-level interaction of these 2 variables.

We ran 2 HLM models. Model 1 was an intent-to-treat analysis of all physicians randomly assigned to QI and control groups. Model 2 was a sensitivity analysis that excluded 6 physicians (2 in the QI arm and 4 in the control arm) with questionable data quality. We conducted the latter analysis in the manner described by Hersh et al,²² who note that EMR data are susceptible to measurement error.²³ Using validity checks and distribution analysis, we identified outliers according to dfbeta and rstudent metrics.^22,24 This approach was used to identify physicians with large changes in the size of their patient panel over the study period. Follow-up inquiries suggested that these changes were due to clinic closures in the region, which resulted in automatic reassignment of patients to the physician’s panel between baseline and follow-up assessments.

For each model, we also calculated Cohen’s d at the physician level as a measure of the magnitude of QI program effects. HLM analyses were conducted by using HLM 6, and all other analyses were conducted in Stata version 13 (Stata Corp, College Station, TX). Statistical analyses were 2 tailed with a critical α of .05.

Our sample included 25 clinics with 77 physicians serving 22 983 patients aged 12 to 14. Clinics were similar in size by arm (P > .05; Table 1). Clinics in the QI arm had a mean of 3.2 (SD = 1.7) physicians and 883 (SD = 454) patients per clinic. Clinics in the control arm had a mean of 2.9 (SD = 1.6) physicians and 958 (SD = 520) patients per clinic. HPV vaccination coverage was higher in the QI arm than the control arm at baseline (53% vs 45%; P < .05).

Among clinics randomly assigned to the QI arm, 100% (13 of 13) scheduled and subsequently received the physician-led training. Among eligible physicians within these clinics, 95% (41 of 43) attended a training session and received assessment and feedback.

In the overall sample (model 1), HPV vaccination coverage (≥1 dose) increased over the 6-month project period by 8.6 percentage points in the QI arm and 6.4 percentage points in the control arm (Table 2). The treatment effect was not statistically significant in HLM analyses (b = 0.023; SE = 0.018; P > .05), but the effect size indicated a small positive effect of the QI program on vaccination (d = 0.29). HLM analyses additionally revealed significant variance in HPV vaccination coverage across physicians and clinics in model 1, with the majority of total variance occurring at the physician level (74%) versus the clinic level (14%; Supplemental Table 3).

In the subsample of physicians with sufficient data quality (model 2), HPV vaccination coverage increased 10.2 percentage points in the QI arm and 6.9 percentage points in the control arm (Table 2). HPV vaccination coverage change was greater in the QI arm compared with the control arm (b = 0.034; SE = 0.015; P < .05; d = 0.51). Similar to model 1, the majority of total variance in model 2 occurred at the physician level (75%) versus clinic level (18%; Supplemental Table 3).

In our QI program, we used local resources to adapt and implement evidence-based strategies for improving HPV vaccine delivery across 25 clinics serving >22 000 adolescents in a large pediatric health care system. Noteworthy features of the program included the rebranding of existing QI materials, physician-level assessment and feedback, and the recruitment and training of high-performing physicians within the system to facilitate QI activities. Using these strategies, we met the goal of achieving excellent reach, with almost all eligible physicians participating in training, as well as assessment and feedback, even in the absence of financial incentives. This level of participation is notably higher than that reported for research-based efforts of a similar scope, including some that did provide financial incentives.^{1,4,5,7,15,16} Broad reach to physicians and other vaccine providers is important given that improving provider communication about HPV vaccination is likely the single most effective way to increase uptake.^25,–31 Although additional data would be needed to understand which components resulted in high levels of participation, our success in engaging a hard-to-reach population likely speaks to the power of leveraging local networks to gain entrée to busy clinical settings.^32,33 

In terms of effectiveness, our evaluation suggests that the project had a small positive impact on HPV vaccination, with physicians in the QI arm achieving increases in coverage that were 2 to 3 percentage points higher than in the control arm at the 6-month follow-up. This impact is in keeping with previous QI efforts that have employed assessment and feedback to improve adolescent vaccination, although it is smaller than increases observed in externally funded research studies of provider-directed vaccination interventions.^{1,–3,5,7,12} Why QI efforts result in lower impact is unclear, but possible reasons include lower intervention fidelity or intensity. In the current study, it may be that local adaptation was effective for bringing physicians and other providers to the table but did not necessarily translate into greater adoption of evidence-based practices when compared with standardized approaches. Alternatively, somewhat higher HPV vaccination coverage among clinics in the QI arm at baseline may have had a ceiling effect, making additional improvements more difficult for that arm compared with the control. Whatever the case, even small increases in vaccination coverage across large patient populations can be clinically meaningful. For example, a 3–percentage point increase in coverage for HPV vaccine initiation across all 12- to 14-year-olds in the Cook Children’s system would translate into an additional 689 vaccine doses delivered over a 6-month period. Future QI efforts should seek to build on our current effort by identifying strategies for increasing impact while maintaining reach. Opportunities might include increasing the frequency of vaccination coverage assessments or the intensity of QI coaching; publicly recognizing physicians or clinical teams who demonstrate improvement; or combining provider-directed approaches with EMR prompts, reminders and/or recall, or other systems-level evidence-based interventions.

Over the course of this evaluation, we identified 2 issues that may be helpful for informing future QI projects. First, our multilevel analysis suggests that most of the variance in HPV vaccination coverage occurred at the physician level. This finding underscores the prominent role of individual physicians and other health care providers in HPV vaccine delivery as well as the need for QI efforts to focus on individuals in addition to clinical systems. The relative prominence of physician-level variance also suggests that evaluations that only assess clinic-level HPV vaccination coverage may overlook important differences in the performance and impact of individuals within clinics in addition to inaccurately estimating clinic-level effects and significance tests. Second, we found that the quality of our provider data were, in several cases, affected by regional clinic closures that resulted in dramatic increases in the size of an individual physician’s patient panel over the course of our project, thereby challenging our ability to accurately track performance for those physicians. Because the national trend of clinic consolidation is likely to be an ongoing phenomenon affecting many health systems,³⁴ evaluators should plan for quality checks to identify compromised assessments.²² At the same time, consolidations may provide opportunities for systems to assess patients’ status for vaccinations and other preventive services so that they may be brought up to date as they establish care within a new clinic.

Strengths of this evaluation include a strong design, the use of vaccination data drawn from medical records, and a multilevel analytic approach that accounted for the nesting of providers within clinics. By evaluating the QI efforts of a large pediatric health care system, we are able to provide novel data on the impact of evidence-based strategies as they are adapted and scaled up in real-world clinical settings. Limitations include our focus on commercially insured patients within a single integrated delivery system that had the ability to mobilize resources even in the absence of external funding. Systems that are smaller, are located in other geographic regions, or serve primarily publicly insured patients may have fewer resources or may face different challenges in reaching vaccine providers. In our QI program, we employed local adaptation in several ways, and future research is needed to determine the relative value of each strategy. Finally, our outcome measure of HPV vaccination coverage can be considered both a limitation and strength. Vaccination coverage, or the proportion of vaccinated patients among all patients in a provider’s panel, is a more conservative way to assess QI effects than other common measures, such as missed opportunities or the proportion of vaccinated patients among the subset of those who present for care during the study period. HPV vaccination coverage may underestimate effects. However, this approach best reflects the public health goal of ensuring that all children are vaccinated against HPV and other vaccine-preventable diseases.

Our findings suggest that our locally adapted QI program achieved excellent reach to physicians as well as small improvements in HPV vaccination coverage among the patients they serve. As research continues to yield evidence-based strategies for improving the delivery of HPV vaccine and other preventive services, health care systems urgently need support in scaling up these strategies efficiently and without reliance on external funding. Our experience suggests that adapting existing materials and harnessing local talent (in the form of physicians who are already high performers) are feasible in the context of a large pediatric health care system and should be considered by other systems as a way to extend reach. At the same time, continued efforts are needed to increase the impact of these and other HPV vaccination QI efforts to ensure that young people receive the greatest possible protection against future HPV cancers.

We thank Cook Children’s leadership for supporting this project, including Britt Nelson, Jim Cunningham, Joann Sanders, and Maynard Dyson. This project also benefited from the administrative efforts of Cathy McCall, Christine Oehlert, Tina Wilkerson, Ashley Lawry, and Tracie Lawson at Cook Children’s Physician Network as well as Chad Jones and Jennifer Cabrera at Cook Children’s Health Care System. The high performers who facilitated physician trainings included Vida Amin, Kim Burgess, Diane Van Maele, and Nicole Wineriter. We thank those who acted in our communication vignettes, including Sara Garza, Catherine Hampton, Kim Mangham, Kimberly Matlock, Amy Medalie, Amani Terrell, and their children. Finally, we thank Emily Newman, who created our video abstract.

EMR

electronic medical record

HLM

hierarchical linear modeling

HPV

human papillomavirus

QI

quality improvement