BACKGROUND:

Although autodialer centralized reminder and recall (C-R/R) from state immunization information systems (IISs) has been shown to raise childhood vaccination rates, its impact on human papillomavirus (HPV) vaccination rates is unclear.

METHODS:

In a 4-arm pragmatic randomized controlled trial across 2 states, we randomly selected practices representative of the specialty (pediatrics, family medicine, and health center) where children received care. Within each practice, patients 11 to 17.9 years old who had not completed their HPV vaccine series (NY: N = 30 616 in 123 practices; CO: N = 31 502 in 80 practices) were randomly assigned to receive 0, 1, 2, or 3 IIS C-R/R autodialer messages per vaccine dose. We assessed HPV vaccine receipt via the IIS, calculated intervention costs, and compared HPV vaccine series initiation and completion rates across study arms.

RESULTS:

In New York, HPV vaccine initiation rates ranged from 37.0% to 37.4%, and completion rates were between 29.1% and 30.1%, with no significant differences across study arms. In Colorado, HPV vaccine initiation rates ranged from 31.2% to 33.5% and were slightly higher for 1 reminder compared with none, but vaccine completion rates, ranging from 27.0% to 27.8%, were similar. On adjusted analyses in Colorado, vaccine initiation rates were slightly higher for 1 and 3 C-R/R messages (adjusted risk ratios 1.07 and 1.04, respectively); completion rates were slightly higher for 1 and 3 C-R/R messages (adjusted risk ratios 1.02 and 1.03, respectively).

CONCLUSIONS:

IIS-based C-R/R for HPV vaccination did not improve HPV vaccination rates in New York and increased vaccination rates slightly in Colorado.

What’s Known on This Subject:

Although centralized reminder and recall by phone (autodialer) has been shown to improve childhood vaccination rates, little is known about its impact on human papillomavirus (HPV) vaccination rates, particularly when coming from state immunization information systems.

What This Study Adds:

In a pragmatic randomized clinical trial, immunization information systems–based centralized reminder and recall for the HPV vaccine did not raise HPV vaccination rates in New York and raised rates minimally in Colorado. More intensive but perhaps less scalable interventions are warranted.

Infection by human papillomavirus (HPV) causes most cases of cervical cancers; 90% of anal cancers; 40% of vulvar, vaginal, and penile cancers; and >60% of oropharyngeal cancers.14  HPV infection leads to 33 700 new cancer cases in the United States annually,5  ∼4000 deaths, and >$4 billion in cancer-related health care costs.6,7  Although HPV vaccines are highly effective,8  and a Healthy People 2020 goal is >80% vaccination coverage among 13- to 17-year-olds, in 2017 (the year this study's fieldwork began) only 53% of US girls and 44% of boys 13 to 17 years of age had completed their HPV vaccine series.911 

An evidence-based method to increase vaccination rates is distribution of reminder and recall (R/R) messages to inform patients or parents of upcoming vaccines (reminders) or past-due vaccines (recall).12,13  A recent Cochrane review revealed that R/R for vaccines, including adolescent vaccines, can increase vaccination rates.14  The Task Force on Community Preventive Services15  and immunization experts12,16  recommend R/R messages for vaccinations. However, less than one-fifth of practices send reminders routinely, and rarely for the HPV vaccine.17,18  Barriers include limited resources, technological challenges, and competing priorities.1921 

One potential solution involves centralized reminder and recall (C-R/R), in which health systems, state immunization information systems (IISs), or managed care organizations use centralized electronic databases as the data source for sending R/R messages, thereby capitalizing on economies of scale. Most health providers and parents are in favor of C-R/R.18,22,23  Previous trials revealed that C-R/R by using IISs for routine childhood vaccines (ie, not adolescent vaccines) was more effective and cost-effective than practice-based R/R.2427 

HPV vaccination is more challenging than other adolescent vaccines8  because of the need for multiple doses28  and frequent parent questions29  or vaccine hesitancy.3032  Thus, IIS-based C-R/R may not be effective for the HPV vaccine.

Studies from 2 health maintenance organization settings,3335  a state Medicaid system,36  and a combination of practice records plus IIS records37  revealed that C-R/R improved HPV vaccination rates by small to moderate amounts. In none of these studies were data or messages from IISs used alone. IIS-based C-R/R might be scalable because all states have an IIS, and phone numbers, addresses, and vaccination dates are often uploaded from electronic health records.19  A recent study from New York,38  using C-R/R based on IIS data alone, revealed that mailed letter reminders to parents of adolescents who had not yet initiated the HPV vaccine series increased HPV vaccination rates by 2 percentage points, a significant effect at the population level. However, mailed letter reminders are costly.34  Little is known about the impact of IIS-based autodialer C-R/R on HPV vaccination rates.

In this study, we assessed the effectiveness and cost of implementing IIS C-R/R by using autodialed phone messages. We performed a pragmatic trial, comparing different numbers of IIS-based C-R/R messages to usual care across New York and Colorado. We hypothesized that adolescents whose families were sent IIS-based C-R/R messages would have higher HPV vaccination rates than adolescents receiving usual care and that more reminders would have a dose-response effect.

We conducted a 4-arm, pragmatic39  (effectiveness) randomized controlled trial (RCT), testing the impact of 1, 2, or 3 autodialer phone reminders per dose of the HPV vaccine needed (versus no C-R/R) sent to parents on behalf of the IIS and primary care practices about HPV vaccination. This study was approved by institutional review boards at University of California, Los Angeles and University of Colorado (with waiver of consent) and by health departments in New York and Colorado. The intervention was conducted between February 2017 and January 2019.

The New York State Immunization Information System (NYSIIS) captures vaccination data within the 57 counties outside of New York City. New York requires any immunization provider who vaccinates children <19 years of age to report vaccine and demographic information to the NYSIIS via electronic or manual upload. On the basis of 2017 census data, 97% of 11- to 17-year-olds in New York had ≥2 vaccine records in the NYSIIS.40  The NYSIIS conducted the afore-mentioned mailed C-R/R for HPV vaccination.38 

In contrast, Colorado does not require providers to report childhood and/or adolescent vaccination data to the Colorado Immunization Information System (CIIS), yet 86% of all known immunizing providers submitted data to the CIIS as of April 2019. Providers can electronically or manually enter data into the CIIS and can set their practice as a patient’s default or primary practice. The Colorado Department of Public Health and Environment has executed large-scale C-R/R studies for childhood but not for HPV vaccines.24,26,27 

We randomly selected practices and adolescents across counties in New York (excluding New York City) and Colorado (Fig 1). Practices were eligible if they had ≥80 adolescents 11 to 19 years of age in New York or ≥50 in Colorado; this was to ensure adequate numbers per study arm. Practices were stratified on the basis of rural, urban, or suburban location in New York and rural or urban location in Colorado. We then used within-practice randomization to allocate adolescents 11 to 17 years (on February 1, 2017) to 1 of 4 study arms; adolescents were excluded if they had completed their HPV series. Patient-level exclusion criteria (Fig 1) were created to identify patients who likely no longer resided in Colorado or New York. We also excluded deceased individuals in both states and adolescents whose parents had opted out of the CIIS (<2%); New York does not allow individuals to opt out of the NYSIIS. We grouped IIS participants into families or households, defined as having the same primary phone number. We randomly selected 1 index adolescent.

FIGURE 1

Consolidated Standards of Reporting Trials diagram plus practice and patient eligibility criteria. HMO, health maintenance organization.

FIGURE 1

Consolidated Standards of Reporting Trials diagram plus practice and patient eligibility criteria. HMO, health maintenance organization.

Close modal

Small differences in subject inclusion were based on preferences within each state (Fig 1). In New York, we stratified practices by practice type (pediatrics, family medicine, and community health center) and selected the number of practices from each type to reflect its proportion in the study population. Practices could opt out of the study (23 of 123 opted out [19%]); practice names were included in all autodialer calls. In Colorado, we selected practices proportionally by practice type (pediatrics, family medicine, and community health center) on the basis of where adolescents were seen. Practices could not opt out of the study but had to opt in for practice names or phone numbers to be included in autodialer calls (49 of 80, or 61%, chose this option). Eligible adolescents were selected at random from practices in numbers proportionate to practice size. All analyses were stratified by state.

We contracted with a cloud-based telephony company (http://www.teletask.com) to send autodialer calls to the family’s primary phone number in the IIS. Autodialer calls were sent in English and Spanish (respondents needed to “presione dos” for Spanish). The calls contained the practice’s name and phone number, or the name and phone number of the county health department in Colorado when practices did not wish for practice names and phone numbers to be included. Messages were essentially identical in the 2 states. We consulted with parents and providers for feedback on the message content during the message development phase. Messages used the Health Belief Model framework with HPV vaccination framed as cancer prevention for all adolescents.41  A typical message is shown in Fig 2. We randomly assigned index adolescent subjects to either usual care (no calls) or up to 1, 2, or 3 autodial calls per needed dose of the HPV vaccine, depending on the study arm. Autodial calls reflected monthly IIS data pulls identifying eligible adolescents. Respondents could opt out of the study by calling a toll-free number included in the message or by pressing “9” during the autodial call.

FIGURE 2

Example of an autodialer reminder script from New York (the script was read by computer-assisted human recording if an individual answered the autodialer call, and the voice sounded like a regular phone call).

FIGURE 2

Example of an autodialer reminder script from New York (the script was read by computer-assisted human recording if an individual answered the autodialer call, and the voice sounded like a regular phone call).

Close modal

We used the updated (October 2016) Advisory Committee on Immunization Practices dosing schedule for the HPV series; adolescents 11 to <15 years required 2 doses and were sent C-R/R messages at time 0 and 12 months after the first dose. Adolescents initiating the series after the 15th birthday, who required 3 doses, were sent calls as follows: dose 1, time 0; dose 2, 2 months after dose 1 was received; and dose 3, 6 months after dose 1 was received and 4 months after dose 2 was received. For adolescents randomly assigned to 2 or 3 calls per dose, we spaced calls by 1 month. Adolescents who had initiated the series at baseline were still eligible for calls for follow-up doses.

The primary study outcome was initiation and/or completion of the HPV 2- or 3-dose vaccine series among adolescents aged 11 to <19 years on the basis of IIS data. We compared outcomes for all ages and then by sex and age subgroups. We also calculated the cost of delivering C-R/R and planned cost-effectiveness analyses if C-R/R raised vaccination rates.

The study was powered to detect an absolute difference of 2 percentage points in vaccine completion when comparing the control arm and any intervention arm. We considered this difference to be the minimum outcome of public health importance.

Primary outcomes were HPV vaccine series initiation and series completion. For New York we used mixed-effects multivariable Poisson regression models with robust SEs and for Colorado we used mixed-effects multivariable log-binomial models to obtain unadjusted and adjusted risk ratios (RRs) along with their 95% confidence intervals (CIs) and P values. The adjusted model contains the prespecified covariates of the child’s age group and sex, the practice type, and the practice’s urban or rural status. We included a random intercept for practice to account for clustering of patients within practices and tested interactions between each predictor and study arm (none were significant, so data are not shown). All major analyses were intention-to-treat analyses. As an exploratory analysis, we also used the Kaplan-Meier estimator to estimate the initiation and completion rate at different ages. In analyses, we used SAS version 9.4 (SAS Institute, Inc, Cary, NC) and R version 3.5.2 (R Foundation for Statistical Computing, Vienna, Austria).

We compared the cost of implementing C-R/R for each intervention arm and for each state (details upon request). Cost domains included the following: (1) consensus building and preliminary work, (2) training, (3) collaboration, and (4) implementation (which included recall and autodialer costs). Costs were stratified by up-front costs (one-time costs to implement the intervention) and intervention costs (associated with activities occurring at each C-R/R round). We calculated total costs for each intervention arm by aggregating costs for each cost domain and reported average costs per randomly assigned child for initiation- and completion-related efforts. We divided the average cost per randomly assigned child by the proportion who achieved initiation (completion) to estimate the within-arm average cost per initiation (or per completion). We then estimated (after adjustment) incremental cost per additional vaccine initiated and completed for all statistically significant study groups. All costs reflected the perspective of the IIS.

In New York, most practices were pediatric and urban (Table 1). In Colorado, one-quarter of practices were pediatric, and most were urban. For both states, <2% of patients opted out of phone calls, and <15% had missing phone numbers in IIS databases.

TABLE 1

Characteristics of Primary Care Practices and Adolescents

Characteristics of Practices and PatientsNew YorkColorado
Practices   
 No. recruited for study 100 80 
 Practice type, n (%)   
  Pediatrics 77 (77) 19 (24) 
  Family medicine 16 (16) 41 (51) 
  Community health center 7 (7) 20 (25) 
 Practice location, n (%)   
  Rural 13 (13) 20 (25) 
  Urban 87 (87) 60 (75) 
 Inclusion of practice name on R/R, % 100 61 
Patients   
  Total patients, N 30 616 31 502 
  11 to <13 y, n (%) 13 412 (44) 14 122 (45) 
  13 to <15 y, n (%) 9832 (32) 7828 (25) 
  15 to <18 y, n (%) 7372 (24) 9552 (30) 
 Sex, n (%)   
  Female 14 402 (47) 14 824 (47) 
  Male 16 214 (53) 16 678 (53) 
 Total patients who opted out of R/R, n (%)a 272 (1) 487 (2) 
 Missing phone numbers, n (%)a 3859 (12) 2838 (9) 
Characteristics of Practices and PatientsNew YorkColorado
Practices   
 No. recruited for study 100 80 
 Practice type, n (%)   
  Pediatrics 77 (77) 19 (24) 
  Family medicine 16 (16) 41 (51) 
  Community health center 7 (7) 20 (25) 
 Practice location, n (%)   
  Rural 13 (13) 20 (25) 
  Urban 87 (87) 60 (75) 
 Inclusion of practice name on R/R, % 100 61 
Patients   
  Total patients, N 30 616 31 502 
  11 to <13 y, n (%) 13 412 (44) 14 122 (45) 
  13 to <15 y, n (%) 9832 (32) 7828 (25) 
  15 to <18 y, n (%) 7372 (24) 9552 (30) 
 Sex, n (%)   
  Female 14 402 (47) 14 824 (47) 
  Male 16 214 (53) 16 678 (53) 
 Total patients who opted out of R/R, n (%)a 272 (1) 487 (2) 
 Missing phone numbers, n (%)a 3859 (12) 2838 (9) 
a

These patients were still included in the intention-to-treat analysis but did not receive autodialer calls.

HPV vaccine initiation rates for the entire cohort were low as expected because we excluded adolescents who had already completed their HPV vaccine series at the start of the study (Fig 3). In New York (Fig 3A), both postintervention vaccine initiation and completion rates were virtually identical across all study arms, suggesting no impact of the intervention compared with standard care, irrespective of the number of messages sent. In Colorado (Fig 3B), initiation and completion rates were similar across study arms except that the study group sent 1 call per dose had an HPV vaccine initiation rate 2.3 percentage points higher than that of controls (RR = 1.07; 95% CI 1.02–1.13); however, vaccine completion rates were not better in this study arm than in the control arm.

FIGURE 3

Vaccine series initiation and completion rates in New York (A) and Colorado (B) at the end of the study.

FIGURE 3

Vaccine series initiation and completion rates in New York (A) and Colorado (B) at the end of the study.

Close modal

To assess whether the C-R/R may have resulted in earlier receipt of the HPV vaccine, we used Kaplan-Meier estimates to show the initiation and completion rate at different ages. There were no statistically (P = .856 and P = .952 for completion, respectively, in NY; P = .09 and P = .75 in CO) or clinically significant differences between study arms and control arm in the age of initiation or completion of HPV vaccine except for the effect of 1-call on initiation rates in Colorado.

Table 2 shows results for a multivariable mixed model used to compare unadjusted and adjusted RRs of vaccine initiation and completion by study arm. In New York, there were no significant findings on either unadjusted or adjusted analyses for any of the study arms (or for all arms combined). In Colorado regarding vaccine initiation, on unadjusted analyses, the 1-call R/R arm had higher HPV vaccine initiation rates than the control arm, and on adjusted analyses, the 1-call R/R (adjusted RR = 1.07; 95% CI 1.04–1.10) and the 3-call R/R (adjusted RR = 1.04; 95% CI 1.01–1.06) arms had higher initiation rates than the control arm. In Colorado regarding vaccine completion, the adjusted RRs were 1.02 (95% CI 1.00–1.05) for the 1-call R/R arm and 1.03 (95% CI 1.00–1.05) for the 3-call R/R arm.

TABLE 2

Multivariable Mixed Model Revealing Unadjusted and Adjusted RR for HPV Vaccine Series Initiation and Completion by Study Group (0, 1, 2, or 3 C-R/R Messages)

Study GroupNew YorkColorado
%Unadjusted RR (95% CI)Adjusted RR (95% CI)a%Unadjusted RR (95% CI)Adjusted RR (95% CI)a
HPV vaccine initiation       
 Control 37.4 Reference Reference 31.2 Reference Reference 
 1 call 37.0 0.99 (0.93–1.05) 0.99 (0.93–1.05) 33.5 1.07 (1.02–1.13)* 1.07 (1.04–1.10)* 
 2 calls 37.0 0.99 (0.94–1.05) 0.99 (0.94–1.05) 31.5 1.01 (0.95–1.07) 1.01 (0.98–1.04) 
 3 calls 37.0 0.99 (0.94–1.05) 1.00 (0.94–1.05) 32.3 1.03 (0.99–1.08) 1.04 (1.01–1.06)* 
HPV vaccine completion       
 Control 29.2 Reference Reference 27.0 Reference Reference 
 1 call 30.1 1.03 (0.97–1.09) 1.02 (0.97–1.08) 27.8 1.03 (0.96–1.10) 1.02 (1.00–1.05) 
 2 calls 29.8 1.02 (0.97–1.07) 1.02 (0.97–1.07) 27.5 1.02 (0.97–1.07) 1.02 (0.99–1.04) 
 3 calls 29.1 1.00 (0.94–1.06) 0.99 (0.94–1.05) 27.7 1.02 (0.98–1.07) 1.03 (1.00–1.05) 
Study GroupNew YorkColorado
%Unadjusted RR (95% CI)Adjusted RR (95% CI)a%Unadjusted RR (95% CI)Adjusted RR (95% CI)a
HPV vaccine initiation       
 Control 37.4 Reference Reference 31.2 Reference Reference 
 1 call 37.0 0.99 (0.93–1.05) 0.99 (0.93–1.05) 33.5 1.07 (1.02–1.13)* 1.07 (1.04–1.10)* 
 2 calls 37.0 0.99 (0.94–1.05) 0.99 (0.94–1.05) 31.5 1.01 (0.95–1.07) 1.01 (0.98–1.04) 
 3 calls 37.0 0.99 (0.94–1.05) 1.00 (0.94–1.05) 32.3 1.03 (0.99–1.08) 1.04 (1.01–1.06)* 
HPV vaccine completion       
 Control 29.2 Reference Reference 27.0 Reference Reference 
 1 call 30.1 1.03 (0.97–1.09) 1.02 (0.97–1.08) 27.8 1.03 (0.96–1.10) 1.02 (1.00–1.05) 
 2 calls 29.8 1.02 (0.97–1.07) 1.02 (0.97–1.07) 27.5 1.02 (0.97–1.07) 1.02 (0.99–1.04) 
 3 calls 29.1 1.00 (0.94–1.06) 0.99 (0.94–1.05) 27.7 1.02 (0.98–1.07) 1.03 (1.00–1.05) 
a

Adjusted for age, practice type, urban or rural status, and sex.

*

P < .05

Table 3 shows HPV vaccine series initiation and completion rates and adjusted RRs by patient and practice characteristics. Vaccine initiation and completion rates were higher for younger adolescents and for pediatric practices. Rates were generally not associated with sex or rurality.

TABLE 3

HPV Vaccine Series Initiation and Completion Rates by Patient and Practice Characteristics

Patient and Practice FeaturesnVaccine InitiationVaccine Completion
%Adjusted RR (95% CI)a%Adjusted RR (95% CI)a
New York      
 Age, y      
  11.0–12.9 13 412 46.7 Reference 36.0 Reference 
  13.0–14.9 9832 33.7 0.72 (0.66–0.78)* 29.2 0.83 (0.77–0.89)* 
  15.0–17.9 7372 24.5 0.55 (0.49–0.61)* 19.7 0.57 (0.52–0.62)* 
 Sex      
  Female 14 402 37.2 Reference 30.6 Reference 
  Male 16 214 37.1 1.00 (0.95–1.06) 28.6 0.94 (0.89–1.00) 
 Practice type      
  Family medicine 2828 30.0 Reference 26.1 Reference 
  Pediatrics 25 353 38.8 1.31 (1.14–1.52)* 30.0 1.27 (0.80–2.02) 
  CHC or RHC 2435 26.9 0.92 (0.66–1.29) 28.6 1.19 (0.39–3.62) 
 Rurality      
  Downstate 14 921 36.6 Reference 27.5 Reference 
  Upstate rural 2614 37.1 0.96 (0.80–1.14) 28.7 1.14 (0.76–1.71) 
  Upstate urban 13 081 37.8 0.98 (0.87–1.10) 32.2 1.15 (0.90–1.46) 
Colorado      
 Age, y      
  11.0–12.9 14 122 39.4 Reference 34.5 Reference 
  13.0–14.9 7828 31.8 0.81 (0.79–0.83)* 30.1 0.88 (0.86–0.90)* 
  15.0–17.9 9552 20.2 0.53 (0.52–0.55)* 15.0 0.46 (0.44–0.47)* 
 Sex      
  Female 14 824 33.3 Reference 28.6 Reference 
  Male 16 678 31.1 0.97 (0.95–0.99)* 26.5 0.95 (0.93–0.97)* 
 Practice type      
  Family medicine 7177 27.1 Reference 21.5 Reference 
  Pediatrics 18 107 33.9 1.17 (1.09–1.25)* 29.2 1.32 (1.22–1.42)* 
  CHC or RHC 6218 33.1 1.20 (1.11–1.30)* 29.2 1.32 (1.21–1.44)* 
 Rurality      
  Rural 3661 28.5 0.93 (0.86–1.00) 26.6 0.96 (0.88–1.04) 
  Urban 27 841 32.6 Reference 27.6 Reference 
Patient and Practice FeaturesnVaccine InitiationVaccine Completion
%Adjusted RR (95% CI)a%Adjusted RR (95% CI)a
New York      
 Age, y      
  11.0–12.9 13 412 46.7 Reference 36.0 Reference 
  13.0–14.9 9832 33.7 0.72 (0.66–0.78)* 29.2 0.83 (0.77–0.89)* 
  15.0–17.9 7372 24.5 0.55 (0.49–0.61)* 19.7 0.57 (0.52–0.62)* 
 Sex      
  Female 14 402 37.2 Reference 30.6 Reference 
  Male 16 214 37.1 1.00 (0.95–1.06) 28.6 0.94 (0.89–1.00) 
 Practice type      
  Family medicine 2828 30.0 Reference 26.1 Reference 
  Pediatrics 25 353 38.8 1.31 (1.14–1.52)* 30.0 1.27 (0.80–2.02) 
  CHC or RHC 2435 26.9 0.92 (0.66–1.29) 28.6 1.19 (0.39–3.62) 
 Rurality      
  Downstate 14 921 36.6 Reference 27.5 Reference 
  Upstate rural 2614 37.1 0.96 (0.80–1.14) 28.7 1.14 (0.76–1.71) 
  Upstate urban 13 081 37.8 0.98 (0.87–1.10) 32.2 1.15 (0.90–1.46) 
Colorado      
 Age, y      
  11.0–12.9 14 122 39.4 Reference 34.5 Reference 
  13.0–14.9 7828 31.8 0.81 (0.79–0.83)* 30.1 0.88 (0.86–0.90)* 
  15.0–17.9 9552 20.2 0.53 (0.52–0.55)* 15.0 0.46 (0.44–0.47)* 
 Sex      
  Female 14 824 33.3 Reference 28.6 Reference 
  Male 16 678 31.1 0.97 (0.95–0.99)* 26.5 0.95 (0.93–0.97)* 
 Practice type      
  Family medicine 7177 27.1 Reference 21.5 Reference 
  Pediatrics 18 107 33.9 1.17 (1.09–1.25)* 29.2 1.32 (1.22–1.42)* 
  CHC or RHC 6218 33.1 1.20 (1.11–1.30)* 29.2 1.32 (1.21–1.44)* 
 Rurality      
  Rural 3661 28.5 0.93 (0.86–1.00) 26.6 0.96 (0.88–1.04) 
  Urban 27 841 32.6 Reference 27.6 Reference 

CHC, community health center; RHC, rural health center.

a

The adjusted estimates are adjusted for all other covariates in the table plus study arm (0, 1, 2, or 3 C-R/R messages).

*

P < .05

Table 4 shows the average costs per vaccine initiated and the average costs per completion for the 3 active study groups. The costs per randomly assigned child were relatively constant (and low) across study groups, ranging from $0.51 to $0.63 in Colorado and from $0.66 to $0.83 in New York. The average cost per vaccine series initiated ranged from $1.53 to $2.24, whereas the average cost per series completed ranged from $1.85 to $2.85, with higher-contact study groups costing more than the 1-call arm. Compared with controls, the incremental cost to achieve 1 additional vaccine series initiation in Colorado was $24 for 1 call and $51 for 3 calls. Compared with controls, the incremental cost to achieve 1 additional series completion in Colorado was $95 for 1 call and $78 for 3 calls (we only include study arms with adjusted significant findings).

TABLE 4

Up-front and Intervention Costs Per Randomly Assigned Child and Per Vaccine Initiation and Completion by Study Arm

Study GroupNew YorkColorado
n or %Up-front Cost, $Intervention Cost, $Total Cost, $n or %Up-front Cost, $Intervention Cost, $Total Cost, $
Cost per randomly assigned child         
 1 call 7682 0.27 0.39 0.66 7864 0.18 0.33 0.51 
 2 calls 7631 0.27 0.48 0.75 7870 0.18 0.39 0.57 
 3 calls 7579 0.28 0.55 0.83 7890 0.18 0.45 0.63 
Cost per vaccine initiation         
 1 call 37.0% 0.73 1.05 1.78 33.5% 0.55 0.98 1.53 
 2 calls 37.0% 0.73 1.30 2.03 31.5% 0.59 1.23 1.82 
 3 calls 37.0% 0.76 1.49 2.24 32.3% 0.57 1.38 1.95 
Cost per vaccine completion         
 1 call 30.1% 0.91 1.30 2.20 27.8% 0.67 1.18 1.85 
 2 calls 29.8% 0.92 1.60 2.52 27.5% 0.67 1.41 2.09 
 3 calls 29.1% 0.95 1.90 2.85 27.7% 0.67 1.61 2.28 
Study GroupNew YorkColorado
n or %Up-front Cost, $Intervention Cost, $Total Cost, $n or %Up-front Cost, $Intervention Cost, $Total Cost, $
Cost per randomly assigned child         
 1 call 7682 0.27 0.39 0.66 7864 0.18 0.33 0.51 
 2 calls 7631 0.27 0.48 0.75 7870 0.18 0.39 0.57 
 3 calls 7579 0.28 0.55 0.83 7890 0.18 0.45 0.63 
Cost per vaccine initiation         
 1 call 37.0% 0.73 1.05 1.78 33.5% 0.55 0.98 1.53 
 2 calls 37.0% 0.73 1.30 2.03 31.5% 0.59 1.23 1.82 
 3 calls 37.0% 0.76 1.49 2.24 32.3% 0.57 1.38 1.95 
Cost per vaccine completion         
 1 call 30.1% 0.91 1.30 2.20 27.8% 0.67 1.18 1.85 
 2 calls 29.8% 0.92 1.60 2.52 27.5% 0.67 1.41 2.09 
 3 calls 29.1% 0.95 1.90 2.85 27.7% 0.67 1.61 2.28 

This 4-arm pragmatic RCT is the first multistate RCT to test the impact of C-R/R autodialer calls using only IIS data on raising HPV vaccination rates. Unfortunately, we found no significant effect of IIS-based C-R/R autodialer calls on HPV vaccine series initiation or completion rates in New York and only small effects in Colorado for 1 or 3 C-R/R calls on vaccine series initiation and completion. In only 1 case (1 C-R/R call in Colorado for vaccine series initiation) did the intervention reach our predetermined minimally significant finding: ≥2-percentage-point rise in vaccination rates.

These largely negative and surprising findings were contrary to our hypotheses, considering the success of this type of C-R/R for childhood vaccinations25  and the generally positive findings of R/R noted in the 2018 Cochrane report on R/R for vaccinations.

We speculate multiple possible explanations. A few practices might have conducted their own R/R for HPV vaccinations. Some IIS HPV vaccine data might have been inaccurate because of underreporting to the IIS; this would blunt but not eliminate the ability to demonstrate a true impact of IIS C-R/R. Similarly, some patient telephone numbers might have been inaccurate, although IIS data (including phone numbers) are updated with each vaccination, including influenza vaccines. Unfortunately, we had no method to verify accuracy of phone numbers, and few patients opted out of phone calls. Overall, 88% of New York practices and 50% of Colorado practices uploaded to IISs via their electronic health records, but we still cannot calculate the degree to which practices uploaded changes in phone numbers. This might reduce but should not eliminate the ability to detect a true impact of IIS-based C-R/R. Of note, because we excluded patients who had completed their HPV series, we expected overall rates to be much lower than state-reported rates in the National Immunization Survey-Teen.

A third possibility is that autodialer phone calls are simply not as effective today as phone R/R appears to have been in the past. We speculate 2 possible reasons: autodialers may be less impactful than traditional phone calls, and autodialers themselves may have diminished effectiveness recently. There has been a marked rise in the number of autodialer calls42  and spam phone calls43  to phones from a variety of sources. Many people screen calls and delete any calls from unknown phone numbers. It is possible that amid the “sensory overload” of autodialer calls and general information overload in today’s society, autodialer R/R may have lost its effectiveness for vaccinations or at least for HPV vaccination.

A fourth possibility is that IIS-based C-R/R by using autodialers does not work for the HPV vaccine specifically. Several previous studies revealed small effects of C-R/R conducted by health systems,3335  and 1 New York IIS study found a 2-percentage-point improvement due to mailed C-R/R messages.38  However, mailed reminders are far more expensive and less scalable than autodialer C-R/R; our goal was to develop a highly scalable intervention.

Because studies have revealed that a strong provider recommendation is important in uptake of the HPV vaccine,30  we included the primary care practice’s names on all autodialer calls from New York and on 61% of calls from Colorado. However, because of legal limitations (needing the practice’s permission to use phone numbers) and technical realities (inserting patient-specific practice phone numbers into autodialer calls), we were unable to display the phone numbers of the adolescents’ doctors’ offices as the source of the C-R/R calls; the phone number appeared as a toll-free number. A phone number from the doctor’s office (which likely would have been recognized by many recipients) might have had a greater impact.

Another likely factor is vaccine hesitancy,3032  which is experienced by many clinicians. It is possible that the effect of HPV vaccine hesitancy outweighs any possible effect of autodialer reminders. However, we suspect this is not the only answer because we found no or little impact of C-R/R calls on subsequent HPV doses and for older adolescents (groups for which hesitancy is less likely). Finally, we do not have any explanation for why we found no impact in New York but a small impact in Colorado for both vaccine series initiation and completion.

Our study has several strengths, including its design as a large 2-state pragmatic trial with rigorous sampling methods to identify practices and patients and analyses that accounted for clustering within practices and several potential confounders. By randomly assigning patients within practices, we limited confounding at the practice level and improved the study’s power to detect intervention effects. Study limitations include generalizability from these 2-state–level studies, inability to determine the accuracy of phone numbers or the potential level of underreporting of HPV vaccine data in the IISs, and inability to examine in depth the reasons for no or little impact of IIS-based C-R/R.

In these 2 states, IIS-based C-R/R autodialer messages for HPV vaccination were not effective in raising HPV vaccination rates in New York and were only slightly effective for vaccine series initiation and completion in Colorado. Because C-R/R is a low-cost, scalable intervention, further study is needed to determine if IIS-based C-R/R should be part of our nation’s strategy to raise HPV vaccination rates.

Dr Szilagyi conceptualized the design of the study, drafted the manuscript, and led the review and revision of the manuscript; Dr Kempe conceptualized the design of the study and reviewed and revised the manuscript; Ms Albertin, Mr Gurfinkel, Ms Valderrama, Ms Breck, and Drs Rand, Schaffer, and Humiston provided substantial contributions to the conception, design, and interpretation of the work and revised the manuscript; Dr Tseng, Mr Zhou, Mr Vangala, Ms Beaty, and Dr Rice performed all data analyses, contributed substantially to the interpretation of the data, and revised the manuscript; Drs Campbell and Whittington performed the cost analyses associated with the intervention and revised the manuscript; Ms Meldrum and Ms Roth led the data acquisition process (vaccine and demographics) from their respective state immunization information system and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

This trial has been registered at www.clinicaltrials.gov (identifiers NCT03057379 and NCT0299396).

Deidentified individual participant data will not be made available.

FUNDING: Funded by the National Cancer Institute of the National Institutes of Health under award R01CA187707. The content is solely the responsibility of the authors and does not necessarily represent the official views of the National Institutes of Health. Funded by the National Institutes of Health (NIH).

     
  • CI

    confidence interval

  •  
  • CIIS

    Colorado Immunization Information System

  •  
  • C-R/R

    centralized reminder and recall

  •  
  • HPV

    human papillomavirus

  •  
  • IIS

    immunization information system

  •  
  • NYSIIS

    New York State Immunization Information System

  •  
  • RCT

    randomized controlled trial

  •  
  • RR

    risk ratio

  •  
  • R/R

    reminder and recall

1
Jemal
A
,
Simard
EP
,
Dorell
C
, et al
.
Annual report to the nation on the status of cancer, 1975–2009, featuring the burden and trends in human papillomavirus (HPV)-associated cancers and HPV vaccination coverage levels
.
J Natl Cancer Inst
.
2013
;
105
(
3
):
175
201
2
Dunne
EF
,
Unger
ER
,
Sternberg
M
, et al
.
Prevalence of HPV infection among females in the United States
.
JAMA
.
2007
;
297
(
8
):
813
819
3
Zandberg
DP
,
Bhargava
R
,
Badin
S
,
Cullen
KJ
.
The role of human papillomavirus in nongenital cancers
.
CA Cancer J Clin
.
2013
;
63
(
1
):
57
81
4
Centers for Disease Control and Prevention (CDC)
.
Human papillomavirus-associated cancers - United States, 2004-2008
.
MMWR Morb Mortal Wkly Rep
.
2012
;
61
:
258
261
5
Van Dyne
EA
,
Henley
SJ
,
Saraiya
M
,
Thomas
CC
,
Markowitz
LE
,
Benard
VB
.
Trends in human papillomavirus-associated cancers - United States, 1999–2015
.
MMWR Morb Mortal Wkly Rep
.
2018
;
67
(
33
):
918
924
6
Markowitz
LE
,
Dunne
EF
,
Saraiya
M
,
Lawson
HW
,
Chesson
H
,
Unger
ER
;
Centers for Disease Control and Prevention (CDC)
;
Advisory Committee on Immunization Practices (ACIP)
.
Quadrivalent human papillomavirus vaccine: recommendations of the Advisory Committee on Immunization Practices (ACIP)
.
MMWR Recomm Rep
.
2007
;
56
(
RR
):
1
24
7
Insinga
RP
,
Dasbach
EJ
,
Elbasha
EH
.
Assessing the annual economic burden of preventing and treating anogenital human papillomavirus-related disease in the US: analytic framework and review of the literature
.
Pharmacoeconomics
.
2005
;
23
(
11
):
1107
1122
8
Stokley
S
,
Szilagyi
PG
.
Improving human papillomavirus vaccination in the United States: executive summary
.
Acad Pediatr
.
2018
;
18
(
2S
):
S1
S2
9
Markowitz
LE
,
Gee
J
,
Chesson
H
,
Stokley
S
.
Ten years of human papillomavirus vaccination in the United States
.
Acad Pediatr
.
2018
;
18
(
2S
):
S3
S10
10
Reiter
PL
,
Gerend
MA
,
Gilkey
MB
, et al
.
Advancing human papillomavirus vaccine delivery: 12 priority research gaps
.
Acad Pediatr
.
2018
;
18
(
2S
):
S14
S16
11
Walker
TY
,
Elam-Evans
LD
,
Yankey
D
, et al
.
National, regional, state, and selected local area vaccination coverage among adolescents aged 13–17 years - United States, 2017 [published correction appears in MMWR Morb Mortal Wkly Rep. 2018;67(41):1164]
.
MMWR Morb Mortal Wkly Rep
.
2018
;
67
(
33
):
909
917
12
Szilagyi
PG
,
Bordley
C
,
Vann
JC
, et al
.
Effect of patient reminder/recall interventions on immunization rates: a review
.
JAMA
.
2000
;
284
(
14
):
1820
1827
13
Jacobson Vann
JC
,
Szilagyi
P
.
Patient reminder and patient recall systems to improve immunization rates
.
Cochrane Database Syst Rev
.
2005
;(
3
):
CD003941
14
Jacobson Vann
JC
,
Jacobson
RM
,
Coyne-Beasley
T
,
Asafu-Adjei
JK
,
Szilagyi
PG
.
Patient reminder and recall interventions to improve immunization rates
.
Cochrane Database Syst Rev
.
2018
;(
1
):
CD003941
15
Task Force on Community Preventive Services
.
The Guide to Community Preventive Services
.
Atlanta, GA
:
Centers for Disease Control and Prevention
;
1999
16
National Vaccine Advisory Committee
.
Standards for child and adolescent immunization practices [published correction appears in Pediatrics. 2004;113(1):184]
.
Pediatrics
.
2003
;
112
(
4
):
958
963
17
Tierney
CD
,
Yusuf
H
,
McMahon
SR
, et al
.
Adoption of reminder and recall messages for immunizations by pediatricians and public health clinics
.
Pediatrics
.
2003
;
112
(
5
):
1076
1082
18
Saville
AW
,
Szilagyi
P
,
Helmkamp
L
, et al
.
Potential strategies to achieve universal influenza vaccination for children: provider attitudes in two states
.
Acad Pediatr
.
2018
;
18
(
8
):
873
881
19
Saville
AW
,
Albright
K
,
Nowels
C
, et al
.
Getting under the hood: exploring issues that affect provider-based recall using an immunization information system
.
Acad Pediatr
.
2011
;
11
(
1
):
44
49
20
Kempe
A
,
Wortley
P
,
O’Leary
S
, et al
.
Pediatricians’ attitudes about collaborations with other community vaccinators in the delivery of seasonal influenza vaccine
.
Acad Pediatr
.
2012
;
12
(
1
):
26
35
21
Humiston
SG
,
Rosenthal
SL
.
Challenges to vaccinating adolescents: vaccine implementation issues
.
Pediatr Infect Dis J
.
2005
;
24
(
suppl 6
):
S134
S140
22
Albright
K
,
Saville
A
,
Lockhart
S
,
Widmer Racich
K
,
Beaty
B
,
Kempe
A
.
Provider attitudes toward public-private collaboration to improve immunization reminder/recall: a mixed-methods study
.
Acad Pediatr
.
2014
;
14
(
1
):
62
70
23
Saville
AW
,
Gurfinkel
D
,
Sevick
C
,
Beaty
B
,
Dickinson
LM
,
Kempe
A
.
Provider preferences and experiences with a countywide centralized collaborative reminder/recall for childhood immunizations
.
Acad Pediatr
.
2016
;
16
(
1
):
50
56
24
Kempe
A
,
Saville
A
,
Dickinson
LM
, et al
.
Population-based versus practice-based recall for childhood immunizations: a randomized controlled comparative effectiveness trial
.
Am J Public Health
.
2013
;
103
(
6
):
1116
1123
25
Coleman
K
,
Austin
BT
,
Brach
C
,
Wagner
EH
.
Evidence on the chronic care model in the new millennium
.
Health Aff (Millwood)
.
2009
;
28
(
1
):
75
85
26
Kempe
A
,
Saville
AW
,
Beaty
B
, et al
.
Centralized reminder/recall to increase immunization rates in young children: how much bang for the buck?
Acad Pediatr
.
2017
;
17
(
3
):
330
338
27
Kempe
A
,
Saville
AW
,
Dickinson
LM
, et al
.
Collaborative centralized reminder/recall notification to increase immunization rates among young children: a comparative effectiveness trial
.
JAMA Pediatr
.
2015
;
169
(
4
):
365
373
28
Rand
CM
,
Goldstein
NPN
.
Patterns of primary care physician visits for US adolescents in 2014: implications for vaccination
.
Acad Pediatr
.
2018
;
18
(
2S
):
S72
S78
29
Head
KJ
,
Biederman
E
,
Sturm
LA
,
Zimet
GD
.
A retrospective and prospective look at strategies to increase adolescent HPV vaccine uptake in the United States
.
Hum Vaccin Immunother
.
2018
;
14
(
7
):
1626
1635
30
Dempsey
AF
,
O’Leary
ST
.
Human papillomavirus vaccination: narrative review of studies on how providers’ vaccine communication affects attitudes and uptake
.
Acad Pediatr
.
2018
;
18
(
2S
):
S23
S27
31
Gilkey
MB
,
Calo
WA
,
Marciniak
MW
,
Brewer
NT
.
Parents who refuse or delay HPV vaccine: differences in vaccination behavior, beliefs, and clinical communication preferences
.
Hum Vaccin Immunother
.
2017
;
13
(
3
):
680
686
32
Perkins
RB
,
Fisher-Borne
M
,
Brewer
NT
.
Engaging parents around vaccine confidence: proceedings from the National HPV Vaccination Roundtable meetings
.
Hum Vaccin Immunother
.
2019
;
15
(
7–8
):
1639
1640
33
Chao
C
,
Preciado
M
,
Slezak
J
,
Xu
L
.
A randomized intervention of reminder letter for human papillomavirus vaccine series completion
.
J Adolesc Health
.
2015
;
56
(
1
):
85
90
34
Szilagyi
PG
,
Albertin
C
,
Humiston
SG
, et al
.
A randomized trial of the effect of centralized reminder/recall on immunizations and preventive care visits for adolescents
.
Acad Pediatr
.
2013
;
13
(
3
):
204
213
35
Rand
CM
,
Brill
H
,
Albertin
C
, et al
.
Effectiveness of centralized text message reminders on human papillomavirus immunization coverage for publicly insured adolescents
.
J Adolesc Health
.
2015
;
56
(
5,
suppl
):
S17
S20
36
Staras
SA
,
Vadaparampil
ST
,
Livingston
MD
,
Thompson
LA
,
Sanders
AH
,
Shenkman
EA
.
Increasing human papillomavirus vaccine initiation among publicly insured Florida adolescents
.
J Adolesc Health
.
2015
;
56
(
5,
suppl
):
S40
S46
37
O’Leary
ST
,
Lee
M
,
Lockhart
S
, et al
.
Effectiveness and cost of bidirectional text messaging for adolescent vaccines and well care [published correction appears in Pediatrics. 2016;138(3):e20161768]
.
Pediatrics
.
2015
;
136
(
5
). Available at: www.pediatrics.org/cgi/content/full/136/5/e1220
38
Coley
S
,
Hoefer
D
,
Rausch-Phung
E
.
A population-based reminder intervention to improve human papillomavirus vaccination rates among adolescents at routine vaccination age
.
Vaccine
.
2018
;
36
(
32, pt B
):
4904
4909
39
Thorpe
KE
,
Zwarenstein
M
,
Oxman
AD
, et al
.
A pragmatic-explanatory continuum indicator summary (PRECIS): a tool to help trial designers
.
J Clin Epidemiol
.
2009
;
62
(
5
):
464
475
40
National Center for Health Statistics
. Table 70. In Health, United States, 2015: With Special Feature on Racial and Ethnic Health Disparities. Hyattsville, MD: National Center for Health Statistics;
2016
. Available at: www.cdc.gov/nchs/data/hus/hus15.pdf#070. Accessed February 27, 2020
41
Janz
NK
,
Becker
MH
.
The health belief model: a decade later
.
Health Educ Q
.
1984
;
11
(
1
):
1
47
42
YouMail
.
Robocall index.
October
2019
nationwide robocall data. Available at: https://robocallindex.com/2019/october. Accessed February 27, 2020
43
Shaban
H.
Nearly half of cellphone calls will be scams by 2019, report says. The Washington Post. September 19,
2018
. Available at: https://www.washingtonpost.com/technology/2018/09/19/nearly-half-cellphone-calls-will-be-scams-by-report-says/. Accessed February 27, 2020

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.