Web-Based Tailored Messaging to Increase Vaccination: A Randomized Clinical Trial

Between April 2016 and June 2019, we conducted a single-site RCT to evaluate the impact of the VAYB on timely childhood vaccination among infants of women recruited during the last trimester of pregnancy. The protocol for the study has been published previously but is summarized below.¹⁹ 

Participants were randomly assigned to 1 of 3 study arms: the VAYB, an untailored version of the intervention (UT), or usual care (UC) only. The intervention was administered at 4 time points across the follow-up: the last trimester of pregnancy or child age <2 months, at child age 4 to 6 months, at child age 10 to 12 months, and at child age 13 to 15 months. Our primary outcome was up-to-date status for vaccines administered from birth to 200 days of age. We hypothesized that infants of participants in the VAYB arm would be more likely to be up to date than infants of participants in the UT or UC arms. This study was approved by the Institutional Review Boards of Kaiser Permanente Colorado (KPCO), University of Colorado, and the University of Michigan.

Participants were recruited from KPCO, a nonprofit integrated health care system that serves ∼5000 pregnant women and 140 000 children across the Denver and Boulder region per year.

Recruitment was conducted in 6-week waves between April 2016 and December 2017. Potential participants were pregnant women identified by using the KPCO electronic health record (EHR). Women had to be in the third trimester of pregnancy, ≥18 years of age, English speaking, and enrolled in the KPCO health plan at the time of recruitment. Pregnant women were excluded if they had a diagnosis of fetal demise, miscarriage, congenital anomaly in the pregnancy, or high-risk maternal condition. Eligible women received a combination of mail, e-mail, and phone calls that directed them to a study registration Web site where they could consent into the study. The recruited woman or their partner could enroll in the study. Study participants were able to enroll from the last trimester of pregnancy to when their child was <2 months of age.

After consent, participants were directed to the study Web site where they created a username and password. They were then administered a validated preintervention survey to assess demographics, vaccination values, intentions to vaccinate, and vaccine attitudes and concerns.^19,20  Vaccine hesitancy was measured with the Parent Attitudes About Childhood Vaccines short (PACV-short), a validated 5-item screening instrument used to assess hesitancy on a scale of 0 to 10.²¹  Participants scoring ≥5 were classified as “vaccine hesitant,” and those scoring <5 were “nonhesitant.” To ensure balance across study arms, random assignment was conducted within the 2 strata of vaccine hesitancy.

We conducted stratified random assignment with a permuted block technique. Participants were assigned to the VAYB, UT, and UC study arms by using a random allocation ratio of 1:1:1. Random assignment was conducted by using the SAS/STAT procedure PROC PLAN (SAS Institute, Inc, Cary, NC). Study participants were not blinded to the assigned study arm. Study investigators and statisticians were blinded to the study arm assignment for analysis.

The intervention was based on a hybrid of the theory of planned behavior²²  and the values-attitude-behavior model,²³  combined with approaches from motivational interviewing and persuasive messaging.^24,25  The values-attitude-behavior model complements the theory of planned behavior by stating that personal values affect attitudes. In contrast to attitudes and beliefs, values are thought to remain stable over time.^26,27  The objective of the intervention was to influence vaccine attitudes and intentions through messages that address each participant’s vaccination attitudes and concerns, framed according to personal values (eg, acknowledging their “right to choose” or desire to “keep their child safe”). We crafted tailored messages that used empathetic language and emphasized autonomy to affirm individual values.²⁸ 

The Web-based tailored intervention was developed by using an iterative, user-driven approach that included surveys, one-on-one interviews, and usability testing. The details of this process have been described previously.¹⁸  Informational content for the intervention was derived from peer-reviewed sources and online materials provided by the Centers for Disease Control and Prevention and the American Academy of Pediatrics.

The messages conveying the information were tailored to each participants’ intention to vaccinate, personal attitudes about vaccination, vaccination values, and the child’s nickname, sex, and age. These data were collected from the preintervention survey, which activated an embedded algorithm to deliver the tailored messaging (Supplemental Information). The details for how the parent vaccine values survey was developed and evaluated have been described elsewhere.²⁰  After participants completed the preintervention survey, they were automatically directed to the VAYB Web site, which was personalized on the basis of their survey responses. Information on the Web site was arranged across 9 clickable tiles. The top 3 tiles were prominently labeled “Just for You” and contained the most highly tailored content that was based on the participants’ vaccination values and top 3 vaccination concerns. The remaining content was lightly tailored on the basis of the participants’ other, less pressing concerns identified by their survey responses. The lightly tailored content did not incorporate vaccination values.

The intervention and surveys were administered again when the child was age 4 to 6, 10 to 12, and 13 to 15 months. The Web site was retailored and refreshed at each time point on the basis of the updated survey responses. Tailoring on attitudes was updated at all 3 follow-up time points, whereas tailoring on values was updated only at the 10- to 12-month time point.

An untailored version of the VAYB Web site was created to isolate the effect of the tailoring. This version (UT) had the same design and factual information as the VAYB Web site, but it was not personalized to the participants’ survey responses, and the content did not change across the time points.

Participants in all 3 study arms were eligible to receive standard pediatric preventive care at KPCO. This consists of scheduled 20-minute well-child visits at 2, 4, 6, and 12 months of age, with an option for a 9-month visit.²⁹  Recommended childhood immunizations are administered at these health supervision visits, and it is standard practice at KPCO to offer parents Vaccine Information Statements relevant to that visit.³⁰ 

Participants in the UT and UC arms were administered the same surveys at the 4 intervention time points as those received by participants in the VAYB arm.

The primary outcome was immunization status assessed over the first 200 days of age to cover a majority of the routinely administered infant vaccines and to minimize loss to follow-up. Immunization data were extracted from the KPCO EHR for the following 6 vaccines recommended by the Advisory Committee on Immunization Practices: hepatitis B, rotavirus, diphtheria-tetanus-acellular pertussis, Haemophilus influenzae type b, pneumococcal conjugate, and inactivated poliovirus. Immunization status was quantified by using days undervaccinated, a continuous metric used to measure the differences between the ages at which the vaccine doses were actually administered and ages at which the doses should have been administered according to the Advisory Committee on Immunization Practices recommendations.²  For these 6 vaccines, days undervaccinated can range from 0 to 648 days.

Days undervaccinated was analyzed as a dichotomous and continuous variable. For the dichotomized measure, infants with 0 cumulative days undervaccinated were considered up to date, and infants with ≥1 days undervaccinated were considered not up to date.

Two secondary outcomes were also assessed. First, we examined up-to-date status for the measles-mumps-rubella (MMR) and varicella vaccines among infants with at least 489 days of follow-up, which is when days undervaccinated for the first doses of these vaccines begin to accrue. Second, we repeated the primary analysis excluding the hepatitis B vaccine from the assessment of up-to-date status. This latter analysis was conducted because the hepatitis B birth dose tends to be administered in the inpatient rather than outpatient setting.

The trial was powered to detect a difference in proportion of up-to-date status between 6.8% and 9.4% between the 3 study arms. This required a sample size of 477 participants for a difference of 9.4% (odds ratio [OR] = 3) and 1002 participants for a difference of 6.8% (OR = 2), assuming a baseline vaccine hesitancy of 15%, a 1:1:1 allocation ratio, 80% statistical power, and 2-sided α = .05.

We conducted a modified intent-to-treat analysis that adhered to study arm assignment but excluded infants if their vaccination data could not be completely ascertained over the 200-day follow-up. Infants were excluded if they died, were not enrolled in KPCO, enrolled in KPCO after 60 days of age, disenrolled from KPCO before the end of follow-up, or were not receiving primary care at KPCO.

For the primary analysis, up-to-date status and days undervaccinated were assessed from birth to 200 days of age. Up-to-date status was analyzed with logistic regression to estimate ORs and 95% confidence intervals (CIs). For days undervaccinated, we used a nonparametric analysis and rank transformation approach because the distribution of days undervaccinated was highly skewed to the right. Days undervaccinated were first ranked in increasing order of magnitude for all infants independent of study arm assignment, and then the mean ranks were compared across the study arms by using a 1-way analysis of variance. As a preplanned subgroup analysis, the data were stratified by baseline vaccine hesitancy status. As a post hoc subgroup analysis, up-to-date status was assessed across the study arms among participants with baseline PACV-short scores between 4 and 7. Participants with these scores may represent vaccine “fence-sitters,” defined as parents who are willing to vaccinate but also have concerns about vaccines.^31,32 

For the secondary analyses, logistic regression was used to assess up-to-date status for the MMR and varicella vaccines among infants with at least 489 days of follow-up and to assess up-to-date status excluding the hepatitis B vaccine. Analyses were conducted with SAS software 9.4.

In the KPCO cohort, there were 4929 pregnant women who were potentially eligible for the study (Fig 1). After exclusions, there were 824 participants enrolled in the study, 795 (96.5%) of whom were recruited during pregnancy, and 29 (3.5%) were recruited when their child was <2 months of age (Table 1). According to the 1:1:1 randomization ratio, 276 were randomly assigned to the VAYB arm, 274 were randomly assigned to the UT arm, and 274 were randomly assigned to the UC arm (Fig 1). Baseline characteristics were evenly distributed across study arms (Table 1). For the entire study population, the mean age at enrollment was 32 years, 85% were white, 12% were Hispanic, and 85.3% were college-educated. At baseline, 14.3% of the participants were categorized as being vaccine hesitant, with a mean hesitancy score of 2.25 (SD = 2.21).

After random assignment, incomplete vaccination data led to 143 (17.4%) exclusions (Fig 1). These exclusions were considered loss to follow-up, which ranged from 13.1% to 19.6% across the study arms (P = .08). The mean vaccine hesitancy scores of the participants lost to follow-up did not differ significantly across the study arms (P = .92).

The proportions of infants who were up to date at age 200 days were 91.4%, 92.9%, and 92.3% for the VAYB, UT, and UC arms, respectively. The 95% CIs for the ORs comparing up-to-date status between study arms all included 1 and were not statistically significant (Table 2). Infants in the VAYB were not more likely to be up to date than infants in the UC arm (OR = 0.89; 95% CI, 0.45–1.76), nor were they more likely to be up to date than infants in the UT arm (OR = 0.82; 95% CI, 0.42–1.63). Similarly, the odds of being up to date did not differ between the UT and UC arms (OR = 1.08; 95% CI, 0.54–2.18). In the overall KPCO cohort of potentially eligible participants who were not enrolled in the study (n = 3366), the proportion of infants who were up to date during the study period was 85.6%.

In the subanalysis stratified by vaccine hesitancy status, infants of vaccine-hesitant parents in the VAYB arm were less likely to be up to date than infants of vaccine-hesitant parents in the UT arm (OR = 0.28; 95% CI, 0.08–0.98) (Table 3). The odds of being up to date did not differ between the VAYB and UC arms (OR = 0.67; 95% CI, 0.23–1.96) or between the UT and UC arms (OR = 2.42; 95% CI, 0.65–9.01). Among infants of parents who were not vaccine-hesitant, the odds of being up-to-date did not differ between the study arms in any of the comparisons (Table 4). In the post hoc subgroup analysis of potential fence-sitters, the OR for up-to-date status comparing the VAYB to UC arm was not statistically significant (OR = 0.69; 95% CI, 0.16–3.01).

For the secondary analysis of MMR and varicella vaccination status at age 489 days, the ORs for up-to-date status comparing the VAYB to UC arm were not statistically significant (OR_MMR = 1.18; 95% CI, 0.49–2.84; OR_varicella = 1.06; 95% CI, 0.41–2.74) (Tables 5 and 6). For the analysis excluding the hepatitis B vaccine, the OR for up-to-date status comparing the VAYB to UC arm was not statistically significant (OR = 0.94; 95% CI, 0.47–1.88).

The mean rank scores for days undervaccinated were 344.3, 338.7, and 340.2 for the VAYB, UT, and UC arms, respectively. These mean rank scores did not differ significantly for any of the study arm comparisons. P values for the VAYB versus UC, UT versus UC, and VAYB versus UT arm comparisons were 0.64, 0.87, and 0.52, respectively (Table 7). In the subgroup analysis stratified by vaccine hesitancy status, the mean rank scores did not differ for 5 of the 6 study arm comparisons (Tables 8 and 9). The only significant difference was between infants of vaccine-hesitant parents in the VAYB and the UT arms is infants of vaccine-hesitant parents in the VAYB arm had significantly more days undervaccinated than those of infants of hesitant parents in the UT arm (mean rank difference = 11.06; P = .03) (Table 8).

In this RCT, provision of a Web-based tailored messaging intervention to expectant and new parents did not have a positive effect on the timely uptake of infant immunizations. Infants of parents in the VAYB arm were not more likely to be up to date for routinely administered vaccines or have fewer days undervaccinated over the first 200 days of life than infants of parents in the UT or UC study arms. These results do not support our hypothesis that the behaviors of parents can be influenced by carefully crafted messages tailored to their vaccination attitudes and values.

In several studies, authors have examined the impact of tailored messaging on vaccine attitudes, intentions, and behaviors related to human papillomavirus and influenza vaccination.^33–38  They tended to focus on Web-based formats or text messaging to deliver the tailored messages, and outcomes were assessed in adolescent girls, parents of adolescent girls, or adults. The results varied, with only 1 RCT revealing a positive effect on both initiation and completion of human papillomavirus vaccination series in adolescent girls.³³  However, it is difficult to relate the VAYB results to these investigations because the study populations, intervention contents, and vaccination outcomes were considerably different. Perhaps most importantly, the other investigations were focused on 1 specific vaccine, whereas the VAYB attempted to address concerns across multiple vaccines.

This study had many strengths. The VAYB, survey instruments, and data collection tools were all developed by using iterative, theory- and user-driven approaches led by a multidisciplinary team.¹⁸  The effectiveness of the intervention was evaluated with a rigorous randomized design, and the study population was recruited during pregnancy, representing a key time period for infant vaccination decisions.^39,40  In addition, rather than relying on recall, vaccination outcomes were extracted from the EHR, which captures vaccine administrations with a high degree of accuracy.^41,42 

The subgroup analysis produced an unexpected result: among vaccine-hesitant participants, the UT arm was significantly more effective than the VAYB arm. Infants in the VAYB arm were significantly less likely to be up to date and had more days undervaccinated than infants in the UT arm. It is possible that our inferences based on the participants’ vaccine values and attitudes overreached and led to messages that were less relevant, thus undermining the credibility of the tailoring.⁴³  In addition, we conducted 9 comparisons in the analysis without a correction for multiple comparisons, suggesting that this subgroup association could be significant by chance alone. Although we did not plan for a Bonferroni multiple comparison test a priori, such a correction would have rendered this association nonsignificant at the α < .05 level.

Although tailored messaging has been shown to be effective at eliciting health behavior change, the magnitude of effect sizes from randomized studies have been small.¹⁵  In integrated health systems such as KPCO, detecting modest effect sizes for infant vaccine acceptance is challenging because vaccination rates in these settings tend to be high.⁴⁴  In our intervention trial, >90% of the infants across all 3 intervention arms were up to date, and 14% of the study participants were classified as vaccine hesitant at baseline. Although this rate of vaccine hesitancy aligns with other published estimates,^4,45,46  it is possible that our intervention could have been effective if it had been implemented in other health care environments with lower infant vaccination rates and higher rates of vaccine hesitancy.

This study also had limitations. The trial was conducted in a single integrated health plan in Colorado, which may have affected the trial’s generalizability. Infants in the trial were more likely to be up to date than infants in the entire KPCO population during the study period (92.2% vs 85.6%), which may have also affected the generalizability of the trial. Overall, the loss to follow-up was 17.4%, largely attributed to parents who disenrolled from KPCO after their child was born. Although this may have also impacted the generalizability of the trial, the loss to follow-up did not differ significantly across the study arms. In addition, the rate of vaccine hesitancy among those lost to follow-up was similar across the arms, suggesting that these exclusions did not affect the internal validity of the results.

We designed the intervention to tailor messages on both vaccine values and attitudes. Although this is a potential strength of the intervention, it also possible that for some participants, the tailored content was on target for one construct but off target for the other construct. Such discrepant effects could explain the null results observed in the primary analysis. However, this could not be formally evaluated because we did not design the trial to evaluate the effects of the 2 types of tailoring separately.

This RCT suggests that Web-based tailored messaging does not positively impact parental immunization behaviors during infancy. Given that vaccine hesitancy remains a significant public health issue,⁴⁷  researchers should continue to develop and test communication approaches to reduce parental vaccination concerns and improve the timely uptake of infant immunizations.

We acknowledge Courtney Kraus and Kathy Gleason who recruited participants and collected data for the trial.

CI

confidence interval

EHR

electronic health record

KPCO

Kaiser Permanente Colorado

MMR

measles-mumps-rubella

OR

odds ratio

PACV-short

Parent Attitudes About Childhood Vaccines short

RCT

randomized clinical trial

UC

usual care intervention

UT

untailored intervention

VAYB

Vaccines and Your Baby intervention