Patient Portal Reminders for Pediatric Influenza Vaccinations: A Randomized Clinical Trial

The study was conducted from October 1, 2019, to March 31, 2020. We evaluated 3 interventions in an intention-to-treat randomized clinical trial of 22 046 patients ages 6 months to <18 years randomly assigned within primary care practices:

1. Reminders for the first influenza vaccine. Patients were randomly assigned to 1 of 3 study arms: no reminder (n = 7391), 3 gain-frame reminders (n = 7318), or 3 loss-frame reminders (n = 7337).

2. Precommitment message. Patients were randomly assigned to 1 of 2 study arms: no precommitment message (n = 10 978) or 1 precommitment message (n = 11 068).

3. Reminders for the second influenza vaccine. Patients ages 6 to <36 months overdue for a second influenza vaccine were randomly assigned to 1 of 2 study arms: no reminder (n = 347) or 1 reminder (n = 342).

The University of California, Los Angeles, (UCLA) Institutional Review Board approved this study and issued a waiver of patient consent. This study followed the Consolidated Standards of Reporting Trials reporting guideline for randomized clinical trials.

P arents or guardians of patients <12 years of age can register for proxy access to their child’s myUCLAhealth (Epic Systems Corporation, Verona, WI) portal. Adolescents 12 to <18 years can register for full access to the portal, and their parents or guardians can register for limited proxy access. We previously reported on the portal usage by adolescents and proxies in our health system.²²  The pediatric study was conducted as part of a larger set of interventions that included both children and adults. For this study, we included all UCLA Health pediatric primary care patients, defined as children from the age of 6 months to <18 years with ≥2 primary care physician visits (by evaluation and management office codes) within 3 years or ≥1 primary care physician visits with a preventive service code within 1 year. We also included managed care patients assigned to UCLA Health (regardless of the number of previous visits) because their assignment established their primary care affiliation with the practice. Finally, we identified active portal users, defined as patients or proxies who signed up for and logged into the portal at least once during the previous 12 months, not including the initial portal login (Fig 1).

The setting was all 53 UCLA primary care practices.

Patients randomly assigned to receive a portal message were sent a notification by e-mail or text (per patients’ portal preference settings) informing them that “a message from your doctor” had been posted on the portal. For patients ≥12 years of age, messages were sent to both the patient and parent and/or proxy. Patients and proxies had to login to the portal to read the message.

The portal reminder message contents were grounded in the health belief model and principles of health literacy^23,24  as well as incorporating behavioral science principles.^17–21  The messages were in English, included the name of the patient’s primary care physician, and had a below–seventh-grade reading level per Flesch-Kincaid analysis.

Gain-frame messages described the positive consequences of receiving an influenza vaccine, whereas loss-frame messages described the negative consequences of not receiving a vaccine. All messages incorporated additional behavioral science principles including scarcity²¹  (eg, “time is running out”), appeal to authority (“UCLA doctors and the American Academy of Pediatrics strongly recommend”),²¹  and commission²⁵  (“choosing to vaccinate”; Table 1). Patients in the reminder message groups received 1 message per month (October, November, and December) if their EHR indicated that they were still due for an influenza vaccine. The messages also included a questionnaire allowing patients to report whether they had received an influenza vaccine outside of the UCLA Health system.

Those in the precommitment intervention group received a message in September prompting them to answer a question to let their provider know whether they were planning to get an influenza vaccine during the upcoming influenza season (Table 1).

For the reminders for the second dose of the influenza vaccine, we included only patients 6 to <36 months old and not 3 to 8 years old, because of concern that the health system medical records might be missing influenza vaccinations from previous years administered outside of our health system. Patients 6 to <36 months of age eligible for a second influenza vaccine who had no record in the EHR of a second influenza vaccination by January were randomly assigned to 2 groups: a single reminder message sent in January or no reminder message.

P atient factors, obtained from the EHR, included age, sex, insurance at the last primary care visit, race and ethnicity, and receipt of ≥1 influenza vaccinations during the previous 2 years.

Influenza vaccinations administered at any UCLA Health site were recorded in the EHR. In addition, UCLA Health practitioners could enter vaccination records manually for vaccines received outside UCLA Health. The UCLA Health system merges data into the EHR from external sources, including Surescripts (pharmacy benefits manager), the California Immunization Registry, and Care Everywhere (Epic’s information exchange application). Patients or proxies could also enter vaccination data when they logged in to the portal or via a link included in the portal reminder letters.

The primary study outcome was documentation in the EHR of ≥1 influenza immunizations, whether from internal health system sources or merged external sources, between October 1, 2019, and March 31, 2020. To eliminate bias in outcome ascertainment, the primary analysis excluded vaccination only reported by patients or proxies in response to portal reminders because the control group did not have this prompt to self-report.

We assessed secondary outcomes related to the receipt of ≥1 influenza vaccinations during the season: (1) vaccinations in predetermined subgroups (age, sex, insurance, and receipt of influenza vaccination in previous 2 years) and (2) vaccination rates including self-reported vaccinations.

An additional secondary outcome was the documentation in the EHR of 2 influenza immunizations by April 1, 2020, for eligible patients 6 to <36 months old that had not completed the second immunization by January.

We assessed the percentage of patients who opened the portal reminder letters and source of influenza vaccination data.

The sample size of 7349 per reminder arm for the first influenza vaccination provided 80% power to detect a 2.6% increase in vaccination rates at a 0.025 2-sided significance level. A sample size of 11 023 per precommitment arm provided 80% power to detect a 1.9% increase in vaccination rates at a 0.05 2-sided significance level. These calculations assumed a control group vaccination rate of 50% (most conservative) and a Pearson’s χ² test.

We anticipated that the study would be underpowered to detect changes in the second vaccination rates because of small sample size, so we considered this an exploratory outcome.

We identified the primary care practice most recently visited and attributed the patient to that practice. W e grouped patients into family units using a matching algorithm that included each patient’s telephone number, address, and insurance member number or patient guarantor identification. A statistician (S.V.) randomly selected 1 index patient per family for inclusion in the analyses. These patients were randomly assigned within practices to each of the study arms. Other study personnel were blinded to the patient allocation. The final sample comprised 22 046 (42.4%) of the 52 056 UCLA Health pediatric primary care patients.

We report descriptive statistics for all patient characteristics in terms of frequency distributions for the full patient sample, stratifying by study arm. In the primary analysis, we compared vaccination rates between study arms using mixed-effects Poisson regression with robust SEs.^26,27  The model included fixed effects for reminder arm (no reminder versus gains-framing versus loss-framing), precommitment arm (no precommitment versus precommitment), random practice effects, and adjustment for patient characteristics (age, sex, insurance, and vaccination history). Race and ethnicity were not included as covariates in any of the analyses because of concerns about the completeness and accuracy of these data in the EHR. Differences between study arms were summarized by using risk ratios and 95% confidence intervals (CIs). Stratified analyses were performed by refitting the model within subgroups defined by patient characteristics.

For the second vaccine-dose reminder intervention, a similar modeling approach was used. In this analysis, no adjustments were made for clustering by practice, given the small sample size, and the analysis did not adjust for vaccination history because of the sample being limited to young children.

A significance level of 0.025 was used for comparing each of the reminder arms with the no-reminder arm in the primary analysis (twofold Bonferroni correction), and a significance level of 0.05 was used for all other analyses. All analyses were performed by using SAS statistical software, version 9.4 (SAS Institute, Inc, Cary, NC).

A total of 22 046 patients were randomly allocated to 1 of 3 groups in the first-dose vaccine reminders and 1 of 2 groups in the precommitment message reminders. For the second-dose vaccine reminders, a total of 698 patients were randomly assigned to treatment or control groups.

A total of 27.0% of patients and proxies in the first vaccine reminder group opened at least 1 reminder letter, 15.0% opened at least 2, and 9.7% opened all 3 letters. A total of 6.9% patients and proxies opened the precommitment letter.

In Table 2, we provide demographic characteristics in each of the study arms.

Across all patients, the vaccination rates for ≥1 influenza vaccines, excluding self-reported vaccinations, were 56.9% in the control group (no reminders), 58.0% in the loss-frame reminders group (P = .07), and 58.0% in the gain-frame reminder group (P = .47). For the precommitment message intervention, vaccination rates were 57.0% in the control group (no reminders) and 58.3% in the precommitment message group (P = .11).

Adjusted risk ratios were 1.02 (95% CI: 1.00–1.04) for the loss-frame reminders, 1.01 (95% CI: 0.98–1.05) for the gain-frame reminders, and 1.02 (95% CI: 1.00–1.04) for the precommitment messages (Table 3).

Among eligible 6- to <36-month-old infants, second-dose vaccination rates were 44.1% in the control (no reminder) group (n = 347) and 55.0% in the reminder group (n = 342), with an adjusted risk ratio of 1.25 (95% CI: 1.07–1.45), revealing a large and statistically significant effect for second vaccine reminders (Table 4).

For subgroups, including age, sex, insurance and previous influenza vaccination, the intervention had small or no effects on influenza vaccination rates (Table 5).

Overall, 10 634 vaccinations (83.7%) were administered within UCLA clinics, whereas 2076 (16.3%) were administered outside UCLA and recorded in external data, such as the California Immunization Registry.

When we included self-reported data about influenza vaccinations, the vaccination rate was 57.0% for controls versus 58.9% for the loss-frame reminder group (P < .01) and 58.7% for the gain-frame reminder group (P = .15).

In this study, we found no significant increases in vaccination rates after portal reminders for the first influenza vaccine, with vaccination rates of 56.9% in the control group (no reminders), 58.0% in the loss-frame portal reminders group (P = .07), and 58.0% in the gain-frame reminder group (P = .47). Additionally, a portal-based precommitment survey, intended to capitalize on the principle that individuals seek a consistent and continuous self-image, did not significantly increase vaccination rates, with 57.0% receiving ≥1 vaccines in the control group and 58.3% in the precommitment group (P = .11). In our previous work, we found limited effectiveness of a generic influenza vaccine reminder.¹⁶  In this study, despite the implementation of a message targeted specifically at pediatric patients and the use of additional behavioral science concepts to enhance message effectiveness, the portal reminders for the first influenza vaccine had no significant impact on immunization rates. Some have speculated that loss-frame messages may be more persuasive in the context of childhood vaccinations because they may possibly arouse emotions of guilt or regret in the parent,²⁰  but we found no significant effect, regardless of framing.

Likewise, the precommitment message did not increase vaccination rates. Plausible reasons for the lack of efficacy include the low percentage (6.9%) of patients and proxies that read the precommitment message and the private nature of the precommitment responses. Commitments that are made publicly, for example, may be more effective, because the individual seeks to avoid reputational damage,²⁸  a dimension we were unable to incorporate in our intervention.

In a commentary accompanying the article of our previous portal message intervention, the authors applied the Green and Kreuter precede-proceed model for health behavior change to suggest that portal messages may be ineffective if they do not sufficiently address predisposing factors (eg, vaccine hesitancy), enabling factors (eg, access barriers), and reinforcing factors (eg, incentives to pursue vaccination).²⁹  These factors may have limited the effectiveness of the interventions presented here as well. With our interventions, we did not directly address vaccine hesitancy. With recent nationally representative samples of US parents, researchers found high levels of hesitancy for influenza vaccines, ranging from 20% to 25%.^30,31  Addressing hesitancy directly may be critical in increasing influenza vaccination rates. With our findings, we suggest that efforts that focus solely on optimizing and reframing the portal message itself are probably not sufficient to increase message effectiveness.

In contrast to the modest efficacy of vaccine reminders for the first influenza vaccine, the reminders for children overdue for a second dose of the vaccine were highly effective, with second-dose vaccine rates of 44.1% in the control group and 55.0% in the reminder group, with an adjusted risk ratio of 1.25 (95% CI: 1.07–1.45). These messages were sent to a group of parents who were unlikely to be vaccine-hesitant or have insurmountable access barriers, given that their children had already received the first dose of the vaccine. The contrast between the effectiveness of first-dose and second-dose reminders suggests that vaccine hesitancy played an important role in reducing the effectiveness of the first-dose reminders. Second-dose reminders possibly also addressed health literacy gaps for parents who, perhaps, did not know about the need for a second influenza vaccine. In a study of text message reminders for the second influenza vaccine in a low-income minority population, researchers found increases in rates of the second dose of the influenza vaccination, particularly for text messages that specifically addressed health literacy.³² 

Our study has several limitations. As a single health system intervention, it potentially lacks generalizability to other health care systems, particularly those serving larger underserved populations or populations with limited English proficiency. An inherent limitation of portal-based interventions is that they are targeted at only active portal users, narrowing their impact and, potentially, leading to disparities if portal usage varies by demographic group. A notable finding of our study was the low rates of reading of the portal messages, which may be due, in part, to the particular implementation of our intervention but may also reflect a more pervasive challenge for portal interventions. Optimizing portal message reading is an area ripe for further work and may include innovations such as more seamless integration of texting and more robust usability testing.

In a large health system, patient portal reminders for influenza vaccines in children were ineffective for the first influenza vaccine but highly effective for the second dose of the vaccine. To increase overall pediatric immunization rates, in future strategies using the patient portal, researchers likely need to directly address influenza vaccine hesitancy or include efforts that address other barriers to vaccination.

We are grateful for the expert contributions of the UCLA Health Information Services and Solutions team, in particular, Alexander E. Kurpinski, Ashley E. Turner, Deidre L. Keeves, Desmond Kelly, Eric M. Cheng, and Michael A. Pfeffer.

CI

confidence interval

EHR

electronic health record

UCLA

University of California, Los Angeles