Impact of a Remote Virtual Reality Curriculum Pilot on Clinician Conflict Communication Skills

This randomized controlled pilot study of a VR training intervention was conducted at our institution’s satellite campus, an inpatient and outpatient facility with a 42-bed inpatient unit with services for HM, adolescent medicine, pulmonary, gastroenterology, pediatric surgery, and oncology. Eligible participants included HM clinicians who work at the satellite campus (33 physicians, 9 fellows, and 9 nurse practitioners). This study was deemed exempt by our institution’s Institutional Review Board. Participant consent was obtained electronically before enrollment as per Institutional Review Board approval of this study.

CIVIC consisted of 2 screen-based VR simulated scenarios and a brief didactic. Participants accessed the curriculum via a teleconferencing network, after which they were greeted by a facilitator (L.E.H.) who provided an orientation to the experience. The facilitator used the screen share function of the teleconferencing network to switch between the virtual environment and a didactic presentation. The virtual environment was designed to replicate an inpatient room and allowed for real-time verbal interaction with avatars (ie, graphical character representatives) (Supplemental Information 1). Participants could engage in dialogue with a nurse avatar and could examine an infant patient avatar. CIVIC began with immediate participation in a VR simulation in which participants managed conflict with a bedside nurse avatar surrounding care decisions for a patient with bronchiolitis. Specifically, the nurse avatar requested albuterol in the setting of no obstructive breathing,²¹  an intervention that was not clinically indicated. The participant verbally communicated with the nurse avatar who responded in real-time according to a structured facilitation algorithm. After completion of the first simulation, the facilitator reviewed evidence-based cooperative conflict management techniques,^5,22–27  with opportunities for participant reflection on performance, before engagement in a second VR simulation to practice skills. In the second VR scenario, the nurse avatar requested an escalation of care for a child without severe respiratory distress. The curricular approach supported Kolb’s experiential learning theory by providing a concrete experience, opportunity for reflection, and subsequent active experimentation to promote learning (Fig 1).¹⁷ 

The VR simulations were adapted from a previously developed VR simulation on respiratory distress in infants with bronchiolitis.²⁸  Avatar dialogue was programmed based on interviews with satellite campus nurses and clinicians who had prior experiences with conflict in the clinical setting. Simulations were developed in an iterative method by pediatricians with experience on the HM service and expertise in medical education and simulation. CIVIC was piloted with 4 chief residents who serve as attending physicians on the HM service but were not eligible to participate in the study and revised based on feedback before implementation.

At time of enrollment, participants were randomized based on clinician role (e.g., physician, fellow, or nurse practitioner) into the intervention or control groups. All participants underwent a 30-minute didactic training on conflict management at a divisional meeting that was disseminated electronically to those unable to attend. The intervention group completed CIVIC training, whereas the control group underwent an alternative VR training on addressing influenza vaccine hesitancy¹⁷  to ensure comparable baseline exposure to the VR platform. Participants completed their allocated VR intervention between November 2020 and February 2021. Approximately 3 months following completion of their allocated VR training, all participants completed a unique standardized VR simulation focused on interprofessional conflict management. We chose 3 months for follow-up based on prior work looking at knowledge decay in adult learners.^29,30  In this simulation, the nurse avatar requested initiation of high-flow nasal cannula for a patient with bronchiolitis demonstrating signs of only mild to moderate respiratory distress. This follow-up performance assessment was purposely designed to allow for demonstration of evidence-based cooperative conflict communication techniques. The follow-up assessments were recorded to allow subsequent appraisal of skills via a performance rubric. Participants received a $50 gift card for completing all elements of the study protocol.

We established validity evidence for our assessment tool according to Messick’s framework, a well-known standard for construct validity.³¹  We conducted a literature review using PubMed and PsychInfo databases to identify existing conflict communication assessment tools. Published tools were reviewed to identify objective, observable items for inclusion in our assessment rubric by mapping to curriculum objectives.^24,32–35  The assessment rubric was piloted using “think-aloud protocol”³⁶  for response process validity and iteratively revised by medical educators and practicing hospitalists (content evidence). Additionally, the rubric’s use for formative feedback is lower stakes than an evaluative assessment (consequences evidence). A global entrustment scale (GES) adapted from the Ottawa Surgical Competency Operating Room scale³⁷  was used to allow holistic assessment of participants’ performances. The final assessment rubric contained 12 items representing observable cooperative conflict management behaviors (Supplemental Information 2).

The recorded follow-up performance assessments were independently reviewed using the conflict communication assessment rubric by 2 blinded raters with advanced medical education and simulation training (K.J., M.Z.). These raters were trained in the application of the rubric using scripted VR recordings, with the primary investigator (L.E.H.) serving as the “gold standard” in application of the rubric. Raters were then trained in an iterative fashion with 4 randomly selected recorded performance assessments until they were consistent with the gold standard. For appraisal of the follow-up assessments, raters reviewed recordings in batches of 5, after which the gold standard supplied them with a breakdown of differences in scoring to identify drift and address it before proceeding to the next batch of videos. No previously submitted scores were changed as a consequence of these reviews.

For follow-up performance assessment, agreement between the 2 raters for dichotomous observed or not observed behaviors was evaluated using the Cohen’s κ statistic. For ordinal scored behaviors, agreement was evaluated using the weighted κ statistic. Strength of agreement based on the κ statistics was classified as poor (<0.20), fair (0.21–0.40), moderate (0.41–0.60), good (0.61–0.80), and very good (0.81–1.00).³⁸  Items with an interrater agreement less than moderate (κ < 0.41) or with 100% positive percent agreement were excluded from subsequent analyses comparing differences between intervention and control groups because the reliability of the items with less than moderate agreement is questionable and items with 100% positive percent agreement have no variance. Differences between intervention and control groups on scored behaviors were evaluated using generalized linear models with a generalized estimating equation to account for 2 raters per participant. Ordinal behaviors were dichotomized, and a logistic regression model was specified in this modeling framework for all dichotomized behavior measures. Model estimated probabilities of observing each behavior are presented by intervention and control group.

Surveys were administered via Research Electronic Data Capture, a secure Web-based application,³⁹  at 3 time points: before curriculum participation (presurvey), immediately after VR curriculum participation (postsurvey), and 3 months after curriculum participation just before the follow-up performance assessment (long-term survey). Surveys assessed participant perceived self-efficacy to manage interprofessional conflict via relevant items from the Student Interprofessional and Interpersonal Communication Self-Efficacy Beliefs questionnaire (SIICSB), a previously published scale.⁴⁰  Fifteen items relating to interprofessional communication (e.g., encouraging open communication and active listening), collaboration (e.g., responding to and giving feedback, developing interdependent relationships), and conflict management (e.g., overcoming communication barriers, communicating respectfully) were included. In addition, after the follow-up performance assessment, participants completed questions from the MEC spatial presence questionnaire,⁴¹  a scale to assess immersion in a VR environment. SIICSB and MEC spatial presence questionnaire survey questions used a 5-point Likert scale (1 = strongly disagree, 5 = strongly agree). Lastly, participants were asked to report the presence and severity of side effects from participation in VR (e.g., blurred vision, nausea, headache).⁴²  Demographic data including provider role, age, gender, race and ethnicity, years since terminal degree, and prior exposure to conflict management training or VR were obtained at the time of consent. All survey data were deidentified, coded, and entered by a research assistant into Research Electronic Data Capture (Fig 2).

For categorical survey data, descriptive statistics in the form of frequencies, median, and ranges were calculated. Self-efficacy measures were dichotomized at the top-box (Likert score of 5 – strongly agree). Differences in the dichotomized self-efficacy measures within and between groups were assessed with paired McNemar’s and independent χ² tests, respectively. Statistical significance was established at P < .05. All analyses were conducted using SAS software, version 9.4 (SAS Institute, NC, USA).

Forty of 51 eligible participants (78%) completed this study. Most participants reported no prior conflict management training or exposure to VR (Table 1). Participants reported a high degree of immersion (Table 2). Adverse side effects were uncommon (n = 5, 6%) and included difficulty concentrating (mild: 2), fatigue (mild: 1), and stomach awareness (mild: 2).

In terms of performance on the follow-up assessment, the intervention group (n = 17) was rated more highly on the GES (P = .003) and demonstrated better performance at summarizing the team member’s concerns (P = .02) and explicitly checking for acceptance of the plan (P = .02) when compared with the control group. Checking for agreement with the assessment (P = .06) and identifying shared interests (P = .06) showed improved performance for the intervention group, though this was not statistically significant (Table 3).

Participants in the intervention group reported improved self-efficacy on the postsurvey for ability to effectively overcome communication barriers (P = .03) and develop positive interdependent relationships with other team members (P = .046). Items with reported improvement on the postsurvey that were sustained on the long-term survey included ability to respond to feedback in a professional manner (P = .01), give feedback in a respectful manner (P = .05), encourage open communication (P = .046), and respectfully convey opinions (P = .01).

Significant differences in self-efficacy between the intervention and control groups were seen on the postsurvey for ability to respond to feedback in a professional manner (P = .047), overcome communication barriers (P = .02), give feedback in a respectful manner (P = .04), and respectfully convey opinions (P = .04). The only statistical difference between groups on the postsurvey that was sustained on the long-term survey was related to the ability to encourage open communication (P = .04) (Supplemental Table 4).

We successfully implemented a novel VR-based communication curriculum to support experiential learning on mitigating and addressing interprofessional conflict. CIVIC was well-tolerated, with participants reporting a high degree of immersion. In this pilot study, exposure to CIVIC was associated with enhanced conflict communication skills when compared with a control group, as measured via a standardized VR simulation. Specifically, individuals who participated in CIVIC demonstrated greater performance on the global entrustment scale in addition to specific observable skills, such as summarizing the concern and assessing for acceptance of the proposed plan, when compared with individuals that underwent didactic training without the opportunity to practice conflict communication skills in the VR environment. In addition, the intervention group reported increased self-efficacy in one-third of the domains related to interprofessional communication.

Team conflict trainings have been previously described in the literature, though many are focused on undergraduate or graduate medical education,^{23,33,43–45}  are time- or resource-intensive (i.e., lengthy workshops, use of standardized patients),^{33,34,43–46}  or do not address unique aspects to interprofessional teamwork, such as trainings that focus on physician-patient or physician-physician dyads.^33,44  In addition to being the first to leverage VR, our curriculum was unique in its focus on practicing clinicians (with on average 6 years in practice), the majority of whom reported no prior formal conflict training. As such, clinicians may have developed their own processes and styles for communication and managing conflict, without the guidance of evidence-based practices or real-time feedback and reflection on performance. Although our pilot intervention was associated with significant differences between our intervention and control groups, our lack of impact on all behaviors assessed may be because of certain ingrained behaviors becoming established over time through real-world clinical practice. Alternatively, behaviors not impacted by the CIVIC curriculum may be more challenging to influence through a single training experience.

We assessed conflict management performance using an assessment rubric derived from existing conflict and communication assessment tools and mapped to our curriculum’s objectives. We focused on framing conflict management through the lens of Alper et al.’s theory on cooperative approaches to conflict,⁵  which informed the items on our assessment rubric. The rubric underwent extensive revision after piloting with medical education experts and hospitalists. However, several items on the assessment rubric were excluded from analysis because of low interrater reliability. Our raters underwent rater training and were provided coaching on areas of discrepancy to mitigate drift. Items with low interrater reliability may reflect the complexity of these skills, highlighting the difficulty in recognizing nuances of specific communication skills. As participants in the intervention group had better global assessment scores, yet inconsistently demonstrated improved performance on specific conflict communication skills, it remains unclear if specific skills lend themselves to successful conflict management more than others. This represents an area for future study.

Although our study supports the use of VR in continuing medical education, a question remains as to the most impactful time in professional development for communication trainings such as CIVIC. There is growing evidence of the impact of residency training on future medical practice.⁴⁷  Thus, communication trainings that provide an evidence-based skillset early in training may have more immediate and long-lasting impact on behavior to support positive interactions among interprofessional teams. By providing a safe and immersive environment to practice foundational skills, VR may be an adjunct to current educational strategies to support acceleration of competency among trainees or junior faculty. It specifically allows for efficient progression through the experiential learning cycle by providing an experience, opportunity for reflection, and subsequent experimentation with continued engagement in the immersive simulation. Additionally, VR may allow for purposeful tailoring of demographic factors such as gender, age, and/or race that may contribute to interprofessional conflict.

This study has several limitations. First, it was a pilot study with a small number of HM clinicians at a single site, which may limit generalizability of results to other institutions or specialties. Additionally, we included clinicians of different training levels and backgrounds to enhance our generalizability, but we did not have sufficient sample size to fully explore differential impacts or subgroup analyses of this intervention on these factors. This is an area for future study. However, given the demonstrated impact on clinicians’ behaviors among this small sample, further study focused on refining the curriculum and extending to additional clinical contexts is warranted. Second, our assessment rubric was created specifically for this study; however, we established validity evidence for our tool according to Messick’s framework.³¹  Several items on our assessment rubric demonstrated low interrater agreement and were not included in subsequent analysis. Low agreement likely occurred because of the complexity of certain conflict communication skills limiting reliable assessment despite coaching. This is an opportunity for continued refinement of our tool to ensure that items deemed critical for effective conflict management are appropriately observable and objective to support reliable assessment. Third, we used VR performance as a surrogate for behavior change and did not assess conflict communication skills in real practice. Evaluation of participants’ behaviors through direct observations of standardized or actual encounters with interprofessional teams represents an important next step. Lastly, VR is not universally available for training, so may limit generalizability of our approach. However, VR has become an increasingly accessible tool for learning over time with decreased costs and equipment needs. Additionally, our pilot study demonstrated the efficacy of a screen-based VR curriculum for conflict management training, providing a potentially scalable platform that might overcome some of the barriers related to scale of VR interventions requiring a 3D-mounted headset.

Our pilot of CIVIC demonstrated improvement in HM clinicians’ skills and self-efficacy with addressing interprofessional conflict. The VR-based simulation provided a realistic and immersive training environment that could be administered remotely via teleconferencing, mitigating some of the time, space, and resource barriers of many simulation-based communication trainings. Next steps include further refinement of the assessment rubric, expansion to additional sites and learner groups (ie, medical students, residents, nurses) where the training may have greater impact, expansion of avatar demographics to address the potential impact of implicit bias on conflict management and linking exposure to CIVIC to real-world clinical performance.

We would like to acknowledge Ethan Elshoff, MPH and McKayla Schloemer, MS, the clinical research coordinators whose contributions were critical for participant recruitment and data management.