OBJECTIVE

Resident physicians are expected to recognize patients requiring escalation of care on day 1 of residency, as outlined by the Association of American Medical Colleges. Opportunities for medical students to assess patients at the bedside or through traditional simulation-based medical education have decreased because of coronavirus disease 2019 restrictions. Virtual reality (VR) delivered remotely via video teleconferencing may address this educational gap.

METHODS

A prospective pilot study targeting third-year pediatric clerkship students at a large academic children’s hospital was conducted from April to December 2020. Groups of 6 to 15 students participated in a 1.5-hour video teleconferencing session with a physician facilitator donning a VR headset and screen sharing interactive VR cases of a hospitalized infant with respiratory distress. Students completed surveys assessing the immersion and tolerability of the virtual experience and reported its perceived effectiveness to traditional educational modalities. Comparisons were analyzed with binomial testing.

RESULTS

Participants included third-year medical students on their pediatric clerkship. A total of 140 students participated in the sessions, with 63% completing the survey. A majority of students reported VR captured their attention (78%) with minimal side effects. Students reported remote VR training as more effective (P < .001) than reading and online learning and equally or more effective (P < .001) than didactic teaching. Most students (80%) rated remote VR as less effective than bedside teaching.

CONCLUSIONS

This pilot reveals the feasibility of remote group clinical training with VR via a video conferencing platform, addressing a key experience gap while navigating coronavirus disease 2019 limitations on training.

Pediatric residents are expected to identify patients requiring urgent escalation of care on day 1 of residency. This expectation was codified by the Association of American Medical Colleges in their core entrustable professional activities.1  Prompt identification of impending respiratory failure in patients with bronchiolitis is one such clinical scenario requiring urgent escalation of care. Bronchiolitis is the most common cause of pediatric hospital admission during the first year of life, accounting for ∼100 000 admissions annually in the United States.2,3  Of these admissions, 1% to 12% eventually require critical care for respiratory failure.4  However, opportunities for medical students to assess patients with impending respiratory failure is limited given restricted learner autonomy,5,6  declines in bedside teaching because of increasing patient turnover and time constraints,7,8  and shortcomings of traditional simulation-based medical education strategies such as mannequins and standardized patients, which cannot easily replicate key examination findings such as altered mental status or work of breathing.911 

The coronavirus disease 2019 (COVID-19) pandemic further limited education opportunities for students when the Association of American Medical Colleges designated students as nonessential health care workers in March 2020, recommending that training institutions prioritize student safety over clinical exposure.12  Although many students have returned to the wards, social distancing guidelines and personal protective equipment conservation efforts limit student clinical exposure. Furthermore, hospitalizations due to bronchiolitis have significantly decreased during the COVID-19 pandemic.13  Educators now face the difficult task of teaching clinical skills to the next generation of physicians away from the bedside, demonstrating a need for novel education techniques.

Virtual reality (VR), a form of simulation-based medical education, may play a key role in addressing this educational gap. Authors of a meta-analysis reviewing the use of virtual patients, including case-based modules and simulated clinical encounters, found a large pooled effect toward improvement in clinical reasoning and communication skills compared with traditional educational methods.14  We have previously described that single-user immersive VR training via a headset improved students’ recognition of respiratory distress and need for escalation of care.15,16  The emergence of COVID-19 and mandatory remote learning for students prompted exploration of the delivery of this VR curriculum via group video teleconferencing. We aimed to assess the acceptability and tolerability of remote VR education to pediatric clerkship students compared with traditional educational strategies.

A prospective pilot study was conducted at a large academic, pediatric medical center, affiliated with a college of medicine, from April to December 2020. Third-year pediatric clerkship medical students were eligible to participate. This study was approved by the institutional review board.

Each 6-week clerkship included student participation in a scheduled education session via the remote teleconferencing platform Zoom (Zoom Video Communications, Inc, San Jose, CA). A 1.5-hour group VR session on respiratory distress was designed by using situated learning, a conceptual framework characterized by guidance from an instructor and active participation from learners to complete a task. The goal of situated learning is enhancing knowledge and skills to promote independent future practice.17  During each session, 6 to 15 students met virtually with a pediatric faculty instructor with extensive group facilitation experience. After a 30-minute didactic, the instructor donned an OculusRift (Facebook Technologies, LLC, Menlo Park, CA) headset and projected the headset view via the video teleconferencing screen share function using a VR-capable computer. The virtual environment replicated an inpatient hospital room with graphical character representatives (ie, realistic animations) of a patient and bedside nurse. The facilitator then delivered the same VR training experience as the previous single-user study15 : 3 cases created on and delivered through the Unity (Unity Technologies, San Francisco, CA) game engine demonstrating no distress, respiratory distress, and impending failure. The virtual patient portrayed vital signs, mental status, accessory muscle usage, and breath sounds audible to the learner that were reflective of the 3 clinical states. The virtual bedside nurse responded in real-time to students’ questions regarding the patient’s history. The students instructed the facilitator where to navigate within the virtual world (eg, lean toward the patient to assess accessory muscle use or turn head to view vital sign monitor) and shared their observed findings and assessment of the patient’s clinical status aloud, similar to a traditional bedside teaching experience.

After completion of the virtual training session, students voluntarily completed a survey. Demographic information was obtained. Attitudes regarding the level of immersion were assessed through a subset of questions from the Measurements, Effects, Conditions Spatial Presence Questionnaire that used a 5-point Likert scale from strongly disagree to strongly agree.18  Tolerability was assessed by measuring side effects related to participation.19  Lastly, students rated the perceived efficacy of remote VR patient cases compared with other learning experiences as less, equal, or more effective. Data were collected in a deidentified manner and stored in a Research Electronic Data Capture database, a secure Web-based application.20 

Descriptive and summary statistics were used for demographics, degree of immersion, and tolerability. Student attitudes comparing remote VR to other educational strategies were analyzed with binomial testing.

A total of 140 students participated in the remote VR education sessions, with 63% completing the survey. (Table 1)

TABLE 1

Student Demographics, N = 88

n (%)
Age, y  
 18–24 36 (41) 
 25–29 48 (55) 
 30–34 4 (5) 
Sex  
 Male 43 (49) 
 Female 45 (51) 
Ethnicity  
 Hispanic 3 (3) 
 Asian American 9 (10) 
 Black 5 (6) 
 White 63 (72) 
 Mixed 7 (8) 
 No answer 1 (1) 
n (%)
Age, y  
 18–24 36 (41) 
 25–29 48 (55) 
 30–34 4 (5) 
Sex  
 Male 43 (49) 
 Female 45 (51) 
Ethnicity  
 Hispanic 3 (3) 
 Asian American 9 (10) 
 Black 5 (6) 
 White 63 (72) 
 Mixed 7 (8) 
 No answer 1 (1) 

A majority of students agreed or strongly agreed that the remote VR experience captured their senses (78%), and they were completely dedicated to the virtual experience (70%), were able to make a good estimate of the size of the presented space (77%), and retained a concrete mental image of the environment (83%). A plurality of students agreed or strongly agreed that they felt physically present in the environment (40%) and took part in the actions of (43%) and could do things with the objects in (43%) the simulation. Reported side effects from the VR experience were minimal (Table 2).

TABLE 2

Student-Reported VR Side Effects, N = 87

Yes, n (%)No, n (%)
Blurred vision 1 (1) 86 (99) 
Burping 1 (1) 86 (99) 
Difficulty concentrating 5 (6) 82 (94) 
Disorientation 3 (3) 84 (97) 
Dizziness 5 (6) 82 (94) 
Drowsiness 0 (0) 87 (100) 
Eye Strain 5 (6) 82 (94) 
Fatigue 1 (1) 86 (99) 
General discomfort 1 (1) 86 (99) 
Headache 1 (1) 86 (99) 
Nausea 0 (0) 87 (100) 
Salivation 0 (0) 87 (100) 
Stomach awareness 0 (0) 87 (100) 
Sweating 0 (0) 87 (100) 
Yes, n (%)No, n (%)
Blurred vision 1 (1) 86 (99) 
Burping 1 (1) 86 (99) 
Difficulty concentrating 5 (6) 82 (94) 
Disorientation 3 (3) 84 (97) 
Dizziness 5 (6) 82 (94) 
Drowsiness 0 (0) 87 (100) 
Eye Strain 5 (6) 82 (94) 
Fatigue 1 (1) 86 (99) 
General discomfort 1 (1) 86 (99) 
Headache 1 (1) 86 (99) 
Nausea 0 (0) 87 (100) 
Salivation 0 (0) 87 (100) 
Stomach awareness 0 (0) 87 (100) 
Sweating 0 (0) 87 (100) 

1 participant did not submit answers.

Students reported remote VR training as more effective than reading (response rate: 80% [95% confidence interval: 71%–88%], P < .001) and online learning (78% [68%–86%], P < .001). Remote VR training was rated as equally or more effective than traditional didactic teaching (91% [83%–96%], P < .001). Most students (80%) rated remote VR as less effective than bedside teaching (Fig 1).

FIGURE 1

Binomial testing demonstrates that a statistical majority of students found remote VR training more effective than reading and online learning and equally or more effective than didactic teaching and low fidelity mannequins. aVR training rated as more effective than comparison modality (P< .001). bVR training rated as equally or more effective than comparison modality (P < .001)

FIGURE 1

Binomial testing demonstrates that a statistical majority of students found remote VR training more effective than reading and online learning and equally or more effective than didactic teaching and low fidelity mannequins. aVR training rated as more effective than comparison modality (P< .001). bVR training rated as equally or more effective than comparison modality (P < .001)

Close modal

This study is novel in the use of VR to facilitate clinical training in a large group setting through video teleconferencing. A majority of students reported remote VR captured their attention with minimal distractions and was tolerable. Students rated remote VR as more effective than reading and online learning and equally or more effective than traditional didactics, the primary options for distance-learning available to students during the COVID-19 pandemic.

Our study was designed to address core entrustable professional activities no. 10, identifying a patient who requires urgent escalation of care, such as a patient with impending respiratory failure.1  Clerkship students who were removed from the bedside because of pandemic restrictions will soon serve as frontline providers. Our VR curriculum allowed students to practice assessing a patient with impending respiratory failure with real-time feedback, a training experience that cannot easily be replicated on the wards, especially during the COVID-19 pandemic era. Our remote VR curriculum offers a unique method to engage with students at the “virtual bedside,” allowing exposure to patients less commonly encountered in the hospitalized setting during the COVID-19 pandemic (eg, bronchiolitis).13 

However, group video teleconferencing does not fully reap the benefits previously reported with immersive VR. A previous cohort of third-year medical students who experienced a single-user immersive VR curriculum rated it as equally or more effective than standardized patients and high-fidelity mannequins, which was not seen with remote VR.21  Similarly, students reported a greater degree of immersion (ie, feeling physically present) in the single-user VR experience (85%) than with remote VR (40%).21  These findings suggest remote and immersive VR may be unequal in terms of perceived effectiveness. Educators should consider the best virtual modality to pursue on the basis of the learning objectives, required degree of immersion, and feasibility of implementation.

This study has several limitations. It was conducted at a single site, with only third-year medical students, and was focused on students’ perceptions toward the effectiveness of remote VR training. Additionally, we are unable to verify the students’ previous experience with the comparator education modalities. However, students were instructed to select “not applicable” for a comparator if they had never experienced it, attempting to minimize the impact of variable previous experiences. With this pilot study, we were unable to capture the impact of remote VR on clinical performance. Formally assessing knowledge application represents a key next step on the path to establishing a clinical performance impact of remote VR. Lastly, the generalizability of our approach is currently limited given the absence of universal access to VR technology or curricula. However, access to VR is becoming more feasible with declining equipment costs22  and increasing availability of content sharing.

The COVID-19 pandemic has revealed the need for innovative educational methods to reach distanced learners and teach key clinical skills, such as assessment of respiratory distress. VR patient cases, delivered via remote video teleconferencing, are a potential strategy. Importantly, the majority of students in our study and in previous work rate bedside teaching as most effective.21  This highlights the difficulty in creating realistic patient encounters virtually and that nothing can fully replace the power of precepting a student in a patient’s room. However, remote VR may serve as a supplement for bedside teaching. As social distancing guidelines are lifted and medical students ward exposure normalizes, educators should redouble their efforts to seek opportunities to engage students at the bedside.7,8 

We thank the medical students from the University of Cincinnati College of Medicine for their participation in this curriculum.

FUNDING: No external funding.

Dr Young conceptualized and designed the study, participated in data collection sessions, assisted with data analysis, and drafted the initial manuscript; Dr Real conceptualized and designed the study, coordinated and facilitated all data collection sessions, and reviewed and revised the manuscript; Dr Sahay performed the data analysis, and reviewed and revised the manuscript; Dr Zackoff conceptualized and designed the virtual reality cases used in this study, conceptualized and designed the study, assisted with data analysis, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted.

1.
Englander
R
,
Flynn
T
,
Call
S
, et al
.
Toward defining the foundation of the MD degree: core entrustable professional activities for entering residency
.
Acad Med
.
2016
;
91
(
10
):
1352
1358
2.
Kyu
HH
,
Pinho
C
,
Wagner
JA
, et al
;
Global Burden of Disease Pediatrics Collaboration
.
Global and national burden of diseases and injuries among children and adolescents between 1990 and 2013: findings from the Global Burden of Disease 2013 Study
.
JAMA Pediatr
.
2016
;
170
(
3
):
267
287
3.
Florin
TA
,
Plint
AC
,
Zorc
JJ
.
Viral bronchiolitis
.
Lancet
.
2017
;
389
(
10065
):
211
224
4.
Lozano
R
,
Naghavi
M
,
Foreman
K
, et al
.
Global and regional mortality from 235 causes of death for 20 age groups in 1990 and 2010: a systematic analysis for the Global Burden of Disease Study 2010 [published correction appears in Lancet. 2013;381(9867):628]
.
Lancet
.
2012
;
380
(
9859
):
2095
2128
5.
Holzman
IR
,
Barnett
SH
.
The Bell Commission: ethical implications for the training of physicians
.
Mt Sinai J Med
.
2000
;
67
(
2
):
136
139
6.
Halpern
SD
,
Detsky
AS
.
Graded autonomy in medical education--managing things that go bump in the night
.
N Engl J Med
.
2014
;
370
(
12
):
1086
1089
7.
Peters
M
,
Ten Cate
O
.
Bedside teaching in medical education: a literature review
.
Perspect Med Educ
.
2014
;
3
(
2
):
76
88
8.
Mazotti
LA
,
Vidyarthi
AR
,
Wachter
RM
,
Auerbach
AD
,
Katz
PP
.
Impact of duty-hour restriction on resident inpatient teaching
.
J Hosp Med
.
2009
;
4
(
8
):
476
480
9.
Cook
DA
,
Hatala
R
,
Brydges
R
, et al
.
Technology-enhanced simulation for health professions education: a systematic review and meta-analysis
.
JAMA
.
2011
;
306
(
9
):
978
988
10.
Akaike
M
,
Fukutomi
M
,
Nagamune
M
, et al
.
Simulation-based medical education in clinical skills laboratory
.
J Med Invest
.
2012
;
59
(
1–2
):
28
35
11.
Lateef
F
.
Simulation-based learning: Just like the real thing
.
J Emerg Trauma Shock
.
2010
;
3
(
4
):
348
352
12.
Association of American Medical Colleges
.
Guidance on medical students’ participation in direct in-person patient contact activities
.
2020
.
13.
Wilder
JL
,
Parsons
CR
,
Growdon
AS
,
Toomey
SL
,
Mansbach
JM
.
Pediatric hospitalizations during the covid-19 pandemic
.
Pediatrics
.
2020
;
146
(
6
):
e2020005983
14.
Consorti
F
,
Mancuso
R
,
Nocioni
M
,
Piccolo
A
.
Efficacy of virtual patients in medical education: a meta-analysis of randomized studies
.
Comput Educ
.
2012
;
59
(
3
):
1001
1008
15.
Zackoff
MW
,
Real
FJ
,
Sahay
RD
, et al
.
Impact of an immersive virtual reality curriculum on medical students’ clinical assessment of infants with respiratory distress
.
Pediatr Crit Care Med
.
2020
;
21
(
5
):
477
485
16.
Zackoff
MW
,
Young
D
,
Sahay
RD
, et al
.
Establishing objective measures of clinical competence in undergraduate medical education through immersive virtual reality
.
Acad Pediatr
.
2021
;
21
(
3
):
575
579
17.
Artemeva
N
,
Rachul
C
,
O’Brien
B
,
Varpio
L
.
Situated learning in medical education
.
Acad Med
.
2017
;
92
(
1
):
134
18.
Vorderer
P
,
Wirth
W
,
Gouveia
FR
, et al
.
MEC spatial presence questionnaire (MECSPQ): short documentation and instructions for application. Report to the European Community, Project Presence: MEC (IST-2001-37661)
.
2004
.
Available at: https://academic.csuohio.edu/kneuendorf/frames/MECFull.pdf. Accessed August 14, 2020
19.
Kennedy
RS
,
Fowlkes
JE
,
Berbaum
KS
,
Lilienthal
MG
.
Use of a motion sickness history questionnaire for prediction of simulator sickness
.
Aviat Space Environ Med
.
1992
;
63
(
7
):
588
593
20.
Harris
PA
,
Taylor
R
,
Thielke
R
,
Payne
J
,
Gonzalez
N
,
Conde
JG
.
Research electronic data capture (REDCap)--a metadata-driven methodology and workflow process for providing translational research informatics support
.
J Biomed Inform
.
2009
;
42
(
2
):
377
381
21.
Zackoff
MW
,
Real
FJ
,
Cruse
B
,
Davis
D
,
Klein
M
.
Medical student perspectives on the use of immersive virtual reality for clinical assessment training
.
Acad Pediatr
.
2019
;
19
(
7
):
849
851
https://doi.org/10.1016/j.acap.2019.06.008
22.
Robertson
A
.
Oculus Rift price drops permanently to $399
.
The Verge. October 11, 2017. Available at: https://www.theverge.com/2017/10/11/16459664/oculus-rift-price-drop-permanent- 399-connect. Accessed October 1, 2020

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.