Virtual Reality to Assess Resident Recognition of Impending Respiratory Failure During COVID-19

This cross-sectional observational study used pediatric residents at a large free-standing academic children’s hospital. A convenience sample of residents spanning postgraduate years (PGY) 1 through 5 were recruited from the hospital-based continuity clinic. PGY4 and 5 residents were completing combined training in Pediatrics-Psychiatry-Child Psychiatry. Recruitment by the clinical research coordinator (A.M.) was via e-mail as well as in person during preclinic teaching sessions with a $50 incentive to participate. Consent was obtained at enrollment, and the study was approved by the local institutional review board.

VR sessions occurred from January to April 2021. The 30-minute simulation included (1) orientation to the VR patient and environment, (2) participation in a case of impending respiratory failure from bronchiolitis, (3) and debriefing the scenario with a facilitator. The curriculum used the framework of reflective practice to support adult learning through debriefings that emphasized reflection on action.¹¹ 

The VR environment was constructed using Unity (Unity Technologies, San Francisco, CA). The 3-month-old VR patient with impending respiratory failure exhibited altered mental status (eyes closed, no vocalizations, no extremity movement); displayed signs of increased work of breathing (head bobbing; nasal flaring; suprasternal, intercostal, and subcostal retractions); and had abnormal vital signs (tachycardia, tachypnea) displayed on an in-room monitor. The VR content underwent extensive piloting by 4 critical care attending physicians, 2 hospital medicine attending physicians, 2 general pediatricians, 4 critical care fellows, and 4 pediatric chief residents with universal consensus that the virtual patient exhibited signs of impending respiratory failure and required transfer to the PICU.^5,6 

The facilitators (J.O., F.R.) operated the VR simulation using an Oculus Rift (Meta Technologies, Menlo Park, CA) headset and transmitted the virtual environment to participants via Zoom (Zoom Video Communications, Inc., San Jose, CA). Residents and medical students have previously indicated acceptability and adequate immersion with such screen-based VR interventions.^12,13 

At the start of the VR experience, residents were provided the case prompt (a “3-month-old male admitted to the hospital medicine service with bronchiolitis”) with basic details on the history of present illness and review of systems. Residents were instructed to share their thought process out loud as they evaluated the patient and were informed the simulation would conclude once they provided a formal assessment and plan.

The primary outcomes were descriptions of residents’ behaviors during the VR simulation using a binary scale of present versus not present for the following tasks: (1) recognized altered mental status (AMS) by stating that the patient was unresponsive or had AMS, (2) specifically verbalized concerns for “(impending) respiratory failure,” and (3) escalated care via calling for the institutional medical response team (MRT) or indicating the need for transfer to PICU. The facilitators scored 3 sample cases jointly to ensure consistent use of the assessment instrument.

As a secondary outcome, we categorized the management plans proposed by the resident to 1 of the following tiers: (1) provide supportive care (oxygen, suctioning), (2) perform additional work-up (chest radiograph, blood gas), (3) increase respiratory support without transferring to the PICU (low- or high-flow nasal cannula), or (4) MRT activation or transfer to the PICU. Residents received credit for the highest level of management proposed according to this hierarchy (Fig 1).

The number of inpatient hospitalizations for admission diagnosis International Classification of Diseases, 10^th Revision codes corresponding to bronchiolitis between July 2018 and June 2021 were queried from the electronic health record to provide context for residents’ exposure to bronchiolitis during the study period (Fig 2).

Data were described as frequency counts and percentages. Statistical differences between and across PGY levels for each outcome were examined using χ² or Fisher’s exact test as applicable, followed by pairwise comparison and posthoc multiple testing using the Hochberg test. All testing was done as 2-sided tests and P ≤ .05 was considered as statistically significant. All analyses were performed using SAS version 9.4 (SAS Institute Inc, Cary, NC).

Of 83 potential participants, 62 (75%) pediatric residents participated in this study (PGY1 34%, PGY2 31%, PGY3 32%, PGY4 2%, PGY5 2%) with balanced representation across PGY levels when PGY3–5 (PGY3+) were cohorted. Sixty residents (97%) had previous exposure to a VR curriculum that included verbal interaction with screen-based avatars.^14,15 

Among all residents, 53% (n = 33) successfully recognized AMS, 16% (n = 10) appropriately identified “(impending) respiratory failure,” and 23% (n = 14) arranged for an escalation of care by activating an MRT or indicating the patient needed to transfer to the PICU. There were no statistical differences between PGY levels in recognition of AMS (PGY1 38%, PGY2 47%, PGY3+ 73%, P = .06) or verbal identification of “(impending) respiratory failure” (PGY1 14%, PGY2 16%, PGY3+ 18%). PGY3+ residents were statistically more likely to escalate care than PGY2 residents (P = .05), but no other differences were observed between PGY levels (PGY1 24%, PGY2 5%, PGY3+ 36%). There were no statistically significant differences between PGY levels with regards to management plans when rated on a hierarchical scale (P = .11) (Table 1).

In the setting of decreased clinical volumes during the COVID-19 pandemic, pediatric residents across all PGY levels at our institution demonstrated limited ability to verbally identify “(impending) respiratory failure” and appropriately escalate care during VR simulations.

COVID-19 led to a reduction in pediatric respiratory admissions during the pandemic.¹⁶  Our institution witnessed a 16-month reduction in admissions for bronchiolitis from historical norms. We hypothesize our findings are potentially because of the pandemic-related decrease in clinical exposure to respiratory pathology that may have led to a delay or depreciation of skills. This is supported by the observation that PGY3+ residents, who had previous exposure to typical volumes of bronchiolitis in the prepandemic period, were statistically more likely to escalate care than PGY2 residents who had more limited exposure. Notably, there were no changes to the educational curriculum, clinical policies around escalation from the acute care setting to the PICU, or orientation or training exposures during the study period.

It is important to consider the limitations of this study. First, this pilot study was conducted with a convenience sample of residents at 1 institution, limiting generalizability. However, their clinical exposure is likely comparable to other large pediatric residency programs. Second, although we hypothesize the findings were because of a lack of clinical exposure, we do not have baseline data before the onset of the COVID-19 pandemic for comparison. Third, we deployed a strict assessment strategy in our primary outcomes that required residents to use precise language to receive credit. Although our findings may represent a lack of comfort with stating “(impending) respiratory failure,” the low rates of appropriate proposed management (ie, escalate care to the PICU) suggests that there is a true gap in recognition of these high-risk patients. Finally, it is possible that observed behaviors during VR simulations do not reflect real-world clinical practice. We used VR in response to social distancing guidelines, low patient volumes, and the necessity for scenario standardization. Moreover, as the majority of participants had prior exposure to VR curricula, we do not anticipate that unfamiliarity with the platform markedly impacted performance. The extensive piloting with experienced providers, improved performance for some metrics with increasing PGY year, and prior reports of efficacy also support the validity and appropriateness of the platform for formative assessment.^5,6  Key next steps include (1) using this VR curriculum to assess the performance of pediatric residents exposed to normalized volumes of respiratory pathology, and (2) characterizing the performance among other acute care providers with varying experience levels using the same VR simulation to further validate this platform as an assessment tool.

An assessment of pediatric residents’ skills at identifying and managing respiratory failure after a period of reduced clinical volumes indicated room for improvement. VR allowed for skills to be assessed in a remote, socially distant manner, and allowed for just-in-time education on respiratory failure. However, further data are needed to understand the role of VR as an adjunct for clinical training and assessment.