Real-Time Intubation and Ventilation Feedback: A Randomized Controlled Simulation Study

We hypothesized that the addition of feedback devices (visible only to supervisor or both supervisor and participant) would increase airway management quality. Furthermore, we hypothesized that guidance from the supervisor alone would allow participants to focus on their task entirely, which could facilitate superior performance compared with the other study groups.

This prospective single-site randomized-controlled trial was conducted in a simulation-based setting at the Medical University of Vienna, Austria, in accordance with the local institutional review board and the Medical University data protection board. Each participant signed a written informed consent. The study is reported according to the Consolidated Standards of Reporting Trials (CONSORT) approach with the extension for simulation-based research.²² 

In total, 167 medical students in their third to sixth year (preclinical and clinical training, approximately comparable to first to fourth year in the United States) partook in this trial to ensure a comparable experience among the participants.

Participants registered by scanning a QR-code posted on the university’s social media page and in on-site curriculum courses (pediatric and neonatal basic life support training). Before the simulation training, a computer algorithm (random.org) randomized participants into 3 groups with a 1:1:1 allocation system.

To provide a homogenous supervision experience for all participants in this study, supervisors were limited to a small number of people and were trained beforehand to deliver standardized feedback and to explain ventilation technique and interpretation of feedback devices in a similar manner. Supervisors were mostly residents from our unit or trained members of the study team.

Respiratory parameters were collected with a RFM (NeoTraining, Monivent AB, Gothenburg, Sweden). A video laryngoscope (InfantView, Acutronic Medical Systems AG, Hirzel, Switzerland) was used for intubations.

The primary outcome for the ventilation quality was the percentage of ventilations with expiratory tidal volumes (pV_Te) within a target range of 4 to 8mL/kg for each participant and each group. A percentage of 100 meant that all ventilations had a V_Te within the target range.

The primary outcome for intubation was the number of attempts until success.

Secondary outcomes included mask leak, PIP, ventilation rate, time to intubation, as well as demographic data.

An a priori power analysis with G*Power predicted that a total sample size of 170 participants would give sufficient power (0.80) to observe significant effects at the α level of 0.05 by a large effect size (Cohen’s f = 0.25). Based on previous studies we expected a difference in pV_Te of 25%. A feasibility study by Schmölzer et al showed a lower proportion of excessive tidal volumes (> 8mL/kg, reduction of 20%) with the use of a RFM.²³  Zeballos et al²⁸  could show a reduction of 17.15% (P = .001) in tidal volume when the RFM was visible during resuscitation in the delivery room. A multicenter trial by Hunt et al also identified a 22% reduction of median tidal volume when the RFM was used in simulated resuscitation training.²⁴ Figure 1 shows the consort flow diagram.

Participants in this study had to complete a simulation-based training that included ventilation and intubation of a newborn manikin. Participants were randomized into 3 groups with different visibility of the feedback devices (see Fig 2). A supervisor from the study team provided guidance throughout all the ventilation and intubation steps as well as quality evaluation. Group A, the “no-access group,” had no visual access to the feedback devices. A supervisor instructed the participants, as routinely done in clinical settings. Although monitors were not visible, data were still recorded. Group B, the “supervisor-access group,” was guided by a supervisor who could see the feedback monitors and gave objective feedback to the participants accordingly. Group C was the “full-access group” in which both the supervisor and the participant could see the feedback monitors. Additionally, a patient monitor displaying vital parameters of the simulated patient was visible to all groups.

At the beginning of the simulation training, all participants watched a short introductory video demonstrating correct intubation and ventilation techniques. Participants in the full-access group also received a short video-demonstration on using and interpreting data from the feedback devices. In the simulated scenario, the infant had to be anesthetized to undergo surgery and, therefore, needed ventilation and intubation. Participants were told to ventilate the infant for at least 1 minute followed by intubation by using the video laryngoscope (1 minute maximum per attempt). Once the participants had placed the endotracheal tube correctly, ventilation of the manikin could proceed for another minute via the endotracheal tube to complete the training. If the intubation attempt was unsuccessful, participants were told to ventilate manually (bag-mask ventilation) again for a minute, before another intubation attempt. Participants were given a maximum of 5 intubation attempts before ending the training, regardless of success.

Approximately 60 to 180 metric data points were collected for each ventilation parameter (pV_Te, leak, PIP, rate) and each participant. One metric value was recorded for intubation attempts and time to intubation, respectively.

The pV_Te was calculated as a proportion of ventilations within the target range of 4 to 8mL/kg in relation to all ventilations performed by each participant, resulting in a value between 0 and 1. For better understanding this value was transformed to “percent, %” after the analysis.

Data were displayed as boxplots for each group. A histogram organized data to assess normality of distribution. Because of the large number of participants, it was not necessary to factor in the normality of distribution when deciding on a statistical test. Instead, descriptive data (mean, median, SD, interquartile range, minimum and maximum [range]) were calculated for each variable (see Supplemental Table 3).

One-way ANOVA detected statistically significant differences between the groups, and in case of statistically significant results, a posthoc test was carried out using the Fisher’s Least Significant Difference (LSD), an approach for pairwise comparisons that finds specific differences between any 2 groups. No adjustment for multiple testing was necessary in this case. The level of significance was set at P < .05 (2-tailed). Statistical analysis was carried out with IBM SPSS Statistics 27.0.

In this study, 180 participants were initially recruited. However, because of drop-outs during the implementation phase, the final number of participants decreased to 167 (drop-out rate: 7.2%). After randomization, 55 participants completed the simulation training in the no-access group, 56 in the supervisor-access group, and 56 in the full-access group. Two participants were unsuccessful on the fifth intubation attempt and were excluded from intubation data analysis (n = 165). Participants in this study were mostly female (73.7%), the majority were between 21 to 25 years old (78.4%) and in their fourth year at medical school (46.1%). Almost all (89.7%) participants had no previous experience using feedback devices, and more than one-half (58.4%) had never taken part in a simulation training (see Table 1 for more detail).

Participants in the full-access group (group C) performed significantly higher-quality ventilations than any other group, with better results for the primary outcome parameter “pV_Te” as well as for all secondary outcome parameters (see Fig 3).

The full-access group maintained ventilations at a mean pV_Te of 63.6% (SD 16.2) within the tidal volume target range (4–8mL/kg). The supervisor-access group kept about one-half (51%, SD 16.4) of the ventilations within target range, whereas the no-access group had the lowest percentage of successful ventilations with only 31.3% (SD 15.1) (A to B [95% confidence interval (CI) 0.14–0.26], A to C [95% CI 0.27 to 0.38], B to C [95% CI 0.07 to 0.19]; P < .001).

Mean mask leak was high for all groups, with an average of 47.6%. The lowest (ie, best) values for mean mask leak were 34.9% in the full-access group, followed by the supervisor-access group with 46.6% and the no-access group with 61.6% (A to B: P < .001 [95% CI 7.43 to 22.60], A to C: P < .001 [95% CI 19.08 to 34.25], B to C: P = .003 [95% CI 4.11 to 19.20]).

The lowest peak inspiratory pressure was delivered by the full-access group with a mean of 19.6 cmH₂O, followed by 21.2 cmH₂O in the supervisor-access group, and the highest values of 23.0 cmH₂O measured in the no-access group (A to B: P = .011 [95% CI 0.41 to 3.17]; A to C: P < .001 [95% CI 2.05 to 4.81]; B to C: P = .02 [95% CI 0.26 to 3.01]).

Ventilation rate did not differ significantly between groups (see Table 2) with a mean rate of 27.5 ventilations per minute overall (SD 5.07), which is appropriate for a term newborn (A to B: P = .759 [95% CI −1.59 to 2.18], A to C: P = .128 [95% CI −3.35 to 0.43], B to C: P = .067 [95% CI −3.64 to 0.12]).

Most participants in the full-access group needed only 1 intubation attempt (75%, mean 1.29, SD 0.53) (see Fig 3). Participants in the supervisor-access group required a mean of 1.77 attempts (SD 0.99), whereas the no-access group required multiple attempts (mean 2.43, SD 1.42). Differences were found to be statistically significant (A to B: P = .001 [95% CI 0.27 to 1.06], A to C: P < .001 [95% CI 0.76 to 1.54], B to C: P = .015 [95% CI 0.09 to 0.87]). No participant in the full-access group required 5 intubation attempts, whereas 1 participant (0.02%) in the supervisor-access group and 7 participants (13.2%) in the no-access group required all 5 attempts to successfully intubate. Two participants in the no-access group did not manage to successfully intubate at the fifth attempt (excluded from analysis).

On average, it took participants 47.8 seconds (SD 9.91) to successfully intubate, with a minimum of 22.2 seconds as the fastest intubation and a maximum of 60.0 seconds. Intubation times were similar in the study groups with 46.6, 45.8, and 51.2 seconds in the full-access, supervisor-access, and the no-access group, respectively. One-way ANOVA using LSD-posthoc testing identified significant differences between groups A and B (A to B: P = .004 [95% CI 1.77 to 9.10]) and groups A and C (A to C: P = .013 [95% CI 0.98 to 8.31]), but not between groups B and C (B to C: P = .665 [95% CI −2.82 to 4.41]).

Total intubation time (combined intubation time of all attempts per participant; this value is correcting for learning effect in participants with multiple attempts) was 96.87 seconds in total, with a mean of 137.24 seconds in group A, 91.84 seconds in group B and 63.70 seconds in group C. Analysis between groups revealed statistically significant differences in all pairwise comparisons (A to B: P < .001 [95% CI 22.75 to 68.06]; A to C: P < .001 [95% CI 50.88 to 96.20]; B to C: P = .014 [5.79 to 50.48]).

This study aimed to evaluate differences in ventilation quality and intubation success with varying feedback monitor visibility in different educational settings. Our results verify the hypothesis that allowing supervisors and participants direct visual access to the feedback monitors increases participants’ ventilation quality and intubation success rate. To our knowledge, this is the first study examining the effectiveness of feedback devices, and their optimal application in medical education. Therefore, our work might offer new insights into training and teaching airway management in simulation-based education but could also positively impact clinical teaching in the coming years. A clear understanding of the benefits of such tools and their most effective use is crucial in furthering medical education.

Our results are in line with previous literature, suggesting the superiority of ventilation quality and intubation success when feedback devices are available.^{5–7,25–29}  However, it is important to note that a previous multicenter randomized controlled trial by van Zanten et al including 288 neonatal patients could not show statistically significant effects on ventilation quality when using a RFM (percentage of ventilations in target range were 30.0% in the RFM visible group and 30.2% in the RFM nonvisible group, P = .721).³⁰  We assume that the difference in results might largely hinge on differences in monitor design (device by different manufacturer), training in monitor use, study setting (simulation versus clinical), and participant collective (students versus physicians or nurses).

Data regarding benefits of video laryngoscopy compared to direct laryngoscopy for outcomes such as first pass success rate, intubation time, or airway trauma are ambiguous. For instance, Savino et al showed that VL could even worsen intubation outcomes when used by providers that have extensive experience with direct laryngoscopy. However, they reported improved intubation performance among an inexperienced study population,¹³  essentially confirming our findings.

Intubation times did not differ greatly among our study groups (mean intubation time: group A 51.21s, group B 45.8s, group C 46.6s). According to previously published data, median or mean intubation time, especially for less experienced providers, was similar to our intubation durations (44.7 seconds median in Law et al,³¹  49 seconds in O’Donnell et al,³²  48 to 69 seconds in Simma et al,³³  57 seconds with VL vs 47 seconds without VL in Sawyer et al³⁴ ). This means compared to previous literature intubation durations in our study were neither especially short nor especially long.

Previous research efforts have revealed that more experienced participants deal with higher mental workload and cognitive demands compared to inexperienced participants (P = .001) when using a feedback device.³⁵  Pearlman et al also demonstrated that differences in ventilation quality between inexperienced and highly experienced providers decreased with the use of a feedback device.²⁵  These conclusions suggest that feedback devices represent a valuable teaching tool for inexperienced residents but might distract more expert physicians who have already consolidated their airway management skills.

On the other hand, the application of feedback devices might help experienced physicians to improve their teaching quality when acting as supervisors for their younger colleagues.

We hypothesize from the results of this study that the presence of a supervisor during skill training with feedback devices is highly beneficial to learning and overall performance. Furthermore, we assume that different levels of interaction and expertise from the instructors can impact the usability and effectiveness of feedback devices.

We have studied the effect of supervision on cardiopulmonary resuscitation quality in a previous study.³⁵  This study identified a difference between learning without an instructor and with an instructor when using a feedback device, but this effect was smaller than compared to the impact of a feedback device alone. It should be considered carefully when feedback device-based trainings are being planned that a supervisor should be present during the training and that this supervisor should have the proper qualifications. A follow-up study investigating the effects of feedback devices in simulation trainings without a supervisor present would be highly interesting.

It can be speculated that the learning curve will be much steeper when a supervisor is present as trainees benefit from the knowledge, experience, and guidance of the supervisor. Within a clinical setting, we would advise to use of feedback devices with supervision to reduce cognitive workload. However, within simulation-based teaching, self-directed learning seems possible. Although it is certainly beneficial to have a supervisor present during all trainings, this might not be manageable and might take up a lot of resources. Therefore, in a situation in which a supervisor is not always available it could be beneficial if a trainee uses feedback devices to acquire airway management skills through self-directed learning.

A priority of this study was to compare airway management teaching and determine if a shared feedback monitor view would translate into better airway management quality. We had postulated that guidance from the supervisor alone would allow participants to focus on their task entirely. Therefore, cognitive workload would be decreased for trainees, which could facilitate improved ventilation quality and increase effectiveness of teaching or learning. However, the opposite was the case, because participants performed significantly better when they also had direct visual access to the feedback devices. We showed similar effects in a previous simulation-based study, in which ventilations and chest compressions during neonatal resuscitation were analyzed.³⁵  Participants receiving input from a feedback device or supervisor plus feedback device performed significantly better than participants relying on a supervisor alone. For certain variables, the group relying on the feedback device alone produced the best results.³⁵ 

A possible explanation of this phenomenon (superiority of full-access group) might be found in the monitor design. Because the image captured by the VL is highly detailed, supervisors can easily guide trainees through access and placement maneuvers and point out the anatomic structures. Also, observation during the training sessions has evidenced that the RFM’s color-coded signals are more efficient than verbal feedback alone. For instance, if the mask leak is very high, the monitor blinks red, alerting participants to improve their mask hold and ventilation technique more than verbal feedback alone.

The use of further monitors in intensive care units, in which medical devices already generate an overwhelming amount of information, might still be a challenging option. Therefore, feedback devices should be carefully placed to facilitate the operators’ response. Monitor placement might also have an impact on the usefulness of feedback devices. However, a previous study determined that the setup is not always ideal (eg, behind the physician, although most physicians find central placement most convenient).³⁶  In a previous simulation-based study we also showed that visual attention significantly impacts the quality and performance of chest compressions and ventilations during neonatal resuscitation.³⁷  We assume that monitor placement plays a big role in the quality and effectiveness of simulation-based teaching and training. Therefore, based on our results, monitor placement should be optimized to improve teaching quality.

During the implementation phase of this study, we noticed that participants familiar with ventilation procedures performed significantly better, especially in minimizing mask leak and intubation risks. In the Austrian educational system, medical students are required to complete at least 12 weeks of clinical practice at different units during their studies, students can choose the specific units. Therefore, it is possible that some participants had previous experience in intensive care or anesthesiology and might have already been able to ventilate or intubate a few times. This factor could have influenced the outcome irrespective of group allocation.

Furthermore, the results of this study cannot be directly transferred to a neonatal patient population because of differences in lung and thorax compliance between manikin and neonate, as well as interindividual differences among neonates. Additionally, our results are only applicable to a neonatal or pediatric patient population and cannot be directly transferred to adults.

The strength of this study is its prospective randomized controlled trial design and the standardized study simulation-based setting as well as the high number of participants.

Participants with access to feedback monitors showed a statistically significant increase in performance compared to participants with no such advantage.

The most practical educational application of feedback devices involved full access for the supervisor and the trainee. Participants in this group showed a significantly higher ventilation quality (higher proportion of tidal volume within range, higher ventilation precision, lower PIP, lower mask leak) and success rate during intubations (lower number of attempts) compared to any other group.

These results implicate a substantial advancement in teaching airway management to young physicians in the clinical setting and in the design of future simulation training and clinical teaching.

We thank the participants and medical staff of the Medical University of Vienna for their contribution to this study. We acknowledge the support of the Scientific Publishing Service provided by P. Voitl and E. Tomasco, on behalf of the Austrian Pediatric Society. Furthermore, the authors would like to acknowledge grant support from the “Gesellschaft für Neonatologie und pädiatrische Intensivmedizin (GNPI).”

CI

confidence interval

CONSORT

Consolidated Standards of Reporting Trials

IQR

interquartile range

PIP

peak inspiratory pressure

pV_Te

percentage of expiratory tidal volumes

RFM

respiratory function monitor

VL

video laryngoscope