Intubation Competence During Neonatal Fellowship Training

This was a retrospective cohort study by using prospectively gathered data in the multicenter National Emergency Airway Registry for Neonates (NEAR4NEOS). NEAR4NEOS is an international neonatal airway registry that has been previously described.^16–18  Participating sites collect and contribute data for all neonatal intubations performed in the NICU or delivery room (DR), including patient, provider, and practice characteristics as well as outcome data, using standardized operational definitions.

We included intubations from October 2014 through December 2018 performed at eligible NEAR4NEOS North American academic centers. The study start time corresponded to the inception of the NEAR4NEOS database. Eligible US and Canadian sites were defined as centers where at least 90% of fellows’ intubation attempts were captured in the NEAR4NEOS database and a direct method of linking each intubation encounter to a specific neonatology fellow existed. All intubations with any fellow participation as an airway provider within the intubation encounter were included, regardless of whether the fellow was the first airway provider or not. Intubations in both the NICU and DR were included. Tracheal tube exchanges were excluded from this study, because tube exchange with ongoing ventilation through the existing endotracheal tube during the procedure represents a different skillset from endotracheal intubation without an endotracheal tube in place. At all sites, the NEAR4NEOS registry was granted institutional review board approval or was designated as quality improvement exempt from institutional review board oversight.

On the basis of NEAR4NEOS operational definitions, an intubation encounter is defined as one sequence of completed advanced airway management, which can include multiple intubation attempts. An attempt is defined as a single advanced airway maneuver starting with the insertion of an airway device into the patient’s mouth or nose.

We ascertained the start date of fellowship training for all included fellows. For this analysis, the exposure of interest was fellowship training experience, which we defined in 2 ways: sequential intubation encounters (for the cumulative sum [CUSUM] analysis), and the quarter of fellowship training (for the aggregate analysis).

The exposure was defined as the sequential encounter count, starting from each individual’s first intubation encounter during the fellowship. Only fellows with intubation data captured from the start of their fellowship training were included in this analysis.

The exposure of interest was the quarter of fellowship training. We calculated the quarter of the fellowship at the time of intubation using the start date of training and date of the intubation encounter. All fellow intubation encounters during the study period were included in this analysis.

The primary outcome for both analyses was intubation success confirmed by chest rise, auscultation, second independent laryngoscopy, carbon dioxide detection, and/or chest radiography, as previously described.¹⁶  We defined success from the perspective of the individual fellow as a successful intubation within 2 attempts performed by that fellow. Of note, if a fellow had 1 unsuccessful attempt and the subsequent attempt was performed by a new provider, this was classified as a failure. If >1 fellow participated in the encounter, each fellow was assessed separately on the basis of these definitions.

Using Stata 16.1 (Stata Corp, College Station, TX), we compared baseline provider, patient, and practice characteristics between successful and unsuccessful intubations with standard bivariate tests (Wilcoxon rank and χ²). Intubation success rates by fellowship training quarter were calculated as the percentage of successful intubations per quarter, with 95% confidence intervals computed by using the binomial distribution.

CUSUM calculations were performed to assess individual performance. CUSUM evaluation requires specifying acceptable (p0) and unacceptable (p1) intubation rates as well as the probability of type I (α) and II (β) error. The acceptable failure rate was set at 20% for successful intubation within 2 attempts (corresponding to 80% success), whereas the unacceptable failure rate was set at 40%. The probability of type I (α) and II (β) error was set at .1. There is no uniformly accepted definition for neonatal intubation competence. We selected an 80% success rate because this is consistent with previous definitions used in neonatal studies^8,9  and comparable with that of observational studies reporting intubation success rates of 72% to 88% for attending neonatologists.^7,11,16  In a post hoc sensitivity analysis, we assessed different definitions of success, using a less stringent acceptable failure rate of 30% (corresponding to 70% success) and more stringent acceptable failure rate of 10% (corresponding to 90% success). In each sensitivity analysis, we set the unacceptable failure rate as twice the acceptable failure rate, as in the primary CUSUM analysis.

The formula used for the CUSUM calculation was similar to those used in studies by Ishizuka et al¹³  and Komatsu et al.¹⁹  From the values of p0, p1, α, and β, the variable s is calculated. Individual fellow charts began at 0; for each success, s was subtracted from the previous CUSUM value, and, for each failure, the amount 1-s was added. A negative trend demonstrates progression toward competence. This is achieved when the line crosses the lower decision limit (h0), which corresponds to p0, indicating the true failure rate does not differ significantly from an acceptable failure rate. Similarly, a positive trend demonstrates progression away from proficiency. If the upper decision limit (h1) is crossed, the true failure rate is significantly greater than the acceptable failure rate (ie, deficiency is reached).

For all fellow intubations, we assessed the impact of fellow experience on intubation success within 2 attempts with a mixed effects logistic model using the independent variable of the quarter of fellowship training (from 1 to 12). The model included a random intercept for individual fellows and fixed effect for the hospital. The model included the following a priori characteristics that have been associated with success during neonatal intubation: paralytic premedication use, use of video laryngoscopy, and patient weight at the time of intubation.^1,16,18,20  A P value <.05 was considered statistically significant.

During the study period, a total of 92 fellows participated in 2297 tracheal intubation encounters in 8 hospitals representing 6 fellowship training programs (Table 1). Across these, 1766 of 2297 (77%) intubation encounters were successful within 2 attempts. Fellow characteristics and factors associated with intubation success are described in Table 2.

We included 40 fellows from 4 US training programs in the CUSUM analysis. Of these, 18 (45%) fellows achieved procedural competence during the period assessed, and 12 (30%) met the failure threshold (Fig 1). Among fellows who achieved competence, the median number of intubations to meet this threshold was variable, with a median of 18 (interquartile range [IQR] 15–25) and absolute range of 8 to 46 intubations. The median number of intubations to cross the deficiency threshold was 7 (IQR 5–14), with some fellows remaining deficient despite exposure to 50 intubation encounters (Supplemental Table 4). Each program had at least 1 fellow who met the deficiency criteria.

The remaining 10 fellows did not cross the success or failure thresholds during the period assessed. These fellows may not have performed a sufficient number of intubations to reach competence during the study period. The study period did not encompass all years of fellowship training in some cases, but other trainees remained undifferentiated at the completion of fellowship training (Supplemental Table 4).

We performed post hoc sensitivity analyses using variable definitions for competence. None of the fellows achieved procedural competency, defined more stringently as 90% success within 2 attempts. There were 29 of 40 (73%) of fellows who reached competence by using a less conservative definition of 70% success within 2 attempts. For those trainees, a median of 7 (IQR 4–11) intubations was needed to achieve this less stringent competence definition.

Across the entire cohort, intubation encounters were more frequent in the first year of training (quarters 1–4), and intubation success rates improved with advancing quarter of fellowship training (Fig 2). In univariable analysis, successful intubation encounters were performed by fellows who were slightly more advanced in the quarter of fellowship training (median quarter: 4; IQR 2–7), compared with fellows involved in unsuccessful intubations (median quarter: 3; IQR 1–6; P < .001; Table 2). In the mixed effects logistic model adjusted for paralytic premedication, video laryngoscopy use, and patient weight <1000 g, increasing quarter of fellowship training was significantly associated with an increasing odds of intubation success (adjusted odds ratio: 1.10; 95% confidence interval 1.07–1.14; Table 3).

Neonatal intubation is a critical procedure for neonatologists, but the amount of exposure needed for novice providers to acquire this skill is unknown. We were able to leverage data collected in a large multicenter neonatal intubation registry to address this question. Although intubation success rates generally increased with ongoing fellowship training, there was important variability in the number of intubations needed for individual fellows to achieve procedural competence. Equally important, some fellows were identified as deficient or undifferentiated (neither competent nor deficient), despite having exposure to 50 intubations. These findings suggest an individualized approach to assess trainee’s acquisition of intubation competence may be more useful than a uniform quota of intubation encounters.

To our knowledge, this is the first study to describe the number of neonatal intubation attempts required to achieve competence in NPM fellows. Although neonatal intubation is designated a core competence for pediatric residents by the Accreditation Council for Graduate Medical Education (ACGME) and Royal College of Physicians and Surgeons of Canada, contemporary pediatric residents have few opportunities to participate in intubations.^{9,11,16,21–23}  Therefore, neonatal intubation skill acquisition occurs almost exclusively in neonatology fellowships.

Previously described intubation success rates range from 52% in a single attempt to 78% within 2 attempts across for neonatology fellows, regardless of the training year or quarter.^7,9,11,16  In the study by Haubner et al,¹¹  third-year fellows achieved 70% success rates (similar to attending physician success rates of 72%), compared with 44% for first-year fellows and 20% for pediatric residents. Our results were similar, with 77% of intubations successful across all fellows in aggregate. Additionally, we identified improving success rates by each quarter of fellowship training.

Among fellows who achieved competence in our study, the median number of intubations to reach this threshold was 18 (IQR 15–25). The number of intubation attempts required for trainees to develop competence in other disciplines ranges from 26 to 75. This contrast is likely due to differences in definitions for procedural competence, which was set at a 90% first attempt success rate in other disciplines.^12–15,24  Consistent with previous neonatology literature, we defined competence more leniently, as an 80% success rate.^8,9  This was also informed by reported neonatology attending physicians’ success rates of 64% for the first attempt and 72% to 88% within multiple attempts.^7,11,16 

As has been previously described, changing definitions for acceptable and unacceptable failure thresholds yielded different results.²⁵  In the post hoc sensitivity analysis, no fellows in our study met the more stringent competence threshold of 90% success within 2 attempts. Going forward, it may be valuable for accreditation agencies and medical educators to define uniform competence and acceptable failure rates that can be used across studies.

Perhaps the most notable study finding was the variability among fellows’ progressive intubation performance, with several trainees not improving or worsening over time. Training programs have often used the number of procedural experiences as criteria for determining procedural competence.²⁶  Repeated practice has generally been associated with acquisition of skilled motor tasks in adults both in and outside of medicine.^27–29  However, our results suggest that repeated clinical exposure alone may not result in competence for some trainees.

Programs should develop methods to identify fellows who are unlikely to intubate successfully at an acceptable rate after fellowship completion. A real time cumulative assessment of procedural performance, such as a CUSUM plot, could help training programs assess learners’ progression toward competence and identify trainees who require additional educational interventions. In the current study, most fellows who met the deficiency threshold did so within 5 to 14 intubation encounters. In addition, given that more intubation encounters occurred in the earlier quarters of training, it is possible that some fellows’ procedural skills degrade later in training. The CUSUM method could be applied to assess skill retention by depicting whether providers continue to meet the defined competence threshold over time.

Future studies should focus on optimal methods to train fellows who are unlikely to achieve procedural competence through repeated clinical exposure alone. Deliberate practice with tailored feedback in both the simulated and clinical setting is a promising strategy.^30–32  In addition, practice characteristics, such as paralytic premedication and video laryngoscopy, were independently associated with an improved odds of procedural success; implementing these strategies could optimize trainees’ intubation success and skill acquisition. Finally, we observed important differences in the number of intubations performed by fellows at similar stages of training in our cohort. Training programs may need to strategically allocate intubation procedures for clinicians who require neonatal airway management skills and provide additional intubation training opportunities, such as rotations with anesthesiology in the operating room.

We acknowledge study limitations. The CUSUM method assesses a sequential count of encounters and does not account for the interval of time between procedures. In addition, we do not have information about fellows’ intubation experiences before starting fellowships. We anticipate that most fellows had limited intubation experience in residency, given the decline in resident participation in intubation.^{9,11,16,21,22}  Ultimately, this potential variability in baseline skill at the start of training further supports an individualized approach to assessing competence, rather than establishing a uniform quota for trainees. Finally, we were unable to assess intubations performed outside of the NICU or DR settings (such as on transport). However, all centers were selected on the basis of having at least 90% data capture for fellows when accounting for intubations in other locations.

Study strengths include analysis of a large number of intubations captured prospectively by using standardized operational definitions in the NEAR4NEOS registry. We were able to link individual fellow identifiers with intubation data captured in the registry to characterize individual fellows’ intubation learning curves. The CUSUM method provides more granular information about providers’ progressive procedural performance, compared with assessing the proportion of successful intubations among all historical encounters. In addition, the multi-site NEAR4NEOS setting increases the generalizability of our study results.

The number of neonatal intubations required to achieve procedural competence is variable among fellows, and the overall intubation competence rates are modest. Some trainees did not improve or even worsened with repeated intubation encounters. These results provide insights for neonatal educators regarding the optimal number of intubation encounters required to achieve procedural competence, tools to identify fellows at risk for not achieving competence, and the need to individualize intubation training for trainees.

CUSUM

cumulative sum

DR

delivery room

IQR

interquartile range

NEAR4NEOS

National Emergency Airway Registry for Neonates

NPM

neonatal-perinatal medicine