Simulation-Based Emergency Team Training in Pediatrics: A Systematic Review

This review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA).⁶  We registered the protocol at the International Prospective Register of Systematic Reviews repository (CRD42019128213; submitted June 24, 2019, and published September 6, 2019).

We included studies of all designs if they met the criteria stated below.

Studies including health care professionals with emergency care for pediatric patients were included. Health care professionals included nurses, physicians, paramedics, and respiratory therapists with clinical responsibilities in hospital or prehospital settings. We excluded studies on pregraduate simulation training.

Interventions included were simulation-based team training of pediatric emergencies in situ or in a training facility. We defined team training as ≥2 health care professionals in an acute clinical situation requiring a coordinated effort for a successful outcome.

We included single-group studies comparing the performance of health care professionals before (comparator) and after (intervention) simulation training, ie, pre- versus postintervention designs. Studies of team skills retention with several evaluations over time (≥2) were also included. In intervention studies with ≥2 groups, either randomized or nonrandomized, the comparator group could be subjected to usual training or no training. We accepted comparators with different types or intensities of the simulation training.

The simulation training outcomes were evaluated according to the framework of Kirkpatrick’s 4 levels of training evaluation and transferring learning to behavior.⁷  Given our aim, we did not include reaction outcomes (level 1), which correspond to professionals’ evaluation of the simulation training. Learning outcomes (level 2) were included as self-reported changes in, eg, knowledge, attitude, confidence, preparedness, self-efficacy, and technical and nontechnical skills. Technical skills were defined as adequacy of the actions taken from a mediacal and techbical perspective; nontechnical skills were defined as decision-making and team interaction processes used during the team’s management of a scenario or a clinical situation. Behavior outcomes (level 3) were included as observed technical skills and team performance (nontechnical skills) in the simulation setting. Patient outcomes (level 4) were included as a change in clinical performance of health care professionals (eg, time to critical task, adherence to protocol) or a change in patient outcomes (eg, survival) of pediatric emergencies.

We performed a combined search aimed at identifying all articles on simulation-based team training in emergency pediatrics and neonatology, doing so because simulation training programs may include both pediatric (nonneonatal) and neonatal patients. An experienced medical librarian (H.L.) and a subject specialist (M.S.L.) devised a search strategy for PubMed to identify studies concerning neonatology and pediatrics and team-based simulation training, and emergencies. The search strategy was then adapted to Embase, CINAHL, and Cochrane Library. Studies were limited to English language; however, no limit was placed on year of publication. Details of the search strategy are available in Supplemental Tables 6–8. The final search was run on February 1, 2021. After data extraction, the articles were divided into a pediatric (nonneonatal) and a neonatal review.⁵  Two studies contained relevant data on both nonneonatal and neonatal simulations and were included in both reviews.^8,9 

Two authors (S.T. and M.S.L.) independently screened titles and abstracts of all identified studies. Studies meeting or potentially meeting the inclusion criteria and studies with insufficient information were included for full-text review (see Fig 1 for the PRISMA flowchart¹⁰ ). Two authors (S.T. and M.S.L.) independently performed the full-text reviews, and any disagreements were resolved by discussion to consensus or by consulting a third author in cases of nonconsensus (T.B.H.). The study selection process was managed by the Covidence review platform (https://www.covidence.org), which facilitates co-reviewer blinded abstract and full-text screening, study inclusion, and resolution of conflicts.

Two authors (S.T. and A.W.S.) extracted data by using a predefined template developed for the purpose. Authors of studies with missing or inadequate information were not contacted. The following information was extracted from the studies: author, year, country, simulation setting, study design, intervention, comparator, number of participants, profession of participants, participant/instructor ratio, simulation and debriefing duration, simulation frequency (if applicable), level of fidelity, in situ or simulation center, retest timing (if applicable), self-reported outcomes, observed simulation outcomes, observed clinical behavior outcomes, change in clinical performance of health care professionals outcomes, and change in patient outcomes.

Two authors (M.S.L. and A.W.S.) evaluated randomized studies according to the revised Cochrane risk-of-bias tool for randomized trials 2.0 (RoB 2).¹¹  Separate score tools were applied for individually randomized parallel group trials and for cluster randomized trials according to study design. The same 2 authors evaluated nonrandomized studies according to the Newcastle Ottawa Quality Assessment Scale (NOS).¹¹  We used the NOS with the adaptations and operational definitions for evaluating medical education research as suggested by Cook and Reed.¹² 

The risk-of-bias evaluation was presented for all randomized studies, regardless of outcome studied. For nonrandomized studies, the risk-of-bias evaluation was presented for studies reporting clinical performance or patient outcomes, regardless of study type, and for studies reporting team performance outcomes with a control group. The risk-of-bias evaluation was used for the synthesis and discussion of the results of this systematic review. The bias evaluation had no impact on study inclusion or exclusion.

We performed a narrative synthesis of the included studies because of lack of comparability in design and interventions and heterogeneous outcomes. No single summary measure was applicable. We focused the main synthesis on the hierarchical Kirkpatrick levels and high-quality studies. We presented all available studies for each level to provide full transparency (Table 1). In our reporting of the results, statistical significance was considered at the P < .05 level. Estimates and P values are reported in the tables.

We used a funnel plot to assess potential publication bias. We used the estimate from the main outcome of each included study. Ratio estimates (odds ratios, risk ratios, incidence rate ratios) were used directly, whereas a ratio was calculated for studies using continuous outcomes, eg, intervention mean divided by comparator mean. A ratio >1.0 favored the intervention group or the postintervention estimate. Eighteen studies did not report the number of participants or effect estimates and were therefore excluded from the funnel plot.

We assessed selective reporting by inspecting preregistered studies or analysis protocols when available; for randomized studies, this was part of the RoB 2 score. Furthermore, we compared the specified analysis from the methods section with the reported outcomes in each included study.

A total of 3262 studies were identified across databases. After duplicate removal, we screened titles and abstracts of 1926 studies. A total of 223 full-text articles were assessed for eligibility. Of these, 110 met the inclusion criteria common to pediatric and neonatal populations, and 79 provided relevant data for this pediatric review (Fig 1). An overview of the excluded full-text articles is provided in the Supplemental Table 9.

Included studies were all published between 2006 and 2021 (Table 1). Eighteen studies contained a control group, and 9 of these used random allocation. Sixteen studies had a time series intervention group design, and the remaining 45 had a pre- and postintervention group design. Several studies reported outcomes at >1 Kirkpatrick level. Fifteen studies reported patient outcome (level 4), 37 reported behavior outcome (level 3), and 47 reported learning outcome (level 2) (Table 1).

The risk-of-bias judgments of the 9 randomized studies are presented in Table 2. Five studies received an overall judgment of some concern,^13–17  and 4 received a high-risk judgment.^18–21  Only 2 studies were judged as low risk for the crucial randomization domain.^15,17  All studies had low risk of bias in the intervention domain, reflecting no deviations in the allocated intervention groups and intention-to-treat analyses.

The risk-of-bias scores for the 15 nonrandomized studies with clinical performance or patient outcomes (level 4) regardless of study type and the 1 nonrandomized study with a control group reporting team performance outcomes (level 3) are presented in Table 3. Two cohort studies received an overall score of 4 out of 6,^22,23  and 8 studies received a score of ≤2, including 1 cohort study with low study retention.²⁴ 

We included 15 studies that reported change in clinical performance of health care professionals or change in clinical outcomes (Table 4). Only 4 of the studies had a control group, but none of these used random allocation. Four of the studies simulated extracorporeal membrane oxygenation (ECMO),^25–28  1 septic shock,²⁹  1 anaphylaxis,³⁰  1 seizure,²²  2 deteriorating patients,^31,32  2 cardiopulmonary arrest,^33,34  2 airway management,^23,35  and 2 closed-loop communication and teamwork skills.^36,37 

Four studies reported survival or mortality (Table 4). Sawyer et al²⁵  performed simulation-based team training in 3 ICUs in a children’s hospital and compared survival after extracorporeal cardiopulmonary resuscitation (CPR) before and during the simulation program. The authors found a 24-hour survival rate of 56% in the period of simulation compared with 45% in the period before simulation. They also found a survival to discharge rate of 38% compared with 27% before simulation. Di Nardo et al²⁶  compared survival before and after an ECMO simulation program at a children’s hospital. They reported a 50% survival rate before intervention compared with 57% after intervention. Theilen et al³¹  explored an intervention with 2 hours of weekly simulation and introduction of a pediatric medical emergency team where each team member had simulation training 4 to 10 times per year. They compared the period before and after introduction of the team and simulation training. They found that overall mortality was significantly reduced. Finally, Andreatta et al³³  performed unannounced simulation mock codes at a minimum of once every month. They reported cardiopulmonary arrest survival rates over a 4-year period, including at the introduction of the simulation program. Survival rates increased from 33% before to ∼50% after simulation was introduced, and the reduction correlated with the number of mock codes.

Six studies reported time to a specific clinical event as outcome (Table 4). Four of these studies investigated time to manage ECMO emergencies (eg, pump failure, circuit failure) and time to initiate ECMO (eg, activation, cannulation, ECMO call to flow initiation). Authors of 3 of the 4 studies reported clinically and statistically significant reductions in time to important ECMO tasks after initiating simulation-based team training in their own hospital compared with before.^25–27  However, 1 study investigated time from blood available to circuit ready and did not find a significant change.²⁸ 

Theilen et al³¹  introduced a pediatric medical emergency team at their hospital where each team member had simulation training 4 to 10 times per year. Compared with a period before introduction of the team and simulation training, they observed a reduction from 4 hours to 1.5 hours from warning sign to first response and a reduction from 10.5 hours to 5 hours in time to pediatric ICU admission. Hazwani et al³⁴  initiated mock code simulations and reported reduced CPR initiation time compared with the period before in the same hospital.

Seven studies reported adherence to guidelines or compliance to checklists or tasks (Table 4). Qian et al²⁹  conducted a cohort study that initiated simulation-based training at 1 hospital, using 2 other hospitals as control. They investigated compliance with first-hour critical tasks in septic shock and reported a significantly higher compliance rate of 62% at the intervention hospital compared with 23% at the control hospitals. Barni et al³⁰  investigated the use of epinephrine for anaphylaxis according to guidelines in a pediatric emergency department. Compared with before the simulation intervention, they observed a significant increase in the use of epinephrine from 2.4% to 10%. Shah et al²²  found that a simulation-based course slightly improved adherence to prehospital seizure protocol, ie, blood glucose measurement and midazolam administration. Authors of 2 studies investigating ECMO reported improvement in compliance with an activation protocol²⁵  and an initiation checklist²⁸  after starting simulation-based training. Hazwani et al³⁴  found improved compliance with guidelines in the treatment of cardiopulmonary arrest after a mock code simulation program, and Diaz and Dawson³⁶  found a reduction in medication errors after simulation that focused on closed-loop communication.

Two studies reported airway management outcomes (Table 4). Nishisaki et al²³  investigated the effect of simulation-based intubation training and skill refresher training in the beginning of an on-call period. They found no difference in first-attempt and overall success of intubation between residents receiving refresher training or not on the basis of real pediatric intubations. Nishisaki et al³⁵  later investigated effects of simulation training on clinical performance and team behavior during intubations as evaluated by a trained observer in a PICU. The team performance was significantly higher in teams with ≥2 simulation-trained members (both technical and behavioral scores) compared with teams with <2 trained members.

Diaz and Dawson³⁶  evaluated closed-loop communication and found a significant reduction in medication errors. Colman et al³⁷  evaluated composite teamwork skills and found significant improvement in 11 of 15 skills after a period of simulation training.

We included 37 studies with observed team performance outcomes. Twenty-nine of these had no control group and are not further described here (Table 1). Below, we describe the 8 studies with a control group, 7 of which used random allocation (Table 5).

Seven of the 8 studies evaluated retention of technical and nontechnical skills using simulation-based retesting of teams. Authors of 6 of these 7 studies retested after 2 to 6 months and found significant improvements in intervention groups compared with control groups.^{14,16,18,20,21,24}  Mariani et al¹⁵  performed a retest after 9 months and found no difference in skill competency between the groups. Stellflug and Lowe²¹  investigated random allocation to either high-fidelity or low-fidelity team training and reported better performance on predefined skills for the high-fidelity group when retested after 6 months. Lemke et al²⁰  investigated random allocation to either rapid-cycle deliberate practice or traditional team training with debriefing and reported improvement in the human factor subscore (eg, team organization, communication, knowledge sharing) for the intervention group after 3 months. Bultas et al¹⁸  investigated a teaching strategy for recognizing a deteriorating pediatric patient, with random allocation to either high-fidelity simulation or traditional training, and reported significantly better teamwork and skills and tasks scores during a respiratory scenario in the intervention group at 6-month retest. Kurosawa et al¹⁴  performed a randomized trial of a reconstructed recertification of pediatric advanced life support with simulation once a month compared with standard recertification. They found significantly improved skills performance when retesting after 6 months compared with the control group. Sudikoff et al¹⁶  conducted a randomized cross-over study of emergency airway management and teamwork. They found improved global competency scores with regard to communication, leadership, and team skills, and the improvement persisted for up to 4 months. Gilfoyle et al²⁴  conducted a workshop with simulation-based training and investigated team leadership skills in pediatric advanced resuscitation. They found a significant increase in checklist scores after 6 months in the intervention group compared with nontrained residents.

Five of the 8 studies evaluated teamwork competencies such as leadership, communication, human factors, and teamwork confidence. In 4 of 5 studies, the intervention group had significantly better team performance scores than the control group.^13,18,20,24 

We included 47 studies with self-reported knowledge or confidence. Thirty-eight of these had no control group and are not further described here (Table 1). Five of the studies had a control group with random allocation. Bragard et al¹³  performed a team training program on cardiac arrhythmias and found reduced perceived stress and improved satisfaction with communication, general skills, and communication skills compared with the control group. Stellflug and Lowe²¹  investigated random allocation to either high-fidelity or low-fidelity team training and found that the low-fidelity group declined more in ratings of their confidence levels on performing 19 different skills or tasks on retest at 6 months. Fagan et al¹⁹  investigated random allocation to recertification of pediatric advanced life support with or without simulation-based team training. Members of the intervention group reported higher perception of teamwork and more situational awareness. Mariani et al¹⁵  investigated the effect of repeated pediatric mock code simulations and found no difference in self-confidence between the groups. Besbes et al¹⁷  found that knowledge test scores improved significantly for both groups, although the improvement was significantly higher in the simulation group.

Four of the studies had a control group without random allocation. McLaughlin et al³⁸  investigated the impact of team training on perceived confidence when handling pediatric trauma patients, and at 2 years follow-up, trained professionals reported less anxiety and greater confidence compared with professionals who did not receive the simulation training. Dowson et al³⁹  investigated the impact of team training on pediatric nurses’ self-assessed clinical confidence and found a significant improvement. Auerbach et al⁴⁰  compared repetitive pediatric simulation training (scenario-debriefing-scenario) to standard simulation training and found that the intervention group reported greater improvement in knowledge and skills. Happel et al⁴¹  evaluated the perceived impact on live critical events in a period with weekly simulation-based team training. The residents in the intervention group reported positive effects on their behaviors and confidence levels as well as feeling more prepared; however, these changes did not differ significantly from the control group.

Within each group of studies (according to Table 1), the funnel plot raised no major concern about publication bias (Fig 2). In the funnel plot, studies with and without control groups appeared similar across study size and effect ratio. We found no indication of selective reporting bias, as methods and results sections were coherent in all studies. Prepublished study protocols were unavailable on all randomized studies, which was reflected in the risk-of-bias score.

We performed this systematic review to describe the current state of evidence in the evolving research field of simulation-based team training in pediatrics. Summarizing effects on patient outcomes (level 4), 2 important findings deserve emphasis.

First, 4 studies indicated improved survival after introducing simulation-based team training.^25,26,31,33  However, given study design limitations, future studies with higher methodological standards would enable more robust conclusions. The authors of the 4 time series studies reported survival rates before and after introduction of a simulation program. The findings were prone to confounding from uncontrolled or unknown factors. For example, Theilen et al³¹  introduced a pediatric medical emergency team at the same time as the simulation program, which hampers conclusions related to effects of simulation-based training per se.

Second, authors of 4 studies designed with a control group (nonrandomized) indicated effects of simulation-based team training on clinical performance, eg, improved compliance to first-hour critical tasks in sepsis treatment,²⁹  enhanced seizure protocol adherence,²²  and improved intubation performance if ≥2 simulation-trained professionals were present.^23,35  Overall, these controlled studies may indicate effects on treatment quality, but ambiguity and lack of precision preclude a strong conclusion. Authors of several studies with a time series design observed improved clinical performance after the introduction of simulation programs (Table 4). However, their conclusions were prone to bias from uncontrolled or unknown factors, eg, other changes during the period of intervention.

In considering team performance outcomes (Kirkpatrick level 3) and learning outcomes (Kirkpatrick level 2), 2 important findings should be noted. First, authors of several randomized studies found improved team performance when retesting 2 to 6 months after intervention.^{14,16,18,20,21,24}  Studies with control groups revealed that team performance, eg, leadership, communication, human factors, and teamwork confidence, improved.^13,18,20,24  Second, authors of most studies with random allocation showed improved perceived confidence, skill levels, and ability to communicate in emergency situations after the introduction of simulation-based team training. Overall, the findings from these controlled studies support improved team performance, knowledge, and confidence after simulation training.

This systematic review applied a comprehensive search strategy in 4 medical databases by an experienced medical librarian. We followed a prespecified study protocol registered in the Prospective Register of Systematic Reviews repository. Two reviewers independently screened and selected studies and performed data extraction and risk-of-bias scoring. Data were presented according to the PRISMA guidelines. The funnel plot in Fig 2 indicates no publication bias.

Meta-analysis was impossible because of heterogeneous study designs, interventions, and outcomes in the included studies. Instead, we performed a narrative synthesis structured by hierarchical outcome, including patient and clinical performance outcomes, team performance outcomes, and learning outcomes. Although this process is less standardized than a meta-analysis, we consider it reproducible and open to scrutiny. Most included studies had significant methodological limitations. Sixty-one of 79 studies did not include a control group, which is concerning in studies using self-reported or simulation-based end points. Most nonrandomized studies had inadequate adjustment for potential confounding factors, such as years of clinical experience, profession, team composition, age, and sex. Small sample size (<50 in 33 studies and not described in 16 studies) may likely have resulted in inadequate power and an inability to perform properly adjusted statistical analyses.

Measuring effects of simulation-based training on clinical outcomes should be preferred whenever possible. However, studying rare clinical end points, such as mortality or severe morbidity, requires large, multicenter studies. Therefore, other clinical outcomes may be more feasible to obtain and may serve as surrogates related to treatment quality, eg, time to critical tasks or guideline adherence. We acknowledge that real-life evaluation of training effects may be difficult and time consuming given the paucity and volatile nature of critical clinical events. We advocate the use of a control group when designing new studies and random allocation to intervention and control whenever possible. Study protocols should be registered to avoid selective reporting. When applying simulation-based evaluation of training effects, we recommend the use of video recordings and blinded assessment using accepted and validated protocols. Compared with planned simulation-based retesting, the use of unannounced simulated mock codes may mimic real clinical encounters more closely because clinical professionals often prepare for planned testing. We recommend that the theoretical framework and process of debriefing is described, as these are important parts of simulation-based training that may affect the learning outcome.

To our knowledge, this systematic review of simulation-based pediatric team training is the first with an aim to collectively describe all current available literature on clinical, team performance, and learning outcomes. We included 79 studies published over the past 15 years and conclude that pediatric simulation-based team training improves clinical performance in time-critical tasks and increases adherence to guidelines. Simulation-based training improves team performance and technical skills for at least 2 to 6 months. Furthermore, it may improve survival rates, but these results are affected by heterogenous interventions and outcomes and studies without a control group. Future research should include longer-term measures of retention of team performance and other skills and patient outcomes or clinical measures of treatment quality whenever possible.

The authors acknowledge and appreciate the scientific contribution of the authors of all articles included in this review. The authors thank the editor and reviewers for valuable comments on an earlier version of this manuscript.

CPR

cardiopulmonary resuscitation

ECMO

extracorporeal membrane oxygenation

NOS

Newcastle Ottawa Quality Assessment Scale

PRISMA

Preferred Reporting Items for Systematic Reviews and Meta-Analysis

RoB 2

Cochrane risk-of-bias tool for randomized trials 2.0