The rare event of handling critically ill children often challenge the emergency care team. Several studies have investigated effects of simulation-based team training to prepare for such events, but the body of evidence remains to be compiled. We performed a systematic review of the effects of simulation-based team training on clinical performance and patient outcome.
From a search of MEDLINE, Embase, CINAHL, and Cochrane Library, we included studies of team training in emergency pediatric settings with reported clinical performance and patient outcomes. We extracted data using a predefined template and assessed risk of bias using the Cochrane risk-of-bias tool for randomized trials 2.0 and the Newcastle Ottawa Quality Assessment Scale.
We screened 1926 abstracts and included 79 studies. We identified 15 studies reporting clinical health care professional performance or patient outcomes. Four studies reported survival data, 5 reported time-critical clinical events, 5 reported adherence to guidelines, checklists or tasks, and 2 reported on airway management. Randomized studies revealed improved team performance in simulated reevaluations 2 to 6 months after intervention. A meta-analysis was impossible because of heterogeneous interventions and outcomes. Most included studies had significant methodological limitations.
Pediatric simulation-based team training improves clinical performance in time-critical tasks and adherence to guidelines. Improved survival was indicated but not concluded because of high risk of bias. Team performance and technical skills improved for at least 2 to 6 months. Future research should include longer-term measures of skill retention and patient outcomes or clinical measures of treatment quality whenever possible.
Pediatricians are expected to manage acutely ill children in high-stakes, time-sensitive situations. Pediatrics is a broad specialty with a wide range of subspecialties, patient ages, and complex clinical conditions. Therefore, specific acute situations are rare, and as a consequence, pediatricians are rarely exposed to them, emphasizing the importance of maintaining a coordinated and efficient emergency team response.1
Simulation-based team training is defined as simulation beyond technical skills that aims to mimic real acute situations where performance is influenced by teamwork, coordination, team behavior, communication, leadership, decision-making, and situational awareness.2–4 The use of advanced medical simulation training in pediatrics has increased rapidly in recent years.2 However, little is known about the retention of attained skills, knowledge, and behavior and how these can transfer to clinical encounters and emergency situations and, ultimately, how they may improve patient safety and outcome.
We have previously reviewed simulation-based team training in neonatal resuscitation and concluded that it improves team performance.5 However, no firm conclusions on survival and other clinical outcomes could be reached. We were unable to identify a systematic review of simulation-based team training in the pediatric nonneonatal setting to answer the following questions: Does simulation-based team training improve the performance of the team? Does it improve patient outcome and safety? We therefore performed a systematic review to describe the current state of evidence and point out areas needing more research.
Methods
This review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-Analysis (PRISMA).6 We registered the protocol at the International Prospective Register of Systematic Reviews repository (CRD42019128213; submitted June 24, 2019, and published September 6, 2019).
Study Eligibility Criteria
We included studies of all designs if they met the criteria stated below.
Population
Studies including health care professionals with emergency care for pediatric patients were included. Health care professionals included nurses, physicians, paramedics, and respiratory therapists with clinical responsibilities in hospital or prehospital settings. We excluded studies on pregraduate simulation training.
Intervention
Interventions included were simulation-based team training of pediatric emergencies in situ or in a training facility. We defined team training as ≥2 health care professionals in an acute clinical situation requiring a coordinated effort for a successful outcome.
Comparators
We included single-group studies comparing the performance of health care professionals before (comparator) and after (intervention) simulation training, ie, pre- versus postintervention designs. Studies of team skills retention with several evaluations over time (≥2) were also included. In intervention studies with ≥2 groups, either randomized or nonrandomized, the comparator group could be subjected to usual training or no training. We accepted comparators with different types or intensities of the simulation training.
Outcomes
The simulation training outcomes were evaluated according to the framework of Kirkpatrick’s 4 levels of training evaluation and transferring learning to behavior.7 Given our aim, we did not include reaction outcomes (level 1), which correspond to professionals’ evaluation of the simulation training. Learning outcomes (level 2) were included as self-reported changes in, eg, knowledge, attitude, confidence, preparedness, self-efficacy, and technical and nontechnical skills. Technical skills were defined as adequacy of the actions taken from a mediacal and techbical perspective; nontechnical skills were defined as decision-making and team interaction processes used during the team’s management of a scenario or a clinical situation. Behavior outcomes (level 3) were included as observed technical skills and team performance (nontechnical skills) in the simulation setting. Patient outcomes (level 4) were included as a change in clinical performance of health care professionals (eg, time to critical task, adherence to protocol) or a change in patient outcomes (eg, survival) of pediatric emergencies.
Search Strategy
We performed a combined search aimed at identifying all articles on simulation-based team training in emergency pediatrics and neonatology, doing so because simulation training programs may include both pediatric (nonneonatal) and neonatal patients. An experienced medical librarian (H.L.) and a subject specialist (M.S.L.) devised a search strategy for PubMed to identify studies concerning neonatology and pediatrics and team-based simulation training, and emergencies. The search strategy was then adapted to Embase, CINAHL, and Cochrane Library. Studies were limited to English language; however, no limit was placed on year of publication. Details of the search strategy are available in Supplemental Tables 6–8. The final search was run on February 1, 2021. After data extraction, the articles were divided into a pediatric (nonneonatal) and a neonatal review.5 Two studies contained relevant data on both nonneonatal and neonatal simulations and were included in both reviews.8,9
Study Selection
Two authors (S.T. and M.S.L.) independently screened titles and abstracts of all identified studies. Studies meeting or potentially meeting the inclusion criteria and studies with insufficient information were included for full-text review (see Fig 1 for the PRISMA flowchart10 ). Two authors (S.T. and M.S.L.) independently performed the full-text reviews, and any disagreements were resolved by discussion to consensus or by consulting a third author in cases of nonconsensus (T.B.H.). The study selection process was managed by the Covidence review platform (https://www.covidence.org), which facilitates co-reviewer blinded abstract and full-text screening, study inclusion, and resolution of conflicts.
Data Extraction
Two authors (S.T. and A.W.S.) extracted data by using a predefined template developed for the purpose. Authors of studies with missing or inadequate information were not contacted. The following information was extracted from the studies: author, year, country, simulation setting, study design, intervention, comparator, number of participants, profession of participants, participant/instructor ratio, simulation and debriefing duration, simulation frequency (if applicable), level of fidelity, in situ or simulation center, retest timing (if applicable), self-reported outcomes, observed simulation outcomes, observed clinical behavior outcomes, change in clinical performance of health care professionals outcomes, and change in patient outcomes.
Risk of Bias in Individual Studies
Two authors (M.S.L. and A.W.S.) evaluated randomized studies according to the revised Cochrane risk-of-bias tool for randomized trials 2.0 (RoB 2).11 Separate score tools were applied for individually randomized parallel group trials and for cluster randomized trials according to study design. The same 2 authors evaluated nonrandomized studies according to the Newcastle Ottawa Quality Assessment Scale (NOS).11 We used the NOS with the adaptations and operational definitions for evaluating medical education research as suggested by Cook and Reed.12
The risk-of-bias evaluation was presented for all randomized studies, regardless of outcome studied. For nonrandomized studies, the risk-of-bias evaluation was presented for studies reporting clinical performance or patient outcomes, regardless of study type, and for studies reporting team performance outcomes with a control group. The risk-of-bias evaluation was used for the synthesis and discussion of the results of this systematic review. The bias evaluation had no impact on study inclusion or exclusion.
Synthesis of Results
We performed a narrative synthesis of the included studies because of lack of comparability in design and interventions and heterogeneous outcomes. No single summary measure was applicable. We focused the main synthesis on the hierarchical Kirkpatrick levels and high-quality studies. We presented all available studies for each level to provide full transparency (Table 1). In our reporting of the results, statistical significance was considered at the P < .05 level. Estimates and P values are reported in the tables.
Characteristics of 79 Included Studies Sorted by Study Design, Outcome Kirkpatrick Level, and Year
. | . | . | . | . | Outcome Kirkpatrick Levela . | ||
---|---|---|---|---|---|---|---|
Source . | Year . | Country . | Design . | No. of Participants . | 2 . | 3 . | 4 . |
Besbes et al17 | 2021 | Tunisia | Randomized | 33 | X | ||
Bragard et al13 | 2019 | Belgium | Randomized | 16 | X | X | |
Mariani et al15 | 2019 | United States | Randomized | 18 | X | X | |
Stellflug and Lowe21 | 2018 | United States | Randomized | 94 | X | X | |
Fagan et al19 | 2018 | United States | Randomized | 26 | X | ||
Lemke et al20 | 2019 | United States | Randomized | 30 | X | ||
Bultas et al18 | 2014 | United States | Randomized | 33 | X | ||
Kurosawa et al14 | 2014 | United States | Randomized | 40 | X | ||
Sudikoff et al16 | 2009 | United States | Randomized | 16 | X | ||
Happel et al41 | 2015 | United States | Cohort | NA | X | ||
McLaughlin et al38 | 2018 | United States | Cohort | 149 | X | ||
Shah et al22 | 2016 | United States | Cohort | 250 | X | ||
Qian et al29 | 2016 | China | Cohort | NA | X | ||
Dowson et al39 | 2013 | England | Cohort | 20 | X | ||
Auerbach et al40 | 2011 | United States | Cohort | 115 | X | ||
Nishisaki et al35 | 2011 | United States | Cohort | 265 | X | ||
Nishisaki et al23 | 2010 | United States | Cohort | 265 | X | ||
Gilfoyle et al24 | 2007 | Canada | Cohort | 15 | X | ||
Lee et al42 | 2021 | United States | Time series | NA | X | ||
Diaz and Dawson36 | 2020 | United States | Time series | 112 | X | ||
Hazwani et al34 | 2020 | Saudi Arabia | Time series | NA | X | ||
Hazwani et al43 | 2020 | Saudi Arabia | Time series | NA | X | ||
Colman et al37 | 2019 | United States | Time series | 165 | X | X | |
Colman et al44 | 2019 | United States | Time series | 165 | X | ||
Sawyer et al25 | 2019 | United States | Time series | 332 | X | ||
Barni et al30 | 2019 | Italy | Time series | 30 | X | ||
Di Nardo et al26 | 2018 | Italy | Time series | NA | X | X | |
Yager et al45 | 2016 | United States | Time series | NA | X | ||
Auerbach et al46 | 2014 | United States | Time series | 398 | X | ||
Su et al27 | 2014 | United States | Time series | NA | X | ||
Theilen et al31 | 2013 | United Kingdom | Time series | NA | X | ||
Andreatta et al33 | 2011 | United States | Time series | 228 | X | ||
van Schaik et al47 | 2011 | United States | Time series | 61 | X | ||
Falcone et al48 | 2008 | United States | Time series | 160 | X | ||
Brown et al49 | 2021 | United States | Pre/post | NA | X | X | |
Cory et al50 | 2020 | United States | Pre/post | 144 | X | X | |
Karageorge et al51 | 2020 | United States | Pre/post | 24 | X | X | |
Lutfi et al52 | 2019 | United States | Pre/post | 367 | X | ||
Saqe-Rockoff et al53 | 2019 | United States | Pre/post | 43 | X | X | |
Bayouth et al54 | 2018 | United States | Pre/post | 99 | X | ||
Brown et al55 | 2018 | United States | Pre/post | 30 | X | X | |
Emani et al56 | 2018 | Asia | Pre/post | 23 | X | X | |
Ryan et al57 | 2019 | United States | Pre/post | 7 | X | X | |
Katznelson et al58 | 2018 | United States | Pre/post | NA | X | ||
Couloures et al59 | 2017 | United States | Pre/post | 23 | X | X | |
Gilfoyle et al60 | 2017 | United States | Pre/post | 300 | X | ||
Martin et al32 | 2016 | United States | Pre/post | 83 | X | X | X |
Stone et al61 | 2014 | United States | Pre/post | 60 | X | ||
Chan et al62 | 2013 | United States | Pre/post | 26 | X | X | |
Kennedy et al63 | 2013 | United States | Pre/post | 26 | X | ||
Patterson et al64 | 2013 | United States | Pre/post | 289 | X | X | |
Patterson et al65 | 2013 | United States | Pre/post | 218 | X | ||
Burton et al28 | 2011 | United States | Pre/post | 19 | X | X | X |
Tofil et al66 | 2011 | United States | Pre/post | 30 | X | X | |
Hunt et al 67 | 2007 | United States | Pre/post | NA | X | ||
Tsai et al68 | 2006 | Canada | Pre/post | 18 | X | ||
Peterson et al69 | 2020 | United States | Pre/post | 27 | X | ||
Monachino et al70 | 2019 | United States | Pre/post | 211 | X | ||
Abulebda et al71 | 2018 | United States | Pre/post | NA | X | ||
Bragard et al9 | 2018 | Belgium | Pre/post | 16 | X | ||
Cristallo et al72 | 2021 | United States | Pre/post | 353 | X | ||
Lind et al73 | 2018 | United States | Pre/post | 39 | X | ||
Kalidindi et al74 | 2018 | United States | Pre/post | NA | X | ||
Raffaeli et al75 | 2018 | Italy | Pre/post | 28 | X | ||
Saavedra et al76 | 2018 | United States | Pre/post | 85 | X | ||
Whitfill et al77 | 2018 | United States | Pre/post | NA | X | ||
Lehner et al78 | 2017 | Germany | Pre/post | 18 | X | ||
Wallace et al79 | 2017 | United States | Pre/post | 260 | X | ||
Ross et al8 | 2016 | United States | Pre/post | 17 | X | ||
Katznelson et al80 | 2014 | United States | Pre/post | NA | X | ||
Figueroa et al81 | 2013 | United States | Pre/post | 37 | X | ||
Popp et al82 | 2012 | United States | Pre/post | 18 | X | ||
Stocker et al83 | 2012 | United Kingdom | Pre/post | 219 | X | ||
Straka et al84 | 2012 | United States | Pre/post | 26 | X | ||
Kane et al85 | 2011 | United States | Pre/post | 65 | X | ||
Allan et al86 | 2010 | United States | Pre/post | 182 | X | ||
Tofil et al87 | 2010 | United States | Pre/post | 30 | X | ||
von Arx and Pretzlaff88 | 2010 | United States | Pre/post | 27 | X | ||
Toback et al89 | 2006 | United States | Pre/post | 97 | X |
. | . | . | . | . | Outcome Kirkpatrick Levela . | ||
---|---|---|---|---|---|---|---|
Source . | Year . | Country . | Design . | No. of Participants . | 2 . | 3 . | 4 . |
Besbes et al17 | 2021 | Tunisia | Randomized | 33 | X | ||
Bragard et al13 | 2019 | Belgium | Randomized | 16 | X | X | |
Mariani et al15 | 2019 | United States | Randomized | 18 | X | X | |
Stellflug and Lowe21 | 2018 | United States | Randomized | 94 | X | X | |
Fagan et al19 | 2018 | United States | Randomized | 26 | X | ||
Lemke et al20 | 2019 | United States | Randomized | 30 | X | ||
Bultas et al18 | 2014 | United States | Randomized | 33 | X | ||
Kurosawa et al14 | 2014 | United States | Randomized | 40 | X | ||
Sudikoff et al16 | 2009 | United States | Randomized | 16 | X | ||
Happel et al41 | 2015 | United States | Cohort | NA | X | ||
McLaughlin et al38 | 2018 | United States | Cohort | 149 | X | ||
Shah et al22 | 2016 | United States | Cohort | 250 | X | ||
Qian et al29 | 2016 | China | Cohort | NA | X | ||
Dowson et al39 | 2013 | England | Cohort | 20 | X | ||
Auerbach et al40 | 2011 | United States | Cohort | 115 | X | ||
Nishisaki et al35 | 2011 | United States | Cohort | 265 | X | ||
Nishisaki et al23 | 2010 | United States | Cohort | 265 | X | ||
Gilfoyle et al24 | 2007 | Canada | Cohort | 15 | X | ||
Lee et al42 | 2021 | United States | Time series | NA | X | ||
Diaz and Dawson36 | 2020 | United States | Time series | 112 | X | ||
Hazwani et al34 | 2020 | Saudi Arabia | Time series | NA | X | ||
Hazwani et al43 | 2020 | Saudi Arabia | Time series | NA | X | ||
Colman et al37 | 2019 | United States | Time series | 165 | X | X | |
Colman et al44 | 2019 | United States | Time series | 165 | X | ||
Sawyer et al25 | 2019 | United States | Time series | 332 | X | ||
Barni et al30 | 2019 | Italy | Time series | 30 | X | ||
Di Nardo et al26 | 2018 | Italy | Time series | NA | X | X | |
Yager et al45 | 2016 | United States | Time series | NA | X | ||
Auerbach et al46 | 2014 | United States | Time series | 398 | X | ||
Su et al27 | 2014 | United States | Time series | NA | X | ||
Theilen et al31 | 2013 | United Kingdom | Time series | NA | X | ||
Andreatta et al33 | 2011 | United States | Time series | 228 | X | ||
van Schaik et al47 | 2011 | United States | Time series | 61 | X | ||
Falcone et al48 | 2008 | United States | Time series | 160 | X | ||
Brown et al49 | 2021 | United States | Pre/post | NA | X | X | |
Cory et al50 | 2020 | United States | Pre/post | 144 | X | X | |
Karageorge et al51 | 2020 | United States | Pre/post | 24 | X | X | |
Lutfi et al52 | 2019 | United States | Pre/post | 367 | X | ||
Saqe-Rockoff et al53 | 2019 | United States | Pre/post | 43 | X | X | |
Bayouth et al54 | 2018 | United States | Pre/post | 99 | X | ||
Brown et al55 | 2018 | United States | Pre/post | 30 | X | X | |
Emani et al56 | 2018 | Asia | Pre/post | 23 | X | X | |
Ryan et al57 | 2019 | United States | Pre/post | 7 | X | X | |
Katznelson et al58 | 2018 | United States | Pre/post | NA | X | ||
Couloures et al59 | 2017 | United States | Pre/post | 23 | X | X | |
Gilfoyle et al60 | 2017 | United States | Pre/post | 300 | X | ||
Martin et al32 | 2016 | United States | Pre/post | 83 | X | X | X |
Stone et al61 | 2014 | United States | Pre/post | 60 | X | ||
Chan et al62 | 2013 | United States | Pre/post | 26 | X | X | |
Kennedy et al63 | 2013 | United States | Pre/post | 26 | X | ||
Patterson et al64 | 2013 | United States | Pre/post | 289 | X | X | |
Patterson et al65 | 2013 | United States | Pre/post | 218 | X | ||
Burton et al28 | 2011 | United States | Pre/post | 19 | X | X | X |
Tofil et al66 | 2011 | United States | Pre/post | 30 | X | X | |
Hunt et al 67 | 2007 | United States | Pre/post | NA | X | ||
Tsai et al68 | 2006 | Canada | Pre/post | 18 | X | ||
Peterson et al69 | 2020 | United States | Pre/post | 27 | X | ||
Monachino et al70 | 2019 | United States | Pre/post | 211 | X | ||
Abulebda et al71 | 2018 | United States | Pre/post | NA | X | ||
Bragard et al9 | 2018 | Belgium | Pre/post | 16 | X | ||
Cristallo et al72 | 2021 | United States | Pre/post | 353 | X | ||
Lind et al73 | 2018 | United States | Pre/post | 39 | X | ||
Kalidindi et al74 | 2018 | United States | Pre/post | NA | X | ||
Raffaeli et al75 | 2018 | Italy | Pre/post | 28 | X | ||
Saavedra et al76 | 2018 | United States | Pre/post | 85 | X | ||
Whitfill et al77 | 2018 | United States | Pre/post | NA | X | ||
Lehner et al78 | 2017 | Germany | Pre/post | 18 | X | ||
Wallace et al79 | 2017 | United States | Pre/post | 260 | X | ||
Ross et al8 | 2016 | United States | Pre/post | 17 | X | ||
Katznelson et al80 | 2014 | United States | Pre/post | NA | X | ||
Figueroa et al81 | 2013 | United States | Pre/post | 37 | X | ||
Popp et al82 | 2012 | United States | Pre/post | 18 | X | ||
Stocker et al83 | 2012 | United Kingdom | Pre/post | 219 | X | ||
Straka et al84 | 2012 | United States | Pre/post | 26 | X | ||
Kane et al85 | 2011 | United States | Pre/post | 65 | X | ||
Allan et al86 | 2010 | United States | Pre/post | 182 | X | ||
Tofil et al87 | 2010 | United States | Pre/post | 30 | X | ||
von Arx and Pretzlaff88 | 2010 | United States | Pre/post | 27 | X | ||
Toback et al89 | 2006 | United States | Pre/post | 97 | X |
NA, not available (information not provided in the original article); Pre/post, pre- and postintervention study.
Kirkpatrick level 2 (learning), level 3 (clinical performance), and level 4 (patient outcome).
We used a funnel plot to assess potential publication bias. We used the estimate from the main outcome of each included study. Ratio estimates (odds ratios, risk ratios, incidence rate ratios) were used directly, whereas a ratio was calculated for studies using continuous outcomes, eg, intervention mean divided by comparator mean. A ratio >1.0 favored the intervention group or the postintervention estimate. Eighteen studies did not report the number of participants or effect estimates and were therefore excluded from the funnel plot.
We assessed selective reporting by inspecting preregistered studies or analysis protocols when available; for randomized studies, this was part of the RoB 2 score. Furthermore, we compared the specified analysis from the methods section with the reported outcomes in each included study.
Results
Study Selection
A total of 3262 studies were identified across databases. After duplicate removal, we screened titles and abstracts of 1926 studies. A total of 223 full-text articles were assessed for eligibility. Of these, 110 met the inclusion criteria common to pediatric and neonatal populations, and 79 provided relevant data for this pediatric review (Fig 1). An overview of the excluded full-text articles is provided in the Supplemental Table 9.
Study Characteristics
Included studies were all published between 2006 and 2021 (Table 1). Eighteen studies contained a control group, and 9 of these used random allocation. Sixteen studies had a time series intervention group design, and the remaining 45 had a pre- and postintervention group design. Several studies reported outcomes at >1 Kirkpatrick level. Fifteen studies reported patient outcome (level 4), 37 reported behavior outcome (level 3), and 47 reported learning outcome (level 2) (Table 1).
Risk of Bias
The risk-of-bias judgments of the 9 randomized studies are presented in Table 2. Five studies received an overall judgment of some concern,13–17 and 4 received a high-risk judgment.18–21 Only 2 studies were judged as low risk for the crucial randomization domain.15,17 All studies had low risk of bias in the intervention domain, reflecting no deviations in the allocated intervention groups and intention-to-treat analyses.
Risk-of-Bias Judgment for Included Randomized Studies Using the Revised RoB 2
. | . | . | Subdomain Judgment of Risk-of-Bias . | Overall Judgment of Risk of Bias . | |||||
---|---|---|---|---|---|---|---|---|---|
Source . | Year . | Design . | Domain 1 Randomization . | Domain 1b Recruitment . | Domain 2 Intervention . | Domain 3 Missing Outcome . | Domain 4 Measuring Outcome . | Domain 5 Selected Results . | |
Besbes et al17 | 2021 | Randomized, 2 arms | Low risk | NA | Low risk | Low risk | Some concern | Some concern | Some concern |
Bragard et al13 | 2019 | Cluster randomized | Some concern | Low risk | Low risk | Low risk | Low risk | Some concern | Some concern |
Mariani et al15 | 2019 | Randomized, 2 arms | Low risk | NA | Low risk | Low risk | Some concern | Some concern | Some concern |
Fagan et al19 | 2018 | Randomized, 2 arms | High risk | NA | Low risk | Low risk | High risk | Some concern | High risk |
Stellflug and Lowe21 | 2018 | Cluster randomized | High risk | Some concern | Low risk | Some concern | Low risk | Some concern | High risk |
Lemke et al20 | 2019 | Cluster randomized | Some concern | Some concern | Low risk | High risk | Low risk | Some concern | High risk |
Bultas et al18 | 2014 | Randomized, 2 arms | Some concern | NA | Low risk | Some concern | Some concern | Some concern | High risk |
Kurosawa et al14 | 2014 | Randomized, 2 arms | Some concern | NA | Low risk | Low risk | Low risk | Some concern | Some concern |
Sudikoff et al16 | 2009 | Randomized, 2 arms | Some concern | NA | Low risk | Low risk | Low risk | Some concern | Some concern |
. | . | . | Subdomain Judgment of Risk-of-Bias . | Overall Judgment of Risk of Bias . | |||||
---|---|---|---|---|---|---|---|---|---|
Source . | Year . | Design . | Domain 1 Randomization . | Domain 1b Recruitment . | Domain 2 Intervention . | Domain 3 Missing Outcome . | Domain 4 Measuring Outcome . | Domain 5 Selected Results . | |
Besbes et al17 | 2021 | Randomized, 2 arms | Low risk | NA | Low risk | Low risk | Some concern | Some concern | Some concern |
Bragard et al13 | 2019 | Cluster randomized | Some concern | Low risk | Low risk | Low risk | Low risk | Some concern | Some concern |
Mariani et al15 | 2019 | Randomized, 2 arms | Low risk | NA | Low risk | Low risk | Some concern | Some concern | Some concern |
Fagan et al19 | 2018 | Randomized, 2 arms | High risk | NA | Low risk | Low risk | High risk | Some concern | High risk |
Stellflug and Lowe21 | 2018 | Cluster randomized | High risk | Some concern | Low risk | Some concern | Low risk | Some concern | High risk |
Lemke et al20 | 2019 | Cluster randomized | Some concern | Some concern | Low risk | High risk | Low risk | Some concern | High risk |
Bultas et al18 | 2014 | Randomized, 2 arms | Some concern | NA | Low risk | Some concern | Some concern | Some concern | High risk |
Kurosawa et al14 | 2014 | Randomized, 2 arms | Some concern | NA | Low risk | Low risk | Low risk | Some concern | Some concern |
Sudikoff et al16 | 2009 | Randomized, 2 arms | Some concern | NA | Low risk | Low risk | Low risk | Some concern | Some concern |
NA, not applicable.
The risk-of-bias scores for the 15 nonrandomized studies with clinical performance or patient outcomes (level 4) regardless of study type and the 1 nonrandomized study with a control group reporting team performance outcomes (level 3) are presented in Table 3. Two cohort studies received an overall score of 4 out of 6,22,23 and 8 studies received a score of ≤2, including 1 cohort study with low study retention.24
Risk of Bias for Included Nonrandomized Studies on the Basis of the NOS With Adaption for Educational Research
. | . | . | NOS Subdomain Risk-of-Bias Score . | Overall Assessment Score (0–6) . | ||||
---|---|---|---|---|---|---|---|---|
Source . | Year . | Design . | Intervention Group Representative . | Comparison Group Selection . | Comparison Group Comparability . | Study Retention . | Outcome Assessment . | |
Shah et al22 | 2016 | Cohort | 1 | 1 | 0 | 1 | 1 | 4 |
Qian et al29 | 2016 | Cohort | 0 | 1 | 0 | 1 | 1 | 3 |
Nishisaki et al35 | 2011 | Cohort | 1 | 1 | 0 | 1 | 1 | 3 |
Nishisaki et al23 | 2010 | Cohort | 1 | 1 | 0 | 1 | 1 | 4 |
Gilfoyle et al24 | 2007 | Cohort | 0 | 1 | 0 | 0 | 1 | 2 |
Diaz and Dawson36 | 2020 | Time series | 1 | 0 | 0 | 1 | 0 | 2 |
Hazwani et al34,43 | 2020 | Time series | 0 | 0 | 0 | 0 | 0 | 0 |
Colman et al37,44 | 2019 | Time series | 0 | 0 | 0 | 0 | 0 | 0 |
Sawyer et al25 | 2019 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Barni et al30 | 2019 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Di Nardo et al26 | 2018 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Su et al27 | 2014 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Theilen et al31 | 2013 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Andreatta et al33 | 2011 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Martin et al32 | 2016 | Pre/post | 1 | 0 | 0 | 1 | 1 | 3 |
Burton et al28 | 2011 | Pre/post | 0 | 0 | 0 | 1 | 1 | 2 |
. | . | . | NOS Subdomain Risk-of-Bias Score . | Overall Assessment Score (0–6) . | ||||
---|---|---|---|---|---|---|---|---|
Source . | Year . | Design . | Intervention Group Representative . | Comparison Group Selection . | Comparison Group Comparability . | Study Retention . | Outcome Assessment . | |
Shah et al22 | 2016 | Cohort | 1 | 1 | 0 | 1 | 1 | 4 |
Qian et al29 | 2016 | Cohort | 0 | 1 | 0 | 1 | 1 | 3 |
Nishisaki et al35 | 2011 | Cohort | 1 | 1 | 0 | 1 | 1 | 3 |
Nishisaki et al23 | 2010 | Cohort | 1 | 1 | 0 | 1 | 1 | 4 |
Gilfoyle et al24 | 2007 | Cohort | 0 | 1 | 0 | 0 | 1 | 2 |
Diaz and Dawson36 | 2020 | Time series | 1 | 0 | 0 | 1 | 0 | 2 |
Hazwani et al34,43 | 2020 | Time series | 0 | 0 | 0 | 0 | 0 | 0 |
Colman et al37,44 | 2019 | Time series | 0 | 0 | 0 | 0 | 0 | 0 |
Sawyer et al25 | 2019 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Barni et al30 | 2019 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Di Nardo et al26 | 2018 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Su et al27 | 2014 | Time series | 0 | 0 | 0 | 1 | 1 | 2 |
Theilen et al31 | 2013 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Andreatta et al33 | 2011 | Time series | 1 | 0 | 0 | 1 | 1 | 3 |
Martin et al32 | 2016 | Pre/post | 1 | 0 | 0 | 1 | 1 | 3 |
Burton et al28 | 2011 | Pre/post | 0 | 0 | 0 | 1 | 1 | 2 |
Pre/post, pre- and postintervention study.
Results Related to Clinical Performance or Patient Outcomes
We included 15 studies that reported change in clinical performance of health care professionals or change in clinical outcomes (Table 4). Only 4 of the studies had a control group, but none of these used random allocation. Four of the studies simulated extracorporeal membrane oxygenation (ECMO),25–28 1 septic shock,29 1 anaphylaxis,30 1 seizure,22 2 deteriorating patients,31,32 2 cardiopulmonary arrest,33,34 2 airway management,23,35 and 2 closed-loop communication and teamwork skills.36,37
Summary of 15 Studies With Change in Clinical Performance or Patient Outcome (Kirkpatrick Level 4)
Source, Year . | Setting . | Design(No. of Participants) . | Intervention . | Comparator . | Participants . | Case Keywords . | Outcomes . |
---|---|---|---|---|---|---|---|
Qian et al, 201629 | 3 tertiary hospitals | Cohort (NA) | High-fidelity simulation every 3 mo at 1 intervention hospital Evaluation after 1 y | 2 control hospital straining as usual | Chief resident, residents, nurses (n = 5) | Septic shock, first-hour basic tasks | Total compliance with first-hour critical tasks: intervention hospital, 61.7%; control hospitals, 23.1% (P = .004) |
Shah et al, 201622 | Prehospital | Cohort (250) | Transports of patients having seizures Intervention training: 9-h course, didactic, hands-on, and 4 pediatric simulations | Training as usual | Paramedics, 250 transports with at least 1 paramedic who attended the course | Seizure | The course enhanced seizure protocol compliance prehospital, 250 transports of patients having seizuresCheck blood glucose: intervention, 72%; control, 66% (P = .35)Midazolam if euglycemic: intervention, 63%; control, 56% (P = .28) |
Nishisaki et al, 201135 | Children’s hospital, tertiary PICU | Cohort (265) | 202 high-fidelity simulations from June 2007 to August 2008: 30-min hands-on, simulation, and debriefingActual intubations (n = 15) were observed in daytime | Same hospital Nontrained residents | Pediatric residents, EM residents, nurses, respiratory therapists | Clinical performance and team behavior during airway management, including tracheal intubation | The presence of ≥2 simulation-trained providers in real events (intubation) were associated with improved observed performance (P = .012) |
Nishisaki et al, 201023 | Children’s hospital, tertiary PICU | Cohort (265) | 202 high-fidelity simulations from June 2007 to August 2008: 30-min hands-on, simulation, and debriefing 401 orotracheal intubations were evaluated: 220 preintervention and 181 during intervention | Same hospital Nontrained residents Period before intervention | Pediatric residents, EM residents, nurses, respiratory therapists | Clinical performance and team behavior during airway management, including tracheal intubation | No differences in first-attempt success or overall success compared with nontrained residents or presimulation period |
Diaz and Dawson, 202036 | Children’s hospital | Time series (112) | 70 simulations in 13 wkFocus was on closed-loop communication after watching 3 videos of simulated resuscitation scenariosStaff participated in 2 scenarios 1 mo apart | Same hospital Period 4 mo before and after simulation Retrospective chart reviews | Pediatric attending physicians, fellows, assistants, nurses, therapists (n = 3–6) | Closed-loop communication | Retrospective chart review of Emergency Severity Index patient category 1 Significant reduction in medication errors (P = .017) after intervention Staff perception of closed-loop communication ability improved and were sustained after 1 mo |
Hazwani et al, 202034 | Children’s hospital | Time series (NA) | Mock code simulation program Random time end place 2–3 times per month | Same hospital Period before mock code simulation program (June 2015–June 2016) | Pediatric cardiac arrest team (n = 6) | Cardiopulmonary arrest | Review of all records of pediatric cardiopulmonary arrest occurring in study periods Significant reduction in CPR initiation time (P = .031) Improvement of compliance with guidelines: initiation of basic life support (P = .019) and electrical therapy for shockable rhythm (P = 00007)No significant change in survival rate or mortality |
Colman et al, 201937 | Pediatric critical care unit (PICU), children’s hospital | Time series (165) | Simulation-based team training where team performance was the primary learning objective3-mo intervention period 30 mandatory team training workshops, each 3 h and including 3 scenarios | Same hospital Data collected on real-time events in the pre- (9 mo) and post intervention (9 mo) period | Multidisciplinary teams (n = 5–6) | Team leader identification, role assignment, direct and closed-loop communication, situational awareness, shared mental model and global assessment | Assessment of 15 teamwork categories immediately following a real medical crisis in the postintervention period Significant improvement in 11 of 15 composite teamwork skills (P < .05) |
Sawyer et al, 201925 | Children’s hospital in 3 ICUs (PICU, CICU, NICU) | Time series (332) | 29 simulations performed between February 2014 and October 2016 Each simulation 2-h high-fidelity in situ training with 30-min introduction, 30 min of simulation, and 60 min of postevent debriefing | Same hospital Period before simulation (December 2012–January 2014) | Multidisciplinary, ICU/ECPR providers (n = 11 ± 3) | ECMO, ECPR | Retrospective chart view of hospital code blue records Clinical parameters: 1, compliance with ECPR activation protocol (pre, 83%; post, 95%; P = .02); 2, ECPR activation time (pre, 7 min; post, 2 min; P < .01); 3, ECPR response time, deployment time, and cannulation time did not change significantly Clinical outcomes: 1, ECLS 24-h survival (pre, 45%; post, 56%; P = .56); 2, survival to discharge (pre, 27%; post, 38%; P = .38) |
Barni et al, 201930 | Pediatric emergency department | Time series (30) | 1 intervention hospital with simulation training(2011–2016)4-h high-fidelity simulation | Same hospital Period before simulation (2004–2010) | Physicians, nurses (n = 6) | Anaphylaxis, use of epinephrine according to guideline | Increase in use of epinephrine from 2.4% to 10% of patients (P < .05) Increase in number of patients with allergy workup (36%–51%; P = .011) |
Di Nardo et al, 201826 | Children’s hospital, PICU | Time series (NA) | High-fidelity simulation program (December 2013–December 2016) Training every 3 mo | Same hospitalPeriod before simulation (2011–November 2013) | PICU personnel: nurses, physicians, perfusionists | ECMO, time to manage most common ECMO emergencies | Median time to oxygenator change (pre, 5.30 min; post, 3.90 min; P = .02)Median time to manage air embolism (pre, 22 min; post, 15 min; P = .048)ECMO survival rate (pre, 50%; post, 57%; P = .72) |
Su et al, 201427 | Children’s hospital | Time series (NA) | 8 high-fidelity simulations over 36 mo (April 2010–March 2013) | Same hospital Period before simulation (February 2009–March 2010) | Multidisciplinary, ICU/ECPR providers | ECMO, ECPR | Retrospective chart view ECMO deployment time, ie, time from the call to the start of ECMO flow (pre, 51 min; post, 40 min; P = .018) |
Theilen et al, 201331 | Children’s hospital | Time series (NA) | 2-h training weekly Each member 4–10 times/year Concurrent with introduction of pMET | Same hospital Period before pMET and team training | Physicians, nurses (pMET members) | Deteriorating patients, pediatric trauma, postoperative complications | Time from warning sign to first response (pre, 4 h; post, 1.5 h; P < .001) Increased nursing observation (P = .03) and consultant review (P = .004)Time to PICU admission (pre, 10.5 h; post, 5 h; P = .024) Overall hospital mortality rate reduced in the same period (P < .001) |
Andreatta et al, 201133 | Children’s hospital | Time series (228) | Randomly called mock codes at least monthly with video recording for immediate debriefing (2005–2009) | Same hospital Change over time | Junior and senior residents | Cardiopulmonary arrest due to sepsis, respiratory distress, increased intracranial pressure, anaphylactic shock, cardiogenic shock | Survival rates (who survived cardiopulmonary arrest and were subsequently discharged) increased from 33% to ∼50% in same period of 4 y |
Martin et al, 201632 | Children’s hospital | Pre/post (83) | 2-h cues of deterioration simulation program (aim = evaluating nurses’ early recognition of deterioration) (all)1-h multidisciplinary mock code (optional educational offer) over 1 y | None | Nurses | Deteriorating patients,respiratory distress, cardiac arrest, seizures, supraventricular tachycardia | No significant change in number of PICU transfers by the number of mock simulations No significant change in accuracy of PEWS score by the number of mock simulations |
Burton et al, 201128 | Children’s hospital | Pre/post (19) | 24 sessions of high-fidelity simulations4-h duration (May 2008–August 2009) | None | ECMO nurses and respiratory therapists | ECMO | Based on 26 cannulations during study period1: time from blood available to circuit ready did not improve2: initiation checklist (24 items) compliance improved from 17.14 at pretraining baseline to 23.23 (SD, 1.61) during training (P < .0001) |
Source, Year . | Setting . | Design(No. of Participants) . | Intervention . | Comparator . | Participants . | Case Keywords . | Outcomes . |
---|---|---|---|---|---|---|---|
Qian et al, 201629 | 3 tertiary hospitals | Cohort (NA) | High-fidelity simulation every 3 mo at 1 intervention hospital Evaluation after 1 y | 2 control hospital straining as usual | Chief resident, residents, nurses (n = 5) | Septic shock, first-hour basic tasks | Total compliance with first-hour critical tasks: intervention hospital, 61.7%; control hospitals, 23.1% (P = .004) |
Shah et al, 201622 | Prehospital | Cohort (250) | Transports of patients having seizures Intervention training: 9-h course, didactic, hands-on, and 4 pediatric simulations | Training as usual | Paramedics, 250 transports with at least 1 paramedic who attended the course | Seizure | The course enhanced seizure protocol compliance prehospital, 250 transports of patients having seizuresCheck blood glucose: intervention, 72%; control, 66% (P = .35)Midazolam if euglycemic: intervention, 63%; control, 56% (P = .28) |
Nishisaki et al, 201135 | Children’s hospital, tertiary PICU | Cohort (265) | 202 high-fidelity simulations from June 2007 to August 2008: 30-min hands-on, simulation, and debriefingActual intubations (n = 15) were observed in daytime | Same hospital Nontrained residents | Pediatric residents, EM residents, nurses, respiratory therapists | Clinical performance and team behavior during airway management, including tracheal intubation | The presence of ≥2 simulation-trained providers in real events (intubation) were associated with improved observed performance (P = .012) |
Nishisaki et al, 201023 | Children’s hospital, tertiary PICU | Cohort (265) | 202 high-fidelity simulations from June 2007 to August 2008: 30-min hands-on, simulation, and debriefing 401 orotracheal intubations were evaluated: 220 preintervention and 181 during intervention | Same hospital Nontrained residents Period before intervention | Pediatric residents, EM residents, nurses, respiratory therapists | Clinical performance and team behavior during airway management, including tracheal intubation | No differences in first-attempt success or overall success compared with nontrained residents or presimulation period |
Diaz and Dawson, 202036 | Children’s hospital | Time series (112) | 70 simulations in 13 wkFocus was on closed-loop communication after watching 3 videos of simulated resuscitation scenariosStaff participated in 2 scenarios 1 mo apart | Same hospital Period 4 mo before and after simulation Retrospective chart reviews | Pediatric attending physicians, fellows, assistants, nurses, therapists (n = 3–6) | Closed-loop communication | Retrospective chart review of Emergency Severity Index patient category 1 Significant reduction in medication errors (P = .017) after intervention Staff perception of closed-loop communication ability improved and were sustained after 1 mo |
Hazwani et al, 202034 | Children’s hospital | Time series (NA) | Mock code simulation program Random time end place 2–3 times per month | Same hospital Period before mock code simulation program (June 2015–June 2016) | Pediatric cardiac arrest team (n = 6) | Cardiopulmonary arrest | Review of all records of pediatric cardiopulmonary arrest occurring in study periods Significant reduction in CPR initiation time (P = .031) Improvement of compliance with guidelines: initiation of basic life support (P = .019) and electrical therapy for shockable rhythm (P = 00007)No significant change in survival rate or mortality |
Colman et al, 201937 | Pediatric critical care unit (PICU), children’s hospital | Time series (165) | Simulation-based team training where team performance was the primary learning objective3-mo intervention period 30 mandatory team training workshops, each 3 h and including 3 scenarios | Same hospital Data collected on real-time events in the pre- (9 mo) and post intervention (9 mo) period | Multidisciplinary teams (n = 5–6) | Team leader identification, role assignment, direct and closed-loop communication, situational awareness, shared mental model and global assessment | Assessment of 15 teamwork categories immediately following a real medical crisis in the postintervention period Significant improvement in 11 of 15 composite teamwork skills (P < .05) |
Sawyer et al, 201925 | Children’s hospital in 3 ICUs (PICU, CICU, NICU) | Time series (332) | 29 simulations performed between February 2014 and October 2016 Each simulation 2-h high-fidelity in situ training with 30-min introduction, 30 min of simulation, and 60 min of postevent debriefing | Same hospital Period before simulation (December 2012–January 2014) | Multidisciplinary, ICU/ECPR providers (n = 11 ± 3) | ECMO, ECPR | Retrospective chart view of hospital code blue records Clinical parameters: 1, compliance with ECPR activation protocol (pre, 83%; post, 95%; P = .02); 2, ECPR activation time (pre, 7 min; post, 2 min; P < .01); 3, ECPR response time, deployment time, and cannulation time did not change significantly Clinical outcomes: 1, ECLS 24-h survival (pre, 45%; post, 56%; P = .56); 2, survival to discharge (pre, 27%; post, 38%; P = .38) |
Barni et al, 201930 | Pediatric emergency department | Time series (30) | 1 intervention hospital with simulation training(2011–2016)4-h high-fidelity simulation | Same hospital Period before simulation (2004–2010) | Physicians, nurses (n = 6) | Anaphylaxis, use of epinephrine according to guideline | Increase in use of epinephrine from 2.4% to 10% of patients (P < .05) Increase in number of patients with allergy workup (36%–51%; P = .011) |
Di Nardo et al, 201826 | Children’s hospital, PICU | Time series (NA) | High-fidelity simulation program (December 2013–December 2016) Training every 3 mo | Same hospitalPeriod before simulation (2011–November 2013) | PICU personnel: nurses, physicians, perfusionists | ECMO, time to manage most common ECMO emergencies | Median time to oxygenator change (pre, 5.30 min; post, 3.90 min; P = .02)Median time to manage air embolism (pre, 22 min; post, 15 min; P = .048)ECMO survival rate (pre, 50%; post, 57%; P = .72) |
Su et al, 201427 | Children’s hospital | Time series (NA) | 8 high-fidelity simulations over 36 mo (April 2010–March 2013) | Same hospital Period before simulation (February 2009–March 2010) | Multidisciplinary, ICU/ECPR providers | ECMO, ECPR | Retrospective chart view ECMO deployment time, ie, time from the call to the start of ECMO flow (pre, 51 min; post, 40 min; P = .018) |
Theilen et al, 201331 | Children’s hospital | Time series (NA) | 2-h training weekly Each member 4–10 times/year Concurrent with introduction of pMET | Same hospital Period before pMET and team training | Physicians, nurses (pMET members) | Deteriorating patients, pediatric trauma, postoperative complications | Time from warning sign to first response (pre, 4 h; post, 1.5 h; P < .001) Increased nursing observation (P = .03) and consultant review (P = .004)Time to PICU admission (pre, 10.5 h; post, 5 h; P = .024) Overall hospital mortality rate reduced in the same period (P < .001) |
Andreatta et al, 201133 | Children’s hospital | Time series (228) | Randomly called mock codes at least monthly with video recording for immediate debriefing (2005–2009) | Same hospital Change over time | Junior and senior residents | Cardiopulmonary arrest due to sepsis, respiratory distress, increased intracranial pressure, anaphylactic shock, cardiogenic shock | Survival rates (who survived cardiopulmonary arrest and were subsequently discharged) increased from 33% to ∼50% in same period of 4 y |
Martin et al, 201632 | Children’s hospital | Pre/post (83) | 2-h cues of deterioration simulation program (aim = evaluating nurses’ early recognition of deterioration) (all)1-h multidisciplinary mock code (optional educational offer) over 1 y | None | Nurses | Deteriorating patients,respiratory distress, cardiac arrest, seizures, supraventricular tachycardia | No significant change in number of PICU transfers by the number of mock simulations No significant change in accuracy of PEWS score by the number of mock simulations |
Burton et al, 201128 | Children’s hospital | Pre/post (19) | 24 sessions of high-fidelity simulations4-h duration (May 2008–August 2009) | None | ECMO nurses and respiratory therapists | ECMO | Based on 26 cannulations during study period1: time from blood available to circuit ready did not improve2: initiation checklist (24 items) compliance improved from 17.14 at pretraining baseline to 23.23 (SD, 1.61) during training (P < .0001) |
CICU, cardiac ICU; ECLS, extracorporeal life support; ECPR, extracorporeal cardiopulmonary resuscitation; EM, emergency medicine; NA, not available (information not provided in the original article); pMET, pediatric medical emergency team; PEWS, pediatric early warning sign; pre/post, pre- and postintervention study.
Survival and Mortality Outcome
Four studies reported survival or mortality (Table 4). Sawyer et al25 performed simulation-based team training in 3 ICUs in a children’s hospital and compared survival after extracorporeal cardiopulmonary resuscitation (CPR) before and during the simulation program. The authors found a 24-hour survival rate of 56% in the period of simulation compared with 45% in the period before simulation. They also found a survival to discharge rate of 38% compared with 27% before simulation. Di Nardo et al26 compared survival before and after an ECMO simulation program at a children’s hospital. They reported a 50% survival rate before intervention compared with 57% after intervention. Theilen et al31 explored an intervention with 2 hours of weekly simulation and introduction of a pediatric medical emergency team where each team member had simulation training 4 to 10 times per year. They compared the period before and after introduction of the team and simulation training. They found that overall mortality was significantly reduced. Finally, Andreatta et al33 performed unannounced simulation mock codes at a minimum of once every month. They reported cardiopulmonary arrest survival rates over a 4-year period, including at the introduction of the simulation program. Survival rates increased from 33% before to ∼50% after simulation was introduced, and the reduction correlated with the number of mock codes.
Time to Critical Task Outcomes
Six studies reported time to a specific clinical event as outcome (Table 4). Four of these studies investigated time to manage ECMO emergencies (eg, pump failure, circuit failure) and time to initiate ECMO (eg, activation, cannulation, ECMO call to flow initiation). Authors of 3 of the 4 studies reported clinically and statistically significant reductions in time to important ECMO tasks after initiating simulation-based team training in their own hospital compared with before.25–27 However, 1 study investigated time from blood available to circuit ready and did not find a significant change.28
Theilen et al31 introduced a pediatric medical emergency team at their hospital where each team member had simulation training 4 to 10 times per year. Compared with a period before introduction of the team and simulation training, they observed a reduction from 4 hours to 1.5 hours from warning sign to first response and a reduction from 10.5 hours to 5 hours in time to pediatric ICU admission. Hazwani et al34 initiated mock code simulations and reported reduced CPR initiation time compared with the period before in the same hospital.
Protocol Adherence Outcomes
Seven studies reported adherence to guidelines or compliance to checklists or tasks (Table 4). Qian et al29 conducted a cohort study that initiated simulation-based training at 1 hospital, using 2 other hospitals as control. They investigated compliance with first-hour critical tasks in septic shock and reported a significantly higher compliance rate of 62% at the intervention hospital compared with 23% at the control hospitals. Barni et al30 investigated the use of epinephrine for anaphylaxis according to guidelines in a pediatric emergency department. Compared with before the simulation intervention, they observed a significant increase in the use of epinephrine from 2.4% to 10%. Shah et al22 found that a simulation-based course slightly improved adherence to prehospital seizure protocol, ie, blood glucose measurement and midazolam administration. Authors of 2 studies investigating ECMO reported improvement in compliance with an activation protocol25 and an initiation checklist28 after starting simulation-based training. Hazwani et al34 found improved compliance with guidelines in the treatment of cardiopulmonary arrest after a mock code simulation program, and Diaz and Dawson36 found a reduction in medication errors after simulation that focused on closed-loop communication.
Airway Management Outcomes
Two studies reported airway management outcomes (Table 4). Nishisaki et al23 investigated the effect of simulation-based intubation training and skill refresher training in the beginning of an on-call period. They found no difference in first-attempt and overall success of intubation between residents receiving refresher training or not on the basis of real pediatric intubations. Nishisaki et al35 later investigated effects of simulation training on clinical performance and team behavior during intubations as evaluated by a trained observer in a PICU. The team performance was significantly higher in teams with ≥2 simulation-trained members (both technical and behavioral scores) compared with teams with <2 trained members.
Team Performance and Communication Outcomes
Results Related to Team Performance Outcomes
We included 37 studies with observed team performance outcomes. Twenty-nine of these had no control group and are not further described here (Table 1). Below, we describe the 8 studies with a control group, 7 of which used random allocation (Table 5).
Summary of Eight Controlled Studies With Team Performance Outcomes (Kirkpatrick Level 3)
Source, Year . | Setting . | Design(No. of Participants) . | Intervention . | Comparator . | Participants . | RetestTiming . | CaseKeywords . | Outcomes . |
---|---|---|---|---|---|---|---|---|
Bragard et al, 201913 | Pediatric department, university hospital | Randomized(16) | 5 high-fidelity simulation sessions with debriefingVideo recorded at sessions 1 and 5 for comparison | No training program, 2 simulations sessions without debriefing, video recorded for evaluation | Residents (pediatric and emergency), nurses | — | 5 scenarios on arrhythmia due to hypovolemia, drug intoxication, hyperkalemia, drowning, and hypoxia | Blinded evaluation of video recordingsEffect on leadership and communication skills (P = .038)Effect on knowledge and clinical skills (P = .026)No significant change in time to initiate CPR |
Mariani et al, 201915 | 5-hospital health system | Randomized(18) | 2 high-fidelity simulation sessions of 1 h, 5 mo apart | Clinical duties as usual | Pediatric nurses | 9 mo | Hypovolemic shock, respiratory distress, seizure | No difference in skill competency during scenarios comparing intervention and control groups |
Stellflug and Lowe, 201821 | Regional PALS course | Randomized(94) | Integration of high-fidelity simulation into PALS course (2 d) | PALS (2 d) + low-fidelity simulation | Nurses, physicians, respiratory therapists, paramedics | 6 mo(written test) | Cases from PALS course | Better retention after high fidelity vs low fidelity (P = .05)Time to critical events (immediate postcourse, intervention vs control):Recognition (100 s vs 112 s)Intervention (141 s vs 159 s)Reassessment (154 s vs 187s) |
Lemke et al, 201920 | Hospital’s simulationcenter | Randomized(30) | Rapid deliberate practicePre- and posttest were video recorded and scored for comparison | Traditional debriefing | PEM fellows, nurses, respiratory therapists | 3 mo | Pediatric resuscitation cases | Improvement in the human factor subscore (eg, establishing roles in the team, communication, knowledge sharing) for the intervention group (P = .013)A trend toward improvement in airway/breathing and circulation scores, but not significant |
Bultas et al, 201418 | Pediatric hospital | Randomized(33) | High-fidelity simulation (comparing teaching strategy for recognizing deteriorating pediatric patient) | Simulation with traditional static mannequins | Pediatric staff | 6 mo | Respiratory failure, cardiogenic shock | Significant change in teamwork confidence score (P = .001)Significant change in skills and tasks performed during respiratory scenario (P = .012) |
Kurosawa et al, 201414 | Children’s hospital, tertiary PICU | Randomized(40) | Reconstructed PALS with 6 in situ scenarios of 30 min in 6 mo for each participant in the intervention group | Standard PALS training | Nurses, respiratory therapists | 6 mo | 12 core PALS scenarios | Skill performance (by clinical performance tool) significantly improved (P = .004), and behavioral performance (improvement in teamwork) significantly improved in each group after training, but no significant difference between intervention and control groups |
Sudikoff et al, 200916 | University hospital | Randomized(16) | High-fidelity simulation trainingIntervention group: 1-d training, 25% didactic, 15% hands on, 60% simulationCross-over study | Standard resident curriculum | Pediatric residents, postgraduate year 2 | 2 and 4 mo | Respiratory failure and airway management | Mean global competency score improved, and improvement persisted for up to 4 mo (P < .05)Critical actions showed a trend toward but did not reach statistical significanceReduction in number of harmful actions |
Gilfoyle et al, 200724 | Children’s hospital | Cohort(15) | Plenary session + 2 simulated scenariosTeam performance evaluated by checklist | Nontrained residents | Pediatric residents | 6 mo | Resuscitation team leadership | Significant increase in checklist scores (leadership, communication, etc) between the 2 scenarios on day 1 (P < .05)Significant increase in checklist scores after 6 mo in the intervention group (P < .001) vs nontrained residents (same scenario as from day 1) |
Source, Year . | Setting . | Design(No. of Participants) . | Intervention . | Comparator . | Participants . | RetestTiming . | CaseKeywords . | Outcomes . |
---|---|---|---|---|---|---|---|---|
Bragard et al, 201913 | Pediatric department, university hospital | Randomized(16) | 5 high-fidelity simulation sessions with debriefingVideo recorded at sessions 1 and 5 for comparison | No training program, 2 simulations sessions without debriefing, video recorded for evaluation | Residents (pediatric and emergency), nurses | — | 5 scenarios on arrhythmia due to hypovolemia, drug intoxication, hyperkalemia, drowning, and hypoxia | Blinded evaluation of video recordingsEffect on leadership and communication skills (P = .038)Effect on knowledge and clinical skills (P = .026)No significant change in time to initiate CPR |
Mariani et al, 201915 | 5-hospital health system | Randomized(18) | 2 high-fidelity simulation sessions of 1 h, 5 mo apart | Clinical duties as usual | Pediatric nurses | 9 mo | Hypovolemic shock, respiratory distress, seizure | No difference in skill competency during scenarios comparing intervention and control groups |
Stellflug and Lowe, 201821 | Regional PALS course | Randomized(94) | Integration of high-fidelity simulation into PALS course (2 d) | PALS (2 d) + low-fidelity simulation | Nurses, physicians, respiratory therapists, paramedics | 6 mo(written test) | Cases from PALS course | Better retention after high fidelity vs low fidelity (P = .05)Time to critical events (immediate postcourse, intervention vs control):Recognition (100 s vs 112 s)Intervention (141 s vs 159 s)Reassessment (154 s vs 187s) |
Lemke et al, 201920 | Hospital’s simulationcenter | Randomized(30) | Rapid deliberate practicePre- and posttest were video recorded and scored for comparison | Traditional debriefing | PEM fellows, nurses, respiratory therapists | 3 mo | Pediatric resuscitation cases | Improvement in the human factor subscore (eg, establishing roles in the team, communication, knowledge sharing) for the intervention group (P = .013)A trend toward improvement in airway/breathing and circulation scores, but not significant |
Bultas et al, 201418 | Pediatric hospital | Randomized(33) | High-fidelity simulation (comparing teaching strategy for recognizing deteriorating pediatric patient) | Simulation with traditional static mannequins | Pediatric staff | 6 mo | Respiratory failure, cardiogenic shock | Significant change in teamwork confidence score (P = .001)Significant change in skills and tasks performed during respiratory scenario (P = .012) |
Kurosawa et al, 201414 | Children’s hospital, tertiary PICU | Randomized(40) | Reconstructed PALS with 6 in situ scenarios of 30 min in 6 mo for each participant in the intervention group | Standard PALS training | Nurses, respiratory therapists | 6 mo | 12 core PALS scenarios | Skill performance (by clinical performance tool) significantly improved (P = .004), and behavioral performance (improvement in teamwork) significantly improved in each group after training, but no significant difference between intervention and control groups |
Sudikoff et al, 200916 | University hospital | Randomized(16) | High-fidelity simulation trainingIntervention group: 1-d training, 25% didactic, 15% hands on, 60% simulationCross-over study | Standard resident curriculum | Pediatric residents, postgraduate year 2 | 2 and 4 mo | Respiratory failure and airway management | Mean global competency score improved, and improvement persisted for up to 4 mo (P < .05)Critical actions showed a trend toward but did not reach statistical significanceReduction in number of harmful actions |
Gilfoyle et al, 200724 | Children’s hospital | Cohort(15) | Plenary session + 2 simulated scenariosTeam performance evaluated by checklist | Nontrained residents | Pediatric residents | 6 mo | Resuscitation team leadership | Significant increase in checklist scores (leadership, communication, etc) between the 2 scenarios on day 1 (P < .05)Significant increase in checklist scores after 6 mo in the intervention group (P < .001) vs nontrained residents (same scenario as from day 1) |
—, data not available; PALS, pediatric advanced life support; PEM, pediatric emergency medicine.
Retention of Skills
Seven of the 8 studies evaluated retention of technical and nontechnical skills using simulation-based retesting of teams. Authors of 6 of these 7 studies retested after 2 to 6 months and found significant improvements in intervention groups compared with control groups.14,16,18,20,21,24 Mariani et al15 performed a retest after 9 months and found no difference in skill competency between the groups. Stellflug and Lowe21 investigated random allocation to either high-fidelity or low-fidelity team training and reported better performance on predefined skills for the high-fidelity group when retested after 6 months. Lemke et al20 investigated random allocation to either rapid-cycle deliberate practice or traditional team training with debriefing and reported improvement in the human factor subscore (eg, team organization, communication, knowledge sharing) for the intervention group after 3 months. Bultas et al18 investigated a teaching strategy for recognizing a deteriorating pediatric patient, with random allocation to either high-fidelity simulation or traditional training, and reported significantly better teamwork and skills and tasks scores during a respiratory scenario in the intervention group at 6-month retest. Kurosawa et al14 performed a randomized trial of a reconstructed recertification of pediatric advanced life support with simulation once a month compared with standard recertification. They found significantly improved skills performance when retesting after 6 months compared with the control group. Sudikoff et al16 conducted a randomized cross-over study of emergency airway management and teamwork. They found improved global competency scores with regard to communication, leadership, and team skills, and the improvement persisted for up to 4 months. Gilfoyle et al24 conducted a workshop with simulation-based training and investigated team leadership skills in pediatric advanced resuscitation. They found a significant increase in checklist scores after 6 months in the intervention group compared with nontrained residents.
Team Performance Outcomes
Results Related to Learning Outcomes
We included 47 studies with self-reported knowledge or confidence. Thirty-eight of these had no control group and are not further described here (Table 1). Five of the studies had a control group with random allocation. Bragard et al13 performed a team training program on cardiac arrhythmias and found reduced perceived stress and improved satisfaction with communication, general skills, and communication skills compared with the control group. Stellflug and Lowe21 investigated random allocation to either high-fidelity or low-fidelity team training and found that the low-fidelity group declined more in ratings of their confidence levels on performing 19 different skills or tasks on retest at 6 months. Fagan et al19 investigated random allocation to recertification of pediatric advanced life support with or without simulation-based team training. Members of the intervention group reported higher perception of teamwork and more situational awareness. Mariani et al15 investigated the effect of repeated pediatric mock code simulations and found no difference in self-confidence between the groups. Besbes et al17 found that knowledge test scores improved significantly for both groups, although the improvement was significantly higher in the simulation group.
Four of the studies had a control group without random allocation. McLaughlin et al38 investigated the impact of team training on perceived confidence when handling pediatric trauma patients, and at 2 years follow-up, trained professionals reported less anxiety and greater confidence compared with professionals who did not receive the simulation training. Dowson et al39 investigated the impact of team training on pediatric nurses’ self-assessed clinical confidence and found a significant improvement. Auerbach et al40 compared repetitive pediatric simulation training (scenario-debriefing-scenario) to standard simulation training and found that the intervention group reported greater improvement in knowledge and skills. Happel et al41 evaluated the perceived impact on live critical events in a period with weekly simulation-based team training. The residents in the intervention group reported positive effects on their behaviors and confidence levels as well as feeling more prepared; however, these changes did not differ significantly from the control group.
Risk of Bias Across Studies
Within each group of studies (according to Table 1), the funnel plot raised no major concern about publication bias (Fig 2). In the funnel plot, studies with and without control groups appeared similar across study size and effect ratio. We found no indication of selective reporting bias, as methods and results sections were coherent in all studies. Prepublished study protocols were unavailable on all randomized studies, which was reflected in the risk-of-bias score.
Funnel plot of 61 studies. Black circle, studies with a control group; white circle, studies with no control group and level 3 to 4 outcome; x, studies with no control group and level 2 outcome. Fourteen studies were excluded from the funnel plot because the number of participants was not specified.26,27,29,31,34,42,43,45,58,67,71,74,77,80 Four studies were excluded from the funnel plot because no effect estimate was specified.46,81,86,88
Funnel plot of 61 studies. Black circle, studies with a control group; white circle, studies with no control group and level 3 to 4 outcome; x, studies with no control group and level 2 outcome. Fourteen studies were excluded from the funnel plot because the number of participants was not specified.26,27,29,31,34,42,43,45,58,67,71,74,77,80 Four studies were excluded from the funnel plot because no effect estimate was specified.46,81,86,88
Discussion
Summary of Evidence
We performed this systematic review to describe the current state of evidence in the evolving research field of simulation-based team training in pediatrics. Summarizing effects on patient outcomes (level 4), 2 important findings deserve emphasis.
First, 4 studies indicated improved survival after introducing simulation-based team training.25,26,31,33 However, given study design limitations, future studies with higher methodological standards would enable more robust conclusions. The authors of the 4 time series studies reported survival rates before and after introduction of a simulation program. The findings were prone to confounding from uncontrolled or unknown factors. For example, Theilen et al31 introduced a pediatric medical emergency team at the same time as the simulation program, which hampers conclusions related to effects of simulation-based training per se.
Second, authors of 4 studies designed with a control group (nonrandomized) indicated effects of simulation-based team training on clinical performance, eg, improved compliance to first-hour critical tasks in sepsis treatment,29 enhanced seizure protocol adherence,22 and improved intubation performance if ≥2 simulation-trained professionals were present.23,35 Overall, these controlled studies may indicate effects on treatment quality, but ambiguity and lack of precision preclude a strong conclusion. Authors of several studies with a time series design observed improved clinical performance after the introduction of simulation programs (Table 4). However, their conclusions were prone to bias from uncontrolled or unknown factors, eg, other changes during the period of intervention.
In considering team performance outcomes (Kirkpatrick level 3) and learning outcomes (Kirkpatrick level 2), 2 important findings should be noted. First, authors of several randomized studies found improved team performance when retesting 2 to 6 months after intervention.14,16,18,20,21,24 Studies with control groups revealed that team performance, eg, leadership, communication, human factors, and teamwork confidence, improved.13,18,20,24 Second, authors of most studies with random allocation showed improved perceived confidence, skill levels, and ability to communicate in emergency situations after the introduction of simulation-based team training. Overall, the findings from these controlled studies support improved team performance, knowledge, and confidence after simulation training.
Strengths
This systematic review applied a comprehensive search strategy in 4 medical databases by an experienced medical librarian. We followed a prespecified study protocol registered in the Prospective Register of Systematic Reviews repository. Two reviewers independently screened and selected studies and performed data extraction and risk-of-bias scoring. Data were presented according to the PRISMA guidelines. The funnel plot in Fig 2 indicates no publication bias.
Limitations
Meta-analysis was impossible because of heterogeneous study designs, interventions, and outcomes in the included studies. Instead, we performed a narrative synthesis structured by hierarchical outcome, including patient and clinical performance outcomes, team performance outcomes, and learning outcomes. Although this process is less standardized than a meta-analysis, we consider it reproducible and open to scrutiny. Most included studies had significant methodological limitations. Sixty-one of 79 studies did not include a control group, which is concerning in studies using self-reported or simulation-based end points. Most nonrandomized studies had inadequate adjustment for potential confounding factors, such as years of clinical experience, profession, team composition, age, and sex. Small sample size (<50 in 33 studies and not described in 16 studies) may likely have resulted in inadequate power and an inability to perform properly adjusted statistical analyses.
Recommendations for Future Research
Measuring effects of simulation-based training on clinical outcomes should be preferred whenever possible. However, studying rare clinical end points, such as mortality or severe morbidity, requires large, multicenter studies. Therefore, other clinical outcomes may be more feasible to obtain and may serve as surrogates related to treatment quality, eg, time to critical tasks or guideline adherence. We acknowledge that real-life evaluation of training effects may be difficult and time consuming given the paucity and volatile nature of critical clinical events. We advocate the use of a control group when designing new studies and random allocation to intervention and control whenever possible. Study protocols should be registered to avoid selective reporting. When applying simulation-based evaluation of training effects, we recommend the use of video recordings and blinded assessment using accepted and validated protocols. Compared with planned simulation-based retesting, the use of unannounced simulated mock codes may mimic real clinical encounters more closely because clinical professionals often prepare for planned testing. We recommend that the theoretical framework and process of debriefing is described, as these are important parts of simulation-based training that may affect the learning outcome.
Conclusions
To our knowledge, this systematic review of simulation-based pediatric team training is the first with an aim to collectively describe all current available literature on clinical, team performance, and learning outcomes. We included 79 studies published over the past 15 years and conclude that pediatric simulation-based team training improves clinical performance in time-critical tasks and increases adherence to guidelines. Simulation-based training improves team performance and technical skills for at least 2 to 6 months. Furthermore, it may improve survival rates, but these results are affected by heterogenous interventions and outcomes and studies without a control group. Future research should include longer-term measures of retention of team performance and other skills and patient outcomes or clinical measures of treatment quality whenever possible.
Acknowledgments
The authors acknowledge and appreciate the scientific contribution of the authors of all articles included in this review. The authors thank the editor and reviewers for valuable comments on an earlier version of this manuscript.
Dr Thim designed and planned the study, registered the study with Prospective Register of Systematic Reviews, performed abstract and full-text screening and data extraction, and drafted the initial manuscript; Dr Lindhard designed and planned the study, helped to specify the literature search, performed abstract and full-text screening, data extraction, and risk-of-bias scoring, and critically revised the manuscript; Mr Laursen specified and performed the systematic literature search, maintained data in the Covidence review platform, wrote detailed documentation for publication, and critically revised the manuscript; Mr Schram performed data extraction and risk-of-bias scoring and critically revised the manuscript; Dr Paltved initiated the study, provided intellectual support for all parts of it, and critically revised the manuscript; Dr Henriksen designed and planned the study, helped to resolve inclusion and risk-of-bias scoring conflicts, and critically revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.
FUNDING: The study was supported by Corporate HR, MidtSim, Central Region Denmark. The funding body had no influence on the design of the study; data collection, analysis, and interpretation; manuscript drafting; or conclusions.
References
Competing Interests
CONFLICT OF INTEREST DISCLOSURES: The authors have indicated they have no conflicts of interest to disclose.
Comments