Evaluation of a Theory-Informed Pediatric Rapid Response Training Program Using the Logic Model

The educational program was developed and implemented after institutional review board approval in a quaternary pediatric hospital with >800 inpatient beds across multiple geographic locations. Our intended audience included multidisciplinary RR activators (nurses, physicians, trainees, advance practice providers [APPs], and respiratory therapists). We announced the workshop hospital-wide and provided information on format and content to leaders across the institution. Participants were enrolled on the basis of voluntary interest and recommendations from their supervisors. The RR workshop was designed to (1) foster participants’ autonomy, competence, and relatedness; (2) equip participants with knowledge and skills to proficiently function during RR events; and (3) improve participants’ competency and satisfaction with performance in RR events. The participants completed preworkshop-postworkshop and follow-up surveys, workshop evaluations, and interviews throughout the program. The instructors and stakeholders shared insights via focus group discussions and team meetings. We studied participants, instructors, and stakeholders to assess (1) the impact of participation on RR activators’ experiences pertaining to the program goals, (2) the effectiveness of the workshop to support components of self-determination, and (3) the applicability of logic model to provide an evaluation approach representative of evaluation standards.²⁴ 

With our initial results from piloting the RRT workshop, we ascertained that the program could be implemented as intended and that immediate outcomes (improvement in RR knowledge and/or skills and SDT components) could be achieved.¹³  These findings are prerequisites to the intermediate and longer-term outcomes of this program according to the logic model (Fig 1).

Funding: The program directors received an educational grant to support the initial program development and had protected time (10% of their full-time equivalent) assigned for education.

Educator commitment: The program directors and instructors worked collaboratively to develop the program. The directors spent ∼6 hours per session for preparing and teaching during the workshop. In each workshop, 8 additional instructors invested 2 hours teaching the concurrent skills stations. All instructors were ICU faculty with expertise in simulation and represented all hospital locations.

Facilities: A conference room was used for the didactic learning. Case discussions and skills stations were conducted in simulation, conference, and debriefing rooms. Equipment used included simulation mannequins, code cart, and respiratory supplies.

Institutional commitment: There was a drive from hospital-wide multidisciplinary leaders to get their providers engaged and trained.

Needs assessment was performed via focus group discussions, surveys, and engagement of stakeholders across the hospital.

Program development: We used the SDT to design learning conditions to foster learners’ autonomy (ie, desire to maintain control), competence (ie, desire to gain knowledge and skills), and relatedness (ie, desire to be part of the team and contribute and/or share with the team). In the introductory didactic learning, we highlighted mechanisms by which participants could maintain control over patient’s care. The case-based learning provided opportunity to practice autonomy of decision-making. The participants engaged in hands-on practice of RR knowledge and skills, and immediate feedback was provided to foster competence. The workshop activities encouraged participants to clearly delineate clinical roles and ways to effectively contribute as a team member. SDT principles were also used to guide program-related decision-making to achieve the goal of the RR program, which was to enhance RR activators’ competency and satisfaction with performance during pediatric RRs.

Instructional methods based on best educational practices: The program was designed as a 4-hour workshop with various instructional methods. These activities were strategically sequenced on the basis of Kolb’s experiential learning cycle.^25,26  Each session started with a presentation of RR situations (concrete experience). In the interactive didactic learning, we introduced critical concepts (reflective observation). Participants then engaged in case-based discussions (abstract conceptualization), followed by skill stations and simulation scenarios (active experimentation).

Iterative curriculum modification and program maintenance: Several modifiable opportunities, such as shortening the didactic learning, targeting 4 clinical domains (respiratory, cardiac, shock, and neurologic decompensation) in cases, creation of a case bank, and addition of low-fidelity simulation, identified during pilot program implementation¹³  were used to guide eventual program activities. We maintained ongoing communication with leaders across the hospital to identify and enroll participants in the workshop sessions.

Outputs of the program included the number of learners engaged, faculty recruited, educational modules and assessment tools created, and the total number of sessions conducted.

We used the standards for program evaluation (feasibility, accuracy, utility, and propriety) to determine outcomes.²⁴  We aimed to gather evaluative data that would be “feasible to collect, inform decisions, and an accurate representation of stakeholder perspectives” via preworkshop-postworkshop learner assessment and workshop evaluation surveys.²⁷  The preworkshop-postworkshop and follow-up surveys were developed by the program directors with guidance from educational and survey experts. The workshop evaluation form was an institution-recommended standard evaluation form used for learning activities. We created a database to organize enrolled participants, educators, clinical cases, prework-postworkshop survey responses, and workshop evaluations for serial sessions. All the surveys are available in the Supplemental Information.

Immediate outcomes: Change in knowledge and skills of the participants was assessed through the preworkshop-postworkshop self-assessments, which comprised clinical case vignettes depicting common pediatric emergencies (respiratory, cardiac, neurologic, and sepsis and/or shock) on the basis of real-life RR experiences in our institution. The responses to the vignettes were scored (score range: 0 [incorrect], 25–75 [partially correct], 100 [completely correct]) by investigators. The scoring system was developed by consensus among the program directors and RR administrative team members. The scoring was performed by the program directors (A.C.B. and J.K.T.). Any disagreements were discussed to derive the final scores. The preworkshop-postworkshop surveys included questions that assessed baseline and improvement in participants’ perceived autonomy (how frequently do you feel you are able to maintain control over patients’ care during RR?), competence (how frequently do you feel you have the knowledge to manage the clinical situation?), and relatedness (how frequently do you feel you are able to contribute as a team member?) on 4-point Likert scale. The evaluation survey examined preworkshop-postworkshop self-efficacy for essential RR domains and effectiveness of workshop structure, organization, and instructional processes on 5-point Likert scale.

Intermediate outcomes: We administered a survey at 6 months after the initial session that included case scenario–based, retrospective, preworkshop-postworkshop, 4-point Likert scale questions to determine the program’s impact on participants’ perceived self-efficacy in RR-related knowledge and skills. Additional questions addressed participants’ confidence levels during real-life RR events in regard to SDT components. We interviewed a sample of participants who participated in real-life RR events after having participated in the workshop to gain contextual insights of participants’ real-life RR experiences. The investigators developed and conducted semistructured interviews to gain participants’ perceptions about their application of lessons learned to clinical practices, impact on patient outcomes due to such application, and ongoing interactions and/or team dynamics during RRs. To assess growth and sustainability of the program, we documented ongoing engagement of hospital-wide leaders, learner participation, recruitment and turnover of instructors. We tracked enrollment, participation, and engagement as indicators for growth of the program.

We documented context and impact of the program over the years. Context refers to “important elements of the environment in which the program takes place, including social, cultural, and institutional political features.”²⁸  We examined the program’s impact pertaining to both intended and unintended changes on the RR system performance and professional development of workshop faculty. Our RR administrative team has a robust method to evaluate system function and team performance. Multidisciplinary leaders (nurses, respiratory therapists, APPs, chief residents, and medical and quality directors) representing the afferent and efferent RR sides debrief with all parties involved in actual RR events. The RR administrative leadership and the multidisciplinary leaders meet bimonthly to review RR events, discuss debrief findings, and rate team performance and outcomes. The identified improvement opportunities are thematically categorized by the group into crisis resource management gaps (communication, situational awareness, teamwork role clarity, and policy and/or procedures). A database of these discussions is maintained. The RR administrative data provided us with the information on teamwork, critical communication, and improvement in RR clinical outcomes. We used the RR and hospital-wide cardiopulmonary resuscitation committees’ data to examine clinical impact on RRs and cardiac arrests.

We used descriptive statistics to summarize participant demographics, means and SDs for continuous variables and frequencies and distributions for categorical variables. We used McNemar test and paired t test for preworkshop-postworkshop comparisons of categorical and continuous variables, respectively, and Wilcoxon rank test when appropriate. We ascertained content validity by developing the clinical vignettes and self-determination–related questions on the basis of real-life experiences and SDT principles. The author team discussed and iteratively refined the questions. Response process validity was assessed by using the instruments on a focus group comprising RR administrative team members, workshop educators, and representatives of various learner groups. For the internal structure validity, Cronbach’s α was used to assess internal consistency between individual items assessing autonomy, competence, and relatedness; overall consistency for the 3 SDT components; and cognitive knowledge assessed via vignettes. Statistical software used was Stata version 15.1 (Stata Corp, College Station, TX). We used a deductive and interpretive approach to content analysis of information obtained during the interviews.

The program delivered 20 workshops from 2015 to 2019 and trained 207 participants. Of these, 51% were registered nurses, 37% APPs, 9% respiratory therapists, and 3% trainees. The range of participants’ clinical experience was 6.1 years, with the average experience of nurses being 9.1 (SD: 8.6) years. From the total, 104 (47.5%) participants ranked their overall RR experience as good. All participants in their clinical roles had assigned responsibility of RR activation and 43 (19.6%) had direct experience of activating a RR event in real-life.

The workshop has been iteratively modified, and the educational content, including case discussions and simulation scenarios, are adapted to the needs of RR activators and customized to the participants’ specialty (eg, surgical cases for surgical APPs or cardiac cases for cardiology nurses). A standardized case bank has been created. The preworkshop-postworkshop assessments and the evaluation surveys modified on the basis of pilot sessions¹³  are formalized and used for all the sessions. There was good internal consistency, as depicted by Cronbach’s α for all self-determination items (0.91) and autonomy (0.84), competence (0.75), and relatedness (0.82) questions. The Cronbach’s α for cognitive knowledge tested with clinical vignettes was 0.602, which depicts reasonable consistency. We have a standardized method to schedule workshops and enroll learners. We have recruited and trained additional instructors to ascertain a standardized approach to teaching and the facilitation of a safe learning environment. We track participation from different specialties, send course completion information to their leaders, and maintain a waitlist for future enrollment. We maintain a roster in which we track contributions from educators and retain information on feedback received from participants. We have had no attrition of educators, and our core program faculty now comprises 25 providers.

Knowledge change: Preworkshop-postworkshop comparisons revealed that knowledge in all 4 clinical domains (neurologic, cardiac, respiratory, and sepsis) improved significantly (P value <.001) (Fig 2). In regard to the RR system, the accurate answers depicting comprehension of emergency response-activation process (preworkshop: 86.8% to postworkshop: 98.6%) and RR resources (preworkshop: 67% to postworkshop: 83.1%) improved significantly (P value <.001).

Self-efficacy and SDT components: Improvement in participants’ self-efficacy related to various RR domains is depicted in Table 1. Participants self-reported preworkshop-postworkshop efficacy revealed significant improvement (P value <.001) across all 3 SDT components: autonomy (preworkshop: 2.56 to postworkshop: 3.61), relatedness (preworkshop: 2.80 to postworkshop: 3.68), and competence (preworkshop: 2.82 to postworkshop: 3.61).

Educational effectiveness: In Table 2, we display the evaluation of workshop structure and instructional activities. The overall workshop mean score was 4.85 (0.27).

Fifty-two (25%) of the 207 participants completed the follow-up survey. Among these, 55.8% were APPs, 32.7% registered nurses, 5.8% respiratory therapists, and 3.9% trainees. The retrospective assessment about utility of the workshop and self-reported preworkshop-postworkshop performance change during RR events is depicted in Table 3. Participants continued to report perceived improvement in self-efficacy for all 3 SDT components (P value <.001): autonomy (preworkshop: 2.47 to postworkshop: 3.42), relatedness (preworkshop: 2.45 to postworkshop: 3.46), and competence (preworkshop: 2.33 to postworkshop: 3.33) during real-life RR events.

At the interviews with 25 former participants, including nurses, APPs, and residents, many shared that the lessons learned had helped them with prompt recognition of patient decompensation, systematic preparation for resuscitation, and effective communication. Some felt empowered to speak up to express concerns confidently during RR events and participate in development of management plans. Some shared that they have used family-centered care during RR events using what they learned about utility of family liaisons in the workshop. Some also observed improvement in role clarity and team dynamics during real-life RR events.

Structural: Starting as a scholarly project supported by an educational grant, this program has become the recommended training for the activators of RR events in our institution. The RR administrative team strongly endorses this training for providers across the hospital. The educational mission and outcomes are now a part of the hospital-wide RR system.

Political: The program has gained traction from different specialties across the hospitals. Surgical and nursing groups have made the workshop an essential component of new provider orientation. Many participants express their interest to help with educating or facilitating future sessions. The program is disseminated externally and has leveraged resources and ongoing support.

Cultural: At the inception, most participants attended the workshop to address their perceived knowledge and skill gaps or were mandated to attend by their leaders. With acknowledgment of educational benefits among RR activators and their leaders, the program is now used as an educational tool to enhance RR skills and teamwork.

Clinical: Our educational program, along with many other system-wide improvement initiatives, has contributed to positive changes with RR system performance. The RR events reviews indicate improvements in teamwork, empowerment on part of the RR activators to trigger RR, increased ability and/or willingness to start stabilizing the patient before the ICU team arrives, comfort and desire to help the ICU team resuscitate, and comprehensive understanding of the RR system by RR activators. Practice of critical communication, professionalism, and teamwork during RR events has become a norm, and the RR administrative leadership receives complaints and/or safety concerns when this is not followed. The number of RR events that evolve into cardiopulmonary arrests outside ICUs has reduced from 8 to 10 per year during the period of 2010 to 2015 to, now, 2 to 4 events per year. Also, the reported number of RR events with lack of role clarity or communication gaps has decreased by 23% in past 3 years.

The program has benefited the core faculty group in terms of their professional development and promotion. Both program directors received educational awards. The instructors have capitalized from their involvement through credits toward their educational portfolios.

We gathered quality evidence of our theory-informed educational program using the logic model. The comprehensive evaluation provides accrual evidence that our program achieved desired educational outcomes (knowledge, skills, and attitudes) and was effective (structure, organization, and process) and applicable (lessons learned transferred to clinical practice). The program has evolved its structural, political, and cultural identity to become a recommended training curriculum of the hospital’s RR system; has positively impacted the RR system; and has enabled the educators’ professional transmutation.

To our knowledge, in this report, we describe the first SDT-based training program for RR activators with a comprehensive description of program evaluation. We hope that the experiences we share and the outcomes of the logic model are informative to other program developers. Our guiding principles and program components are adaptable to educational activities, such as continuing education, simulation, and/or practice-based learning exercises, for training residents, nurses, and other providers.

The logic model framework was used to guide us through all the steps of this multiyear program. With this model, we facilitated the building of program systematically and provided clarity and alignment of evaluation components. It enabled us to capitalize on the evaluation data to inform program improvement and optimize its impact. The evaluation data are comprehensive, tested for validity, and composed of learner knowledge tests, surveys, and interviews. In addition to improvements in knowledge, skills, and attitudes among participants, the lasting effects of the workshop on supporting 3 self-determination components are worth highlighting. The program directors provided oversight, audited sessions, and solicited feedback from educational experts to ascertain a safe and supportive learning environment.

According to literature, negative emotions exist among RR activators related to team dynamics during RR events.^15,16  Team psychological safety studied extensively in many fields, including health care and medical education, comprises components of inclusion, learner, contributor, and challenger safety.^29,30  Psychological safety is the perception that it is safe to take interpersonal risks within the work environment, which is foundational to promote participation in teams and sharing of thoughts and/or concerns. When team members feel safe, they exhibit curiosity, ask clarifying questions, and reveal their vulnerabilities.³¹  We assert that psychological safety within work or learning environments has profound impact on RR activators’ participation and team performance during RR events. Hence, we emphasized fostering autonomy, competence, and relatedness among workshop participants. Our participants reported that they felt comfortable to participate as a team member, seek information, and communicate concerns during the RR events after the workshop.

Our results reveal (1) that our SDT-guided RR educational program enhanced participants’ self-efficacy and perceptions of autonomy, competence, and relatedness with RR events; (2) that the program was recognized as valuable and is recommended as an RR training curriculum within the organization; and (3) that the program faculty achieved professional development. The comprehensiveness of the logic model enabled us to assess and attain the desired outcomes.

Although we have collected objective data through knowledge tests, a portion of the results are subjective and self-reported. The self-reported data provide comprehensive evaluation but are also at risk for social desirability or recall bias. We hope that the positive learning environment we created has facilitated acquisition of valid feedback. The intermediate outcomes represent only a portion of all participants. Possible reasons for low response include change in roles or lack of available time and/or motivation to complete a repeat survey. We anticipated challenges with response rate at the follow-up and attempted to gather quality evidence from multiple sources over time to maintain credibility of the evaluation. We acknowledge that only a small proportion of intended audience has been trained with the workshop, and although providers disseminate the knowledge and skills to their peers, ongoing work to train providers across our institution is needed. We acknowledge that the educational impact on clinical outcomes across any system is multifactorial. The impact of a single program or intervention could be contributory to the improvement of the whole system, but the definitive cause-and-effect relationship cannot be established.

Our SDT-informed educational program was successful in improving participants’ competency and satisfaction with performance in pediatric RRs. The logic model, rigorously used from the program inception, provided a useful framework to guide systematic program development and comprehensive evaluation.