Leveraging a Safety Event Management System to Improve Organizational Learning and Safety Culture Free

This quality improvement project occurred at a pediatric health system with a freestanding 206-bed children’s hospital with 31 service locations, 2 ambulatory surgery, 3 urgent care, and 20 primary care locations. Mean annual volumes were ∼46 000 total patient days and ∼759 000 ambulatory visits from 2017 to 2020.

The literature, process audit findings, and 1:1 and small group informal user feedback were reviewed in the evaluation of the SEMS to identify desired traits to create the ideal SEMS. It was determined that our health system’s internally crafted incident reporting system, KiD Katches, would require considerable redesign to be comparable to other commercially available SEMS; however, no commercial options were designed for pediatric health systems. A key driver diagram (Supplemental Fig 5) was developed. One of the drivers was the creation and implementation of a customized version of the Verge Health (Mount Pleasant, South Carolina) Converge Platform Patient & Employee Safety - Event Management module to best address pediatric safety events. The second driver focused on improved learning and engagement with the new SEMS. Our operational definitions and outcome, process, and balancing measures are described in Table 1. This project was undertaken as an organizational quality improvement initiative, did not constitute human subject research, and therefore, did not require Institutional Review Board approval.

All sites used the previous KiD Katches system, which was accessible from the EMR. Entries started with the selection of an overarching event group with one level of subgrouping. It used narrative, free text fields for both event entry and follow-up documentation. Over time, some event groups’ questions were modified to be more event-specific; however, this was the exception. The information collected was not standardized for data analysis nor readily searchable to facilitate breaking down event data beyond event groups, subgroups, and locations. Event submitters and reviewers were, at times, frustrated with these perceived and realized time-intensive workflow processes, which, in turn, limited organizational engagement with related safety efforts. This format limited analysis and data trending without time-intensive review and indexing of event details, which further deterred the use of the prior system to identify patient safety issues. Identification of safety trends from event reports progressively shifted away from local leadership and transformed into a reliance on the Quality and Safety Department to complete this essential activity.

Over the course of 3 years (2018–2020), feedback was garnered from across the organization to develop standardized, event entry and follow-up forms with accompanying processes to permit optimal tracking and trending of safety event data over time and increase local unit-based engagement, responsibility, and accountability with timely event problem solving.

In the new system, each entry starts with patient demographic information from the EMR and includes key data to future health equity analysis of events. Event detail forms contain specific questions based on the overarching event type selected. To ensure essential facts are collected, predefined, conditional branching logic triggers the addition or subtraction of questions based on answers provided to foundational questions. This format minimizes the number of “click selection” responses required, while maximizing discrete data points for building meaningful reports. For example, customized forms contain questions related to compliance with bundle elements used to reduce the risk of hospital-acquired conditions, defined by the Children’s Hospitals’ Solutions for Patient Safety Learning Network (SPS) (Cincinnati, OH; https://www.solutionsforpatientsafety.org).^5,6  As such, in the event type “Hospital Acquired Infections and Infection Prevention and Control Concerns” form, when the answer to the first question is “Central Line Associated Blood Stream Infection,” follow-up questions focus data collection on catheter details (eg, type, size, location), insertion (if applicable) and maintenance bundle elements, which become discrete data points for reports, analysis, and dashboards (Supplemental Fig 6). The follow-up form build mirrored that of the initial entry to capture expected follow-up actions and documentation.

To meet evolving organizational needs, the new SEMS is easily modifiable. Improvements to the original customized forms were made based upon frontline feedback over the first 6 months post implementation and continue to occur as ongoing process improvement. For example, a large iteration, completed in July 2019, included creating new event type forms identified through analysis of “Other” events submitted as well as adding questions and answers to address outpatient reporting needs.

“Locations” were created and included departments and service lines. Each was assigned an “Area Manager” responsible for reviewing events for their Location. A new process to improve direct engagement with our providers was initiated by creating new provider locations. Medical directors were then assigned as Area Managers, including the Pediatric Residency program with chief residents in this role. The organization monitored provider follow-up separately, as a subset of the overall follow-up compliance because of implementing this new process. Specialized reviews were performed by identified subject matter experts based on event types, such as the medication safety pharmacist for all medication events. The SEMS generated reminder e-mails for outstanding event reviews and color-coded task notices to identify events not completed per policy, thus it created visual cues and built accountability. When there was persistent incomplete follow-up, the Quality and Safety team provided reeducation and used progressive escalation to include the chief quality and safety officer who communicated concerns to senior leadership.

Standardized event entry streamlined entry times and facilitated review by local leaders. By ensuring each entry contained the correct information, events were routed to the proper reviewer, thereby increasing the completed follow-up compliance rate. Time for event entry and event review compliance rates were tracked.

Implementation of the new SEMS included establishing expectations for reviewers in the policy and platform training.⁵ 

The event reporting policy revisions included step-by-step procedures to define reporting expectations, local review and follow-up processes, and resource materials. Embedding JC principles^9,15,17  and psychological safety²⁰  in the policy enabled balanced reporting to support a safe learning environment and problem-solving culture. A JC algorithm (Supplemental Fig 7) was developed to clearly define when safety events resulting from an individual failure could result in disciplinary actions.

Anonymous submissions were allowed to improve reporting by submitters who did not feel safe identifying themselves. All anonymous submissions were reviewed by quality and safety leadership and, when the submitter identified themself within the event details, follow-up with the submitter occurred. A confidential response e-mail was developed to encourage non anonymous submissions, explain system limitations created and, most importantly, affirm psychological safety. Documented follow-up became a means of closed-loop communication with submitters regarding details gathered during event review, corrective actions taken, and lessons learned, preventing recurrence. This new system generated automated submission acknowledgment e-mails and follow-up notifications upon the review's conclusion, including event classification and completed causal analysis.

During implementation, multiple modes of training were used, including live 1-on-1 and group sessions, recorded videos, and tip sheets. The policy and training materials are available on our intranet and new users use those resources. Annual group training is performed with each class of rising chief residents. Additional training is provided as needed.

It was anticipated that a more efficient SEMS, with user support and resources available to overcome knowledge gaps, would result in increased overall reporting. When coupled with policies which support both JC and psychological safety to eliminate the fear of retaliation, without removing the anonymous submission option, a culture of reporting should be encouraged. Anonymous submissions will continue to be monitored to determine if the anticipated changes in the reporting culture are impacting the organization’s safety culture.

Statistical process control charts displayed data over time and Shewhart control chart rules were used for analysis and interpretation.²¹  All data analysis used Microsoft Excel (Redmond, WA) and QI Macros (Denver, CO).

Analysis of event reporting data from the previous system for January 2017 through September 2018, consisting of 9247 reports, was used to establish baselines for comparison. Any changes from the baseline data were tracked after the interval process change in October 2018, used to represent implementation of the new SEMS, annotated on figures, where comparison data were available.²¹  Analysis of post implementation event reporting data (October 2018 – March 2021) included 13 979 event reports.

The p-chart (Fig 1) shows an overall increase in event follow-up. The mean rate increased from 60.9% to 93.6% from January 2017 to March 2021. Beginning in November 2019 and again in June 2020, there was an upward shift in the centerline due to 8 data points above the centerline.²¹  Provider follow-up compliance, a subset of overall data from Fig 1, was tracked as a measure of engagement. The follow-up centerline for attending and resident physicians was 68.5% and shifted in May 2020 to 83.3% (21.6% improvement) (p-chart, Fig 2). The mean time necessary to submit an event report, a balancing measure for increased reporting, between 2017 and 2020 (X-bar chart, Fig 3) decreased from 16.6 to 10.9 minutes, a 5.7 minute reduction per the findings of automated reports from the previous and new systems.

The average number of event entries per month (c-chart, Fig 4) increased from 440 to 478 (8.6%) from implementation through March 2021. However, from April to June 2020, the coronavirus disease 2019 (COVID-19) pandemic impacted our organization with a resulting decline in overall patient volume, reducing the number of reports submitted and decreasing the average monthly entries, representing special cause variation. These data points were omitted from the calculations for the centerline and control limits.²¹  When more normal operations resumed in July 2020, event entries started showing a return to the pre-COVID disruption levels (c-chart, Supplemental Fig 8).

Anonymous submissions were tracked in the new SEMS as an indicator of psychological safety associated with event reporting (p-chart, Supplemental Fig 9). The centerline started at 12.1%, with downward shifts in December 2019 (9.9%) and June 2020 (5.4%), when 8 data points were consecutively below the centerline, representing a 55.3% reduction.

In summary, our redesigned SEMS was associated with increased event entry and ultimately event-related action item follow-up. The submitting process became simpler and consistent, allowing staff to anticipate time and information needed to report. When team members saw value placed on reporting through the SEMS, they felt safe and wanted to report more, as demonstrated by the decline in anonymous reporting and increase in overall entries. Actionable data elements from the SEMS allow the organization to align strategic goals, identify high-risk areas of concern, monitor zero harm initiatives, and track patient outcomes.^5,6,18  This information is compiled and fed into our annual organizational strategic planning process. The ongoing SEMS-mediated learning is key to organizational agility and supports the journey to becoming a high reliability organization.²² 

Event follow-up documentation allows the health system to learn from every event.^14,18,23  By constructing an efficient SEMS, we experienced improvements across all disciplines in identifying, reporting, addressing, and eliminating risks, thus making our workplace safer.^4,24  By integrating JC principles and providing closed-loop communication of completed follow-up, we created trust in the event reporting process, which is evident in the decline of anonymous reporting. By making the submitting process simpler, it is more predictable and reliable, thereby reducing barriers to reporting caused by time constraints and competing priorities.

Timely event review and follow-up, including problem solving, implementing mitigating solutions, and direct feedback to the submitter, are vital in sustaining a reporting culture.^5,6  Submitters need to see organizational changes resulting from their event reporting. At our organization, these changes were shared broadly at the daily safety brief, safety coach meetings, and other system and leadership meetings. Health systems must also recognize team members’ efforts to identify and report events and opportunities for improvement.²⁴  Creating a climate of safety requires the leadership team to model behaviors that demonstrate patient safety as a priority and coach frontline staff to speak up for safety.^5–7,9  Adequate event follow-up, which can now be reviewed by submitters, reflects leadership’s appreciation of frontline staff’s commitment to safety and results in long-term culture changes within the organization.⁵ 

There is a debate as to whether SEMS can improve culture and organizational learning by itself. Most hospital SEMS depend upon voluntary reporting rather than advanced automated incident monitoring systems, like those found on an automotive assembly line, which provide alerts when defects are noted in real time. Nonetheless, a SEMS is important for organizational learning when information is disseminated transparently with shared responsibility and accountability. There are multiple approaches to organizational learning. Defect mitigation often requires modification of organizational values and norms to optimize learning.^8,25–27 

Senge²⁸  described 5 key disciplines that need to converge to innovate learning organizations: systems thinking, personal mastery, mental models, building shared vision, and team learning. We have tried to implement these aforementioned traits into our organization in an attempt to build the ideal LHS through removing the punitive elements of event reporting and replacing them with learning opportunities. As team member psychological safety is assured, safety reporting increases in frequency. This results in identification of more learning and improvement opportunities, which then reinforces the organizational value placed on event reporting and restarts the cycle of positively impacting the organization’s safety culture. Our organization is still on the journey to becoming a learning system, facilitated by our robust SEMS.

There were several limitations. Our SEMS depends upon voluntary reporting, which captures only a fraction of the safety events that are otherwise detectable through chart reviews or automatic trigger systems. The relationship between the frequency of SEMS reports and objective measures in health systems is unclear.²⁹  The ability to monitor and measure user logins for the sole purpose of event review is not available from the SEMS vendor.

Definitions for ambulatory encounters are not fully established, so we used the broader definition as adopted by the SPS Ambulatory Safety workgroup as of August 2020, which will likely change as this work evolves.

The overall quality of the event follow-up documentation was not pursued as part of this paper but will be a future goal. A new analytic platform for building dashboards customizable to quality initiatives of a clinical division at the local department level was implemented. These dashboards permit regular visual management of safety event data and will become integral to risk identification.

The creation of a robust SEMS supports a culture of reporting and allows for local problem solving and learning to occur from every reported event. Barriers to reporting are reduced when staff can anticipate how much time and information are needed to facilitate the review and follow-up of safety events. By responding to these events in a timely and effective manner, local leadership demonstrates their commitment to safety and quality, continuous learning, and improvement.^30,31  This behavior encourages a transparent culture, where acceptance and consideration of all viewpoints is normalized, assuring psychological safety, and providing staff with confidence that their concerns will be met with timely follow-up.^30,31  A health system becomes safer by learning from these events and implementing cyclical, systematic changes to improve the safety of care delivery.¹⁸  The importance of having a solid organizational safety culture that supports the journey to becoming an LHS, and eventually a high reliability organization, cannot be overstated.

We thank Mr Christopher Mangum, Mr Robert Sepanski, and Dr Arno Zaritsky for their review and guidance with documenting these efforts. We thank Ms Susan Berman, Ms Yvette Conyers, Ms Elizabeth Martinez, Ms Elizabeth Rogers, Ms Angela Ortiz, and Ms Erin Smith, along with many other subject matter experts across the organization, for their contributions to building the safety event management system forms and conditional branching. We thank Ms Karen Mitchell, Dr Christopher Foley, and Dr Robert Kelly for providing fundamental support during implementation as agents of change management. We thank Dr Lloyd Provost and Dr Robert Lloyd for providing invaluable statistical process control chart guidance.