A Customized Triggers Program: A Children’s Hospital’s Experience in Improving Trigger Usability Free

Several years after implementation of Children’s National Hospital (CNH) triggers program, enthusiasm for the program was limited, perhaps driven by concerns for clinical usefulness. The report logic and definitions were broad, resulting in false-positive triggers and ultimately lower use among clinical end users (eg, physicians, nurse managers, unit directors). We sought to revitalize the CNH triggers program as an innovative safety tool using quality improvement methods to increase the value-add of the program, focusing on automation and customization of triggers to create a stronger, more impactful workflow.

As part of our ongoing efforts to increase adoption of the triggers program across the organization, we launched an improvement effort using a novel, tailored approach that sought to increase the trigger signal. We defined a measure to quantify the clinical utility of our triggers, termed “trigger signal,” as the percent of cases identified automatically by the triggers program that represent a true occurrence of a targeted adverse or precursor adverse event per total triggers activated. The specific aim was to double overall trigger signal from our baseline of 8% to 16% and sustain for 12 months.

CNH is a free-standing, academic children’s hospital located in Washington, DC with 323 acute-care beds with over 8000 employees. The CNH pediatric triggers tool program (created in 2013) first adopted concepts from the GAPPS and IHI trigger tools, which provide a framework for useful quality metrics to detect adverse events and opportunities for process improvement.^17,22  Our preintervention data showed a baseline of 8% trigger signal before the changes described in this quality improvement report. The current program, a hybrid model of automatic trigger reports with human oversight, aims to capture data from the EHR and apply it in a clinically meaningful, customized manner to improve patient safety and organizational quality.

This project is a quality improvement initiative determined by the CNH Institutional Review Board to not constitute human subjects’ research. Standards for Quality Improvement Reporting Excellence (SQUIRE) guidelines were used to present this work.²³ 

Table 1 describes the desirable characteristics of triggers: automaticity, usability, and end user reliability. CNH’s original triggers were chosen based off the IHI Trigger Tool.¹⁷  Since 2018, the program expanded from 6 triggers to 10 triggers, highlighting the adaptability of the program through an iterative process of retirement and addition of triggers based on feedback from end users and improvement in logic creation (Table 2). See Appendix for the description and logic used to create each trigger.

A key driver diagram, developed by a multidisciplinary team, shows opportunities to improve triggers data in 4 areas: program structure, reliable data analytics, organizational awareness, and multidisciplinary end user engagement (Fig 1). We developed the following interventions to accomplish these goals (Table 3):

A key aspect of the program is the new creation of a collaboration between the Performance Improvement department, reporting to the Chief Quality and Safety Officer, and the department of Clinical Informatics, reporting to the Chief Medical Informatics Officer. These teams include data analysts and specialists who work on performance improvement measures for the organization, which represents a total of 4 to 6 individuals across the 2 departments, including a Triggers Program Lead. The structure utilizes each department’s strengths, primarily the ability to use data that comes from Clinical Informatics, in combination with the Performance Improvement department’s culture of safety. Although triggers are data-driven, they require quality and process improvement methods for the information to be useful to the organization. This connection improves the efficiency with which a trigger can be developed and the reception by clinical teams.

Previously, there was a central steering committee that met monthly to review all triggers; however, this committee was not effective at discussing triggers in an actionable manner because of lack of accountability for trigger reports and follow-up. The new structure links clinical committees or divisions with individual triggers to capture clear stakeholders for continuous, iterative improvement of the triggers. The new committees meet monthly to review trigger data in detail when reporting to their chair. Additionally, many of the committees report annually to the clinical effectiveness committee, and they use trigger data for pertinent quality and safety meetings. As an example, the peripheral intravenous (IV) infiltrate group has specific organizational safety goals that are aligned and customized within the peripheral IV infiltrate trigger (Appendix).

When customizing a trigger, we work directly with end users to align with the team’s clinical requirements. This iterative process requires regular conversations between the end users and the triggers program team. Continuous engagement allows end users to add fields and relevant clinical information to triggers to increase trigger signal. For example, the initial trigger for acute kidney injury (Table 2) focused on increased creatinine but was not specific and produced a significant number of false positives, yielding poor trigger signal. Therefore, after review with end users, this trigger was replaced with more specific criteria.

We migrated our triggers data source from the EHR database (Cerner Millennium, Kansas City) to our Electronic Data Warehouse (EDW) (Cerner HealtheIntent, Kansas City), which is a platform that allows for aggregation of near real-time data from the EHR. Triggers created using data from the EDW can identify more specific cohorts of information (guided by the clinical teams), which reduces false positives. The EDW allows for a more fully automated, secure, self-service model for the clinical teams, without relying on scheduled retrospective reports to be distributed. This automation reduces dependency on technical staff, reduces turnaround time, is readily available for clinical teams, and decreases the time required to create new triggers or modify existing triggers.

Although CNH’s safety event reporting system is linked to the triggers program, not all triggers represent adverse events. This created dissatisfaction with the program because clinical teams felt there were many false positives. The burden was placed on clinical teams to investigate a trigger, creating nonvalue-added work. Each trigger now has clear guidelines, determined by the clinical teams, that prompt a safety report. For example, the peripheral IV infiltrate trigger identifies hyaluronidase administrations, which were sometimes not known to the clinical teams (Appendix). Before the trigger, the detection and reporting of IV infiltrates were low without much surveillance, but the addition of the trigger based on hyaluronidase administration fosters communication with clinical end users.

CNH’s triggers team works with end users closely to continuously discuss the development process, what types of triggers may be possible, and what characteristics are likely to produce higher trigger signal. Using previous trigger logic, we can build new or adapt old triggers at an even faster rate. Fields extracted from the EHR that have been successful in previous triggers (ie, antibiotic time ordered and administered, reassessment times, laboratory results, etc.) can be used in other triggers as well. For example, the logic for administration of naloxone (opioid antagonist trigger) can be repurposed to build new triggers, such as administration of hyaluronidase for the peripheral IV infiltration and extravasation trigger (Appendix).

The primary outcome measure to evaluate the utility of CNH’s customized program is trigger signal, a term developed to track each trigger’s positive predictive value. The denominator is all reports that result from each trigger in a day. This information is reviewed using established end users’ guidance and iterative feedback to determine which reports are truly clinically useful (numerator). For example, the acute kidney injury trigger may produce a report in 2 patients who received nephrotoxic antibiotics 3 days in a row with an increase in serum creatinine (Appendix). However, if 1 patient’s creatinine is not above 0.5mg/dL (as previously determined with the acute kidney injury committee), that report would be considered a false positive, resulting in a trigger signal of 50%.

Secondary outcomes include: the number of clinical committees involved in redefining the triggers program (a surrogate marker for involvement of key stakeholders) and the number of event reports filed as a direct result of the program. Here, we use event reports as a proxy of improving patient safety, as these reports are used directly to inform safety changes and decisions in the hospital.⁷  As a balancing measure, it is important to ensure that with increased customization and user-centered design, the time needed to create and implement a new trigger does not increase. As a high-reliability organization, our institution places priority on not increasing nonvalue-added work or time for our employees; therefore, we want to ensure that any increase in the complexity of the trigger development process is not wasteful.

To assess study progress, we use statistical process control charts. We identified special cause variation and shifting of the centerline as outlined in the Improvement Guide: 8 or more consecutive points above or below the centerline (shift) or 6 consecutive increasing or decreasing points (trend).²² 

We collected data monthly from January 2018 through August 2021 (baseline January–December 2018, implementation January–December 2019, and sustainability January 2020–August 2021). The control chart shows the cumulation of every trigger’s trigger signal (termed: “overall trigger signal”) by month (Fig 2). Trigger signal increased, surpassing the goal of doubling our baseline by December 2020. In early 2019, trigger signal shifted significantly to 44%, likely because of the implementation of several interventions (change in program structure, increasing stakeholder engagement, and development of self-service reports for end users) all at the end of 2018 or beginning of 2019 (Table 3). Additionally, 2 triggers were introduced to the program at the beginning of 2019 that use the HealtheIntent platform with more specific criteria (Table 2). There is a decrease during 2020 to 31% (annotated for hospital-level changes during the coronavirus disease 2019 pandemic related to cohorting of patients based on respiratory viral status and overall decline in census), with a recovery to 41% during 2021 as census increased. Additionally, in 2021, as the census returned to prepandemic ranges, a few triggers retired and more specific triggers were introduced to the program (Table 2).

We identified the number of dedicated clinical committee owners overseeing each trigger, which has increased from 2 to 8 committees. One example is linking the Suspected Severe Sepsis trigger to the Improving Pediatric Sepsis Outcomes Committee, which led to refinement of that trigger over time. The number of event reports resulting from triggers increased dramatically from 9 in 2017 to 188 in 2021 (Fig 3). The balancing measure (time it takes to create and implement a trigger) decreased over the study period from approximately 8 to 12 months to 3 to 4 months.

The growth of CNH’s triggers program is an opportunity to employ novel strategies to remain impactful for patients and providers; this evolution requires a customized, iterative methodology rather than a static, standardized program to increase trigger signal and optimize utility for the clinical teams. The interventions that have the largest impact on improving trigger signal include instituting triggers program oversight by both quality and clinical informatics, creating self-service reports for usability by clinical teams, and facilitating continuous stakeholder engagement. These interventions create sustainability in the triggers program by linking multiple departments and facilitating committees and end users to have direct input in the creation and usability of the trigger data. Shifting to multiple trigger committees and linking the triggers to specific safety event reports coincide with the coronavirus disease 2019 pandemic, where we saw a drop in patient numbers and a change in our hospital’s admission protocols. We do not feel that these interventions alone had a negative impact on trigger signal, as evidenced by the recovery at the end of the year with return to prepandemic protocols and census. Strengths of this project include the reliance on multidisciplinary teamwork and the focus on integrating electronic data with end user feedback to promote a process improvement culture that is the foundation of a high reliability organization. With improved real-time awareness of the safety events, committees can use the data obtained from the program to achieve these goals.

Although focus on retrospective harm identification is important, this tool is limited in improving patient safety; only a minority of adverse events are captured retrospectively.²⁴  Therefore, the future of patient safety includes developing innovative safety tools (such as customized, iterative triggers programs) that incorporate information technology and real-time identification of potential adverse events.²⁵  However, there is not currently robust literature that identifies how to improve the utility of triggers or how to adapt a triggers program.²⁶  Recently published work by the Children’s Hospitals Solutions for Patient Safety (SPS) Network highlighted engagement with key stakeholders (including parents, clinicians, and health system leaders) to identify a research agenda for pediatric patient safety; 1 of the top-priority research topics identified was early detection of patient deterioration.²⁷  Triggers programs serve as an example of this aspect of patient safety. Early pediatric work demonstrated that trigger-based systems performed poorly in terms of predictive performance compared with score-based systems for outcomes (ie, respiratory and/or cardiac arrest, unplanned transfer to the pediatric ICU, and/or unexpected death) and highlighted the gap in literature for how to optimize trigger-based programs.²⁶  As shared by SPS, parent responders highlighted early detection of deterioration as key areas for patient safety research; our work with the CNH triggers program contributes to this aspect of patient safety research.²⁷ 

Most pediatric early warning systems have been developed by expert opinion alone, and only a minority have undergone any assessment of predictive validity.^26,28  One challenge is defining and assessing meaningful outcomes as mortality is a rare event in hospitalized children. Measures that are associated with mortality, such as critical deterioration (defined as ICU transfer and ventilation or pressor support within 12 hours), may have potential for complementing existing patient safety measures.²⁵  However, although validation and outcomes data are important, implementation of systems is also a crucial step. A system should have structure for communication and build consideration of risk deterioration into daily practice.²⁶  Efforts to understand the usability, validity, and calibration of tools (including triggers) is essential because a tool that provides false alerts while missing critical deteriorations carries harm potential by triaging resources incorrectly and increasing response times through “alarm fatigue.”²⁹ 

One important novelty of this work lies in the continuous modifications of the program and the use of trigger signal to monitor clinical utility. Although a higher trigger signal may be considered better, the optimal trigger signal is not necessarily 100%; a high trigger signal may indicate an overly narrow definition, producing data that are not sensitive enough, missing precursor cases of interest, and eroding trust in the program. We have seen tremendous increase in end user engagement with an average trigger signal of 40%, however, more investigation, not only at CNH, but within pediatrics and healthcare at large, needs to occur to determine the optimal trigger signal for a triggers program. Our goal initially was to increase trigger signal by doubling our baseline rate (16% from baseline of 8%), but through iterations of our triggers program over the last 3 years, it has become clear that a trigger signal of under 20% is still not sufficient. Although there are no known benchmarks, we consider individual trigger signal to be optimized once a trigger is deemed clinically meaningful and actionable by the clinical end users. This may be shown by the real-time patient care changes and increased number of event reports resulting from the program that in turn allows for process improvement and patient safety changes. Our program’s trigger data fuels many of the improvement initiatives throughout the organization, with data continuously being used for CNH’s preventable harm index.²¹ 

CNH’s trigger tool is a continually evolving system, and the future lies in expanding the program in clinically meaningful directions. First, the data that arises from trigger reports has created several new quality improvement projects to improve patient safety, including surveillance of sepsis and monitoring nephrotoxic medications in acute kidney injury. Second, we are looking to expand to areas outside of inpatient care, such as ambulatory and surgical centers, as this has been identified as a pediatric safety research priority.²⁷  In addition, novel triggers can be implemented using a framework to detect nonbinary information such as social determinants of health and socioeconomic risk factors, perhaps identifying patients at who are at-risk for poorer health outcomes. From this program, we hope to improve outcomes in safety events, including reduction in peripheral IV infiltrations, nephrotoxic acute kidney injuries, and unplanned transfers, along with improving sepsis-related outcomes and processes.

Limitations of this work and its generalizability include that the resources dedicated to triggers may not exist at other programs. For institutions with limited resources, we recommend focusing on 1 or 2 key triggers or trends identified from event reports and/or institutional priorities. Additionally, this work takes place at a single institution, but future work can involve multiple hospitals. Finally, there is no singular outcome that directly links an increase in trigger utility to an improved patient safety experience. Despite this, our institution believes that the triggers program is useful to identify improvement areas, therefore we have committed time and effort to improve the utility of this program and are evaluating more direct patient safety outcomes to track.

The triggers program at CNH shows that there can be successful improvement of clinical support by optimizing trigger signal to provide actionable, near real- time, surveillance tools while encouraging a culture of process improvement. Trigger signal may be a measure that hospitals can use to track the success of their programs; however, more studies are required to develop the optimal trigger signal percentage. Linking the oversight of a triggers program with quality improvement and clinical informatics, improving stakeholder engagement, and developing trigger reports that are easily available for end users improved our trigger signal. Hospitals of comparable complexity and size can implement a triggers program and see similar success by demonstrating the clinical significance of their trigger tools.

We thank the current leads of each trigger tool committee for their work to improve patient safety at Children’s National Hospital: Dr. Nada Mallick and Rosemary Szeles of Late Rescue and Codes Outside the ICU Reduction Committee; Jacqueline Harnarine, Dr. Aadil Kakajiwala, and Dr. Marva Moxey-Mims of the Nephrotoxic Acute Kidney Injury Prevention Committee; Sofia Perazzo and Smitha Israel of the NICU; Daniela Herrera of the Pain Committee; Alia Fink of the Improving Pediatric Sepsis Collaborative at Children’s National Hospital; Dr. Andrew Matisoff of the Sedation Committee; and Eliana Maldonado of the peripheral IV infiltration and extravasation Prevention Committee. We would also like to acknowledge the Quality and Safety Department for their guidance and involvement in the triggers program, Dr. Eva Rubio and Katherine Moore.

CNH

Children’s National Hospital

EDW

electronic data warehouse

EHR

electronic health record

GAPPS

Global Assessment of Pediatrics Patient Safety

IHI

Institute for Healthcare Improvement