OBJECTIVES:

Excessive alerts are a common concern associated with clinical decision support systems that monitor drug-drug interactions (DDIs). To reduce the number of low-value interruptive DDI alerts at our hospital, we implemented an iterative, multidimensional quality improvement effort, which included an interdisciplinary advisory group, alert metrics, and measurement of perceived clinical value.

METHODS:

Alert data analysis indicated that DDIs were the most common interruptive medication alert. An interdisciplinary alert advisory group was formed to provide expert advice and oversight for alert refinement and ongoing review of alert data. Alert data were categorized into drug classes and analyzed to identify DDI alerts for refinement. Refinement strategies included alert suppression and modification of alerts to be contextually aware.

RESULTS:

On the basis of historical analysis of classified DDI alerts, 26 alert refinements were implemented, representing 47% of all alerts. Alert refinement efforts resulted in the following substantial decreases in the number of interruptive DDI alerts: 40% for all clinicians (22.9–14 per 100 orders) and as high as 82% for attending physicians (6.5–1.2 per 100 orders). Two patient safety events related to alert refinements were reported during the project period.

CONCLUSIONS:

Our quality improvement effort refined 47% of all DDI alerts that were firing during historical analysis, significantly reduced the number of DDI alerts in a 54-week period, and established a model for sustained alert refinements.

Electronic health records (EHRs) integrate clinical decision support (CDS) within computerized prescriber order entry (CPOE) systems to provide clinicians with intelligently filtered, person-specific information at appropriate times to improve health care delivery.1 Active interruptive CDS presents unsolicited information and requires a clinician’s response to continue.2 Excessive interruptive CDS alerts can lead to alert fatigue, possibly resulting in clinicians ignoring clinically relevant alerts.3,4 Alert fatigue can also be affected by inferior interface design and limitations in knowledge bases.5,6 

To reduce the potential for alert fatigue, consensus groups recommend reducing the number of ineffective alerts by analyzing alert metrics and perceived satisfaction of alerts.7,8 Current alert metrics that are easily retrievable from the EHR provide limited insight into the clinician’s perspective.9 Alert frequency (the number of times an alert is presented during a given period) cannot differentiate between clinically appropriate and inappropriate alert presentations. Alert override rate (the number of continued actions that generated an alert divided by the total number of alerts) contains both justified and unjustified overrides.7 Determining if an override is justified or unjustified requires detailed analysis of alert and patient data.9,10 Recently, a drug-drug interaction (DDI) CDS workgroup recommended combining clinician feedback and perceptions of alert systems with current alert metrics to target alerts for deactivation and tracking the performance of alert system improvements.7 

One mechanism for alert system improvements is the guided review of an interdisciplinary clinician panel,11,16 which can evaluate DDI alert frequencies both individually11,15 and by drug class.12,14 Organizing DDI alerts into drug-class and class-class categories allows for a large number of individual DDI alerts to be evaluated for clinical effectiveness.12,14 DDI alerts that are identified as having little clinical value can be completely suppressed or selectively filtered according to patient-specific factors by using contextual awareness.12 One analysis of hospital-wide DDI alerts revealed that 25% of alerts could be improved with contextually aware filtering.17 

Although systematically evaluating DDI alert satisfaction is recommended, only 1 known survey instrument has been developed and evaluated psychometrically.18 The developers of this tool recommended that it be used as part of comprehensive efforts to assess clinicians, including alert data and metrics.18 However, we did not find published studies in which this tool was used to guide DDI alert improvement efforts.

In this quality improvement (QI) report, we describe amultidimensional approach to improve DDI alert effectiveness at St. Jude Children’s Research Hospital (St. Jude), with the goal of reducing the frequency of DDI alerts per 100 medication orders by 20% in 1 year. Our efforts comprised evaluation by an interdisciplinary advisory group, alert metric analysis, and assessment of clinician perceptions of DDI alert value with a validated survey.

St. Jude is a 78-bed hospital with integrated outpatient clinics and treats children with cancer, blood disorders, and related life-threatening diseases. Since 2010, St. Jude has used a fully implemented EHR system with CPOE (Millennium; Cerner Corporation, North Kansas City, MO) for all aspects of inpatient and outpatient care.19 

The EHR system primarily generates DDI alerts using a commercial knowledge base (Cerner Multum, Denver, CO). During initial CPOE implementation in 2008, alert fatigue was considered, and only “major” and “major contraindicated” DDI alerts were presented to clinicians. Duplicate therapy alerts were not presented. When the EHR system detects a DDI, a pop-up window interrupts clinician workflow. To proceed, it must be acknowledged through an override (which requires an override reason) or acceptance (ie, removing the offending order; Supplemental Fig 3). With oversight from the Pharmacy and Therapeutics (P&T) Committee, the DDI alert database has been intermittently modified according to clinician recommendations, review of alert data, and literature reviews. All medication orders entered by a midlevel practitioner (eg, nurse practitioner [NP]) require an attending cosignature, and alerts are generated at order entry and at cosignature.

The alert advisory group (AAG), which was established to provide oversight and guidance, met routinely to guide, improve, and review project results (Supplemental Information). To align with St. Jude’s quality and safety structure, the AAG reported routinely to the P&T Committee, which authorized the AAG to implement refinements and report them retrospectively (Supplemental Information).

The AAG identified new alert-refinement processes and opportunities over time. Monthly DDI alert database updates received from the vendor were evaluated for clinical value, especially to determine if alerts should be presented to clinicians. Also, the value of DDI alert presentation for each clinician role (eg, prescribing, dispensing and/or verifying, and attending cosignature) was evaluated.

DDI alert refinements that were identified during the project included both suppression (ie, complete deactivation of an alert) and contextual-awareness filtering of alerts (ie, presenting alerts to clinicians that were based on patient-specific data within the EHR).8 Throughout the project, input was sought from clinicians about additional alerts for potential refinement. Refinement strategies were operationalized with custom alerts and tools and filters from our EHR vendor.

The administration of a validated satisfaction survey to clinicians who encountered interruptive DDI alerts provided a source of DDI alert refinements.18 Survey data were collected in the following 2 formats: in-person at departmental meetings and e-mail invitation. In both formats, project leaders engaged with respondents by explaining the project’s purpose and encouraging clinicians to send alert system–improvement recommendations.

To facilitate the analysis of a large number of DDI alerts, DDI alert medications were classified (when applicable) into class-class and drug-class categories of DDI alerts. The frequency of interruptive medication CDS alerts received by clinicians was determined by calculating the number of alerts per 100 medication orders. The override rate for DDI alerts was calculated by dividing the number of alerts that were overridden by the total number of alerts that were fired. DDI alerts were the most frequent type of interruptive medication alerts and selected as the initial focus of alert refinements.

As a balancing measure to detect any unintended consequences from alert refinements, we tracked patient safety event reports related to DDIs during the project. Clinicians were also encouraged to report any unexpected outcomes of alert modifications.

The average alert frequency and alert override rate were calculated for each class-class and drug-class category, ranked, and evaluated for possible CDS refinements. Weekly DDI alert frequency was analyzed by using a statistical process control (SPC) chart (u-chart) that was created with SPC for Excel (version 5.0.1.6; SPC for Excel, Cypress, TX). Weekly DDI alert frequency was measured by dividing the number of DDI alerts per week by the number of 100 medication orders each week to generate an alert per 100 orders metric. Upper and lower control limits for the u-chart were calculated by using prerefinement data to detect a significant change in frequency due to refinement efforts or other special causes. The average center line was shifted when a special cause was detected after initiating improvement efforts. The software evaluated the following standard rules to identify special cause: 1 point beyond the control limits, 2 out of 3 consecutive points in the outer one-third section and within control limits, 4 out of 5 consecutive points in the middle one-third section and within control limits, ≥7 consecutive points on 1 side of the average, ≥7 consecutive points trending up or down, 8 consecutive points that are not contained within the inner one-third section and are within control limits, 15 consecutive points that are contained in the inner one-third section and within control limits, and 14 consecutive points alternating up and down.

The percentage of DDI alerts due to the requirement of an attending cosignature was calculated by dividing the number of DDI alerts that were received by midlevel practitioners and would require an attending cosignature by the total number of alerts.

The total number of patient safety event reports that were related to DDI alerts and any unexpected outcomes of alert modifications were calculated.

This project was reviewed by the St. Jude Institutional Review Board and found to be a QI project.

To identify alerts for potential refinement, the AAG analyzed the frequency of alerts from June 2, 2012, to June 2, 2015, and decided to target DDIs for refinement (Supplemental Fig 4). We sorted the 106 528 DDI alerts into 282 class-class or drug-class categories. Then, we selected the top 25 class-class or drug-class categories with high alert frequency and override rates for potential refinement. During the project, the average override rate for alerts was 93.9%. In Table 1 and Fig 1, we present alert-refinement strategies that were used for specific DDI alert categories and the refinement date. We implemented a total of 26 alert refinements during the project period, which represents 46.8% of all alerts that fired during the analysis period (Table 1). We used alert suppression for 16.5% of all DDI alerts that fired and introduced contextual awareness to refine 30.2% of all DDI alerts that fired during the time frame.

TABLE 1

DDI Alert Refinements by the AAG During February 2015–February 2017

CategoryFrequency, %Override Rate, %Refinement DateAlert Refinement StrategyAAG Clinical Rationale for Alert Refinement
Potassium and potassium-sparing diuretics 9.2 95.9 December 14, 2015 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 
Potassium ACEI and/or ARB 6.7 95.8 December 14, 2015 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 
Posaconazole and H2 receptor blocker or PPI 4.1 95.0 December 14, 2015 Alert only fires when posaconazole solution is ordered. Absorption affected by gastric pH only for oral solution dosage form of posaconazole. 
Potassium and anticholinergics 2.2 94.4 December 14, 2015 Alert only fires when anticholinergic is ordered with extended-release potassium. Risk of gastric damage due to delayed gastric emptying of anticholinergics only for extended-release dosage forms of potassium. 
Fluoroquinolones and steroids 1.2 95.9 December 14, 2015 Suppress This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level. 
Iohexol (intrathecal only) and lidocaine 1.1 90.8 December 14, 2015 Suppress (suppression of all Iohexol intrathecal DDI alert interactions that occurred in July 2016) Iohexol is not administered via the intrathecal route in our patient population. 
Methotrexate and P-glycoprotein inhibitors 0.7 98.3 December 14, 2015 Suppress Methotrexate doses are monitored and adjusted via pharmacokinetic monitoring. 
Diuretics and aminoglycosides 0.7 92.9 December 14, 2015 Suppress Benefits of both medications outweigh risk of ototoxicity and nephrotoxicity. Patients are evaluated for both toxicity risks during evaluation before ordering. 
NSAID and NSAID 3.4 90.1 March 9, 2016 Alert only fires for 2 inpatient NSAID orders. Alert fired inappropriately for outpatient NSAID orders with inpatient ones. Alert only relevant when 2 active inpatient NSAID orders are present. 
CNS depressant and CNS depressant 0.5 89.0 March 9, 2016 Suppress Potential increased risk of CNS depression monitored closely in our patient population. 
Methotrexate and PPI 3.6 90.3 August 26, 2016 Alert only fires with high-dose methotrexate (doses >300 mg/m2). Elimination of methotrexate at high doses is impaired by PPIs. 
5HT3 antagonist and serotonin modulator 6.3 95.3 September 12, 2016 Suppress Majority of patient population receives serotonin antagonist. Low risk of serotonin syndrome that will be detected from inpatient and frequent outpatient clinic monitoring. 
Midazolam and CYP3A4 inhibitor 1.6 94.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Fentanyl and CYP3A4 inhibitor 0.4 97.8 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Methadone and CYP3A4 inhibitor 0.3 94.3 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Oxycodone and CYP3A4 inhibitor 0.3 92.5 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Citalopram and CYP3A4 inhibitor 0.1 98.6 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Salmeterol and CYP3A4 inhibitor 0.1 97.3 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Escitalopram and CYP3A4 inhibitor 0.1 95.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Itraconazole and CYP3A4 inhibitor 0.1 95.9 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Oxycodone and CYP3A4 inhibitor 0.1 91.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Fluticasone and CYP3A4 inhibitor 0.1 94.8 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Olanzapine and benzodiazepines 1.5 96 November 21, 2016 Suppress DDI occurs only with intramuscular formulations of both drugs. Olanzapine is not orderable via intramuscular route at our institution. 
Deferasirox and NSAIDs 1.3 97 November 21, 2016 Suppress for all NSAIDs (previously on suppressed for ibuprofen). This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level. 
Tacrolimus and spironolactone 97 November 21, 2016 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 
CategoryFrequency, %Override Rate, %Refinement DateAlert Refinement StrategyAAG Clinical Rationale for Alert Refinement
Potassium and potassium-sparing diuretics 9.2 95.9 December 14, 2015 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 
Potassium ACEI and/or ARB 6.7 95.8 December 14, 2015 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 
Posaconazole and H2 receptor blocker or PPI 4.1 95.0 December 14, 2015 Alert only fires when posaconazole solution is ordered. Absorption affected by gastric pH only for oral solution dosage form of posaconazole. 
Potassium and anticholinergics 2.2 94.4 December 14, 2015 Alert only fires when anticholinergic is ordered with extended-release potassium. Risk of gastric damage due to delayed gastric emptying of anticholinergics only for extended-release dosage forms of potassium. 
Fluoroquinolones and steroids 1.2 95.9 December 14, 2015 Suppress This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level. 
Iohexol (intrathecal only) and lidocaine 1.1 90.8 December 14, 2015 Suppress (suppression of all Iohexol intrathecal DDI alert interactions that occurred in July 2016) Iohexol is not administered via the intrathecal route in our patient population. 
Methotrexate and P-glycoprotein inhibitors 0.7 98.3 December 14, 2015 Suppress Methotrexate doses are monitored and adjusted via pharmacokinetic monitoring. 
Diuretics and aminoglycosides 0.7 92.9 December 14, 2015 Suppress Benefits of both medications outweigh risk of ototoxicity and nephrotoxicity. Patients are evaluated for both toxicity risks during evaluation before ordering. 
NSAID and NSAID 3.4 90.1 March 9, 2016 Alert only fires for 2 inpatient NSAID orders. Alert fired inappropriately for outpatient NSAID orders with inpatient ones. Alert only relevant when 2 active inpatient NSAID orders are present. 
CNS depressant and CNS depressant 0.5 89.0 March 9, 2016 Suppress Potential increased risk of CNS depression monitored closely in our patient population. 
Methotrexate and PPI 3.6 90.3 August 26, 2016 Alert only fires with high-dose methotrexate (doses >300 mg/m2). Elimination of methotrexate at high doses is impaired by PPIs. 
5HT3 antagonist and serotonin modulator 6.3 95.3 September 12, 2016 Suppress Majority of patient population receives serotonin antagonist. Low risk of serotonin syndrome that will be detected from inpatient and frequent outpatient clinic monitoring. 
Midazolam and CYP3A4 inhibitor 1.6 94.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Fentanyl and CYP3A4 inhibitor 0.4 97.8 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Methadone and CYP3A4 inhibitor 0.3 94.3 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Oxycodone and CYP3A4 inhibitor 0.3 92.5 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Citalopram and CYP3A4 inhibitor 0.1 98.6 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Salmeterol and CYP3A4 inhibitor 0.1 97.3 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Escitalopram and CYP3A4 inhibitor 0.1 95.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Itraconazole and CYP3A4 inhibitor 0.1 95.9 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Oxycodone and CYP3A4 inhibitor 0.1 91.4 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Fluticasone and CYP3A4 inhibitor 0.1 94.8 September 12, 2016 Suppress on the basis of analysis of all CYP3A4 substrates and inhibitor interactions. CYP3A4 inhibition clinically insignificant for this medication for our patient population. 
Olanzapine and benzodiazepines 1.5 96 November 21, 2016 Suppress DDI occurs only with intramuscular formulations of both drugs. Olanzapine is not orderable via intramuscular route at our institution. 
Deferasirox and NSAIDs 1.3 97 November 21, 2016 Suppress for all NSAIDs (previously on suppressed for ibuprofen). This extended a previous suppression of an individual DDI alert before the AAG formation to the class-class level. 
Tacrolimus and spironolactone 97 November 21, 2016 Alert only when previous potassium level was >4.8 mEq/L or when no potassium level has been obtained in 7 d. Serum potassium monitored routinely. Risk of unnoticed hyperkalemia present when patient is not monitored or has high serum potassium at initiation. 

None of the refinements above were detailed as DDIs that should always be active in pediatric EHRs.20 ACEI, angiotensin-converting enzyme inhibitor; ARB, angiotensin II receptor blocker; CNS, central nervous system; CYP3A4, cytochrome P450 3A4; PPI, proton pump inhibitor.

FIGURE 1

SPC u-chart of DDI alerts per 100 medication orders per week. Dates of AAG alert refinements are indicated by arrows, and individual refinements are detailed in Table 1. Upper control limit and lower control limit were determined by using weekly data before AAG refinement efforts (from February 1, 2015, to December 20, 2015). 5SL, 5 σ limit; −5SL, negative 5 σ limit.

FIGURE 1

SPC u-chart of DDI alerts per 100 medication orders per week. Dates of AAG alert refinements are indicated by arrows, and individual refinements are detailed in Table 1. Upper control limit and lower control limit were determined by using weekly data before AAG refinement efforts (from February 1, 2015, to December 20, 2015). 5SL, 5 σ limit; −5SL, negative 5 σ limit.

Close modal

After multiple alert refinements from December 2015 to September 2016, the SPC u-chart revealed a significant (40%) decrease from 22.9 to 13.7 DDI alerts per 100 medication orders per week (Fig 1). This decrease was sustained for 4 months after the end of the QI project (Fig 1). The override rate also decreased significantly during this time period (Supplemental Fig 5). During the project period, DDI alerts to clinicians at cosignature represented 69% of all DDI alerts encountered by physicians. The AAG approved the suppression of DDI alerts presented to physicians at cosignature, and the refinement was implemented in August 2016. We observed the following significant decreases in DDI alert frequency for each clinician role over the project period: 29% for pharmacists, 21% for NP and/or physician assistant, and 82% for attending physicians (Fig 2).

FIGURE 2

Run chart used to compare DDI alert frequency by clinician role. Averages were compared before and after the final round of refinements for each role. Pharmacists decreased from 10.4 to 7.4 DDI alerts per 100 orders. NPs and physician assistants decreased from 3.4 to 2.7 DDI alerts per 100 orders. Attending physicians decreased from 6.5 to 1.2 DDI alerts per 100 orders. These decreases were statistically significant for each clinician role.

FIGURE 2

Run chart used to compare DDI alert frequency by clinician role. Averages were compared before and after the final round of refinements for each role. Pharmacists decreased from 10.4 to 7.4 DDI alerts per 100 orders. NPs and physician assistants decreased from 3.4 to 2.7 DDI alerts per 100 orders. Attending physicians decreased from 6.5 to 1.2 DDI alerts per 100 orders. These decreases were statistically significant for each clinician role.

Close modal

We monitored automatic vendor-provided updates to the DDI alert content database, which proved to be an important intervention. An update on May 18, 2016, led to an increase in DDI alert severity for a commonly used drug (sulfamethoxazole-trimethoprim) in our patient population and in multiple drugs and drug classes that can increase serum potassium. We found a significant increase in alerts to clinicians that resulted from this May 2016 update (Fig 1). The AAG did not agree with the knowledge base vendor’s rationale for increasing alert severity for St. Jude patients, and these alerts were suppressed by early August 2016.

During the project, 2 safety events related to DDI alert refinements were reported to the hospital’s event reporting system. First, pantoprazole was ordered and administered to a patient who was receiving high-dose methotrexate. The alert had been refined and functioned properly, but the alert text was ignored and overridden. We revised the alert text to more clearly explain that coadministration of these medications is contraindicated. Second, oral potassium was administered to a patient who was receiving olanzapine, which prompted an event report. Using a vendor tool, we refined this alert to only present with relevant potassium dosage forms, but the tool did not function. Thus, we created a new custom alert to provide an alert that would include all relevant dose forms of potassium. Finally, 1 event was informally communicated after the AAG approved suppressing DDI alerts in the vendor-supplied DDI database that fired when prescribing 2 nonsteroidal anti-inflammatory drugs (NSAIDs). It was recommended to create a new custom alert after observing a patient with active inpatient orders for 2 NSAIDs. Thus, we implemented a custom alert that interrupts clinicians when an inpatient NSAID order is placed for a patient with an existing active NSAID order.

Our QI efforts reduced the weekly number of interruptive DDI alerts per 100 medication orders by 40% by both suppressing and modifying alerts, which exceeded our original goal of a 20% reduction in 1 year. We identified potential low-value alerts through alert analysis and interaction with clinicians when administering the satisfaction survey. We used these best practices to improve our hospital’s DDI alert system, and our project was also the first known effort to employ a construct-validated clinician survey on DDI alert quality.18 To our knowledge, our QI effort is the first to demonstrate significant reductions in DDI alert frequency by combining the following 3 unique methods: systematic alert data extraction and review, a multidisciplinary panel, and use of a validated survey instrument.

Alert fatigue can negate the potential patient safety benefits of interruptive CDS, and it is a widely discussed concern.3,4,7,9,10,14,21,22 Previous research has reported effective techniques for categorizing historical interruptive DDI alert data for critical review, organizing at the drug or drug category level, and calculating frequencies of alert occurrence.11,16 Interdisciplinary panels can review an institution’s alert data and safely identify alerts to deactivate or modify.7,8 

Using a validated survey provided broad clinician insights and helped fulfill previous recommendations to use multiple sources of information to evaluate interruptive alert quality.7,9,23,24 Although the response rate was too low to provide a meaningful representation of clinician alert satisfaction, the act of administering the survey in-person during staff meetings was valuable. By explaining the project objectives face-to-face and engaging clinicians with the audience response devices, we generated support for the project and created an opportunity for end users to offer novel ideas to improve the alert system.

The need for increased use of contextually aware filtering has been discussed repeatedly in previous research, and our efforts addressed these calls by including this technique in 6 of the 17 modified alerts.7,9,14,25,27 As previously reported, classifying DDI alerts by drug class increased the efficiency of our efforts and identified more DDI alerts for refinement than in our previous efforts to analyze metrics for individual DDI alerts, and it identified that only 25 alert groups were responsible for 46% of alert firings.11,16,21 Refinement of these alert groups resulted in a significant decrease in the frequency of DDI alerts across multiple clinician roles (Fig 2). Importantly, our experience was consistent with previous efforts showing that refining a relatively small number of drug-drug and drug-class alerts can encompass a large percentage of all occurring DDI alerts.12 

Limitations of our project included our balancing measure, the survey response rates, and applicability to other institutions. Although voluntary event reporting is used in similar efforts, it is not able to reliably and comprehensively capture all harm from medications, including DDIs.15,28,29 The low response rates were due to poor response rates from the online response collection efforts. Future efforts to collect online responses should involve a broad awareness campaign and the use of multiple recruitment tactics, such as printed promotional materials, announcements during staff meetings, e-mailed invites from senior leadership, and possibly incentives for participation. Importantly, the engagement generated from discussing the project, the AAG, and administering the survey during these meetings was valuable in generating support and eliciting ideas to improve the alert system. Our comprehensive approach to DDI alert management is resource intensive and may not be replicable in institutions with more limited resources to devote to the project. To help others apply what we have learned, our AAG charter and guidance for replication and adaptation is provided in the supplement (Supplemental Information). Key ingredients to our success that may be replicated with fewer resources include development of a close partnership between clinical leaders and information services and the use of SPC to interpret alert data.

Another limitation (which is broader than our project) is the lack of alert metrics to easily identify potentially inappropriate DDI alerts.23,24,30 Although the frequency and override rate can be easily calculated, their ability to reveal alert appropriateness without further clinician review is poor.9,10 Alert frequency and override rate do not provide insight into an alert’s clinical use because clinicians may still value some alerts that occur frequently but are still overridden. Also, override rates can be falsely high in instances in which it is easier to override the DDI alert for a new medication and immediately discontinue the old medication than to acknowledge the alert, discontinue the old medication, and then reenter the new medication. Override rates and reasons for overriding can also vary across departments within a single institution and across alert types.22,31,32 Metrics that can provide more information on appropriateness and value, such as alert adherence rate,9,10 require increased clinician insight and cannot be easily obtained from an automated report. Other metrics have been recently introduced but need additional refinement.23,24 

We demonstrated that multiple alert-refinement methods within a single QI effort decreased DDI alert frequency. We created a sustainable model for future alert-refinement efforts at our hospital, which may be applied more broadly (Supplemental Information). Further research on easily obtainable alert metrics that provide a clear understanding of alert value and clinician responses will help to improve the efficiency of future alert-improvement efforts.

Dr Daniels designed the quality improvement effort, drafted the initial manuscript, organized and conducted alert advisory group meetings, and conducted primary analysis of the data; Dr Burlison designed the quality improvement effort, drafted the initial manuscript, and conducted primary analysis of the data; Dr Robertson designed the quality improvement effort and drafted the initial manuscript; Dr Hoffman designed the quality improvement effort, drafted the initial manuscript, and organized and conducted the alert advisory group meetings; Drs Sablauer, Campbell, and Baker contributed significantly to the design of the intervention and acquisition of data and critically reviewed and revised the manuscript; Dr Flynn contributed significantly to the development of the intervention and critically reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Supported by the Cancer Centers core grant NIH CA 21765 and the American Lebanese Syrian Associated Charities. Funded by the National Institutes of Health (NIH).

We thank all members of the AAG for their time and effort toward this QI effort to optimize alerts at St. Jude Children’s Research Hospital. We acknowledge Nick Keeling, MS, for help with data analysis and article review and Vani Shanker, PhD, for copyediting the article.

     
  • AAG

    alert advisory group

  •  
  • CDS

    clinical decision support

  •  
  • CPOE

    computerized prescriber order entry

  •  
  • DDI

    drug-drug interaction

  •  
  • EHR

    electronic health record

  •  
  • NP

    nurse practitioner

  •  
  • NSAID

    nonsteroidal anti-inflammatory drug

  •  
  • P&T

    Pharmacy and Therapeutics

  •  
  • QI

    quality improvement

  •  
  • SPC

    statistical process control

1
Osheroff
JA
,
Teich
JM
,
Middleton
B
,
Steen
EB
,
Wright
A
,
Detmer
DE
.
A roadmap for national action on clinical decision support [published correction appears in J Am Med Inform Assoc. 2007;14(3):389].
J Am Med Inform Assoc
.
2007
;
14
(
2
):
141
145
[PubMed]
2
Troiano
D
,
Jones
MA
,
Smith
AH
, et al;
American Society of Health-System Pharmacists
.
ASHP guidelines on the design of database-driven clinical decision support: strategic directions for drug database and electronic health records vendors.
Am J Health Syst Pharm
.
2015
;
72
(
17
):
1499
1505
[PubMed]
3
van der Sijs
H
,
Aarts
J
,
Vulto
A
,
Berg
M
.
Overriding of drug safety alerts in computerized physician order entry.
J Am Med Inform Assoc
.
2006
;
13
(
2
):
138
147
[PubMed]
4
Carspecken
CW
,
Sharek
PJ
,
Longhurst
C
,
Pageler
NM
.
A clinical case of electronic health record drug alert fatigue: consequences for patient outcome.
Pediatrics
.
2013
;
131
(
6
). Available at: www.pediatrics.org/cgi/content/full/131/6/e1970
[PubMed]
5
McEvoy
DS
,
Sittig
DF
,
Hickman
TT
, et al
.
Variation in high-priority drug-drug interaction alerts across institutions and electronic health records.
J Am Med Inform Assoc
.
2017
;
24
(
2
):
331
338
[PubMed]
6
Ancker
JS
,
Edwards
A
,
Nosal
S
,
Hauser
D
,
Mauer
E
,
Kaushal
R
;
HITEC Investigators
.
Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system.
BMC Med Inform Decis Mak
.
2017
;
17
(
1
):
36
[PubMed]
7
Payne
TH
,
Hines
LE
,
Chan
RC
, et al
.
Recommendations to improve the usability of drug-drug interaction clinical decision support alerts.
J Am Med Inform Assoc
.
2015
;
22
(
6
):
1243
1250
[PubMed]
8
Tilson
H
,
Hines
LE
,
McEvoy
G
, et al
.
Recommendations for selecting drug-drug interactions for clinical decision support.
Am J Health Syst Pharm
.
2016
;
73
(
8
):
576
585
[PubMed]
9
McCoy
AB
,
Thomas
EJ
,
Krousel-Wood
M
,
Sittig
DF
.
Clinical decision support alert appropriateness: a review and proposal for improvement.
Ochsner J
.
2014
;
14
(
2
):
195
202
[PubMed]
10
McCoy
AB
,
Waitman
LR
,
Lewis
JB
, et al
.
A framework for evaluating the appropriateness of clinical decision support alerts and responses.
J Am Med Inform Assoc
.
2012
;
19
(
3
):
346
352
[PubMed]
11
Cornu
P
,
Steurbaut
S
,
Gentens
K
,
Van de Velde
R
,
Dupont
AG
.
Pilot evaluation of an optimized context-specific drug-drug interaction alerting system: a controlled pre-post study.
Int J Med Inform
.
2015
;
84
(
9
):
617
629
[PubMed]
12
Helmons
PJ
,
Suijkerbuijk
BO
,
Nannan Panday
PV
,
Kosterink
JG
.
Drug-drug interaction checking assisted by clinical decision support: a return on investment analysis.
J Am Med Inform Assoc
.
2015
;
22
(
4
):
764
772
[PubMed]
13
Parke
C
,
Santiago
E
,
Zussy
B
,
Klipa
D
.
Reduction of clinical support warnings through recategorization of severity levels.
Am J Health Syst Pharm
.
2015
;
72
(
2
):
144
148
[PubMed]
14
Phansalkar
S
,
van der Sijs
H
,
Tucker
AD
, et al
.
Drug-drug interactions that should be non-interruptive in order to reduce alert fatigue in electronic health records.
J Am Med Inform Assoc
.
2013
;
20
(
3
):
489
493
[PubMed]
15
Simpao
AF
,
Ahumada
LM
,
Desai
BR
, et al
.
Optimization of drug-drug interaction alert rules in a pediatric hospital’s electronic health record system using a visual analytics dashboard.
J Am Med Inform Assoc
.
2015
;
22
(
2
):
361
369
[PubMed]
16
Zenziper
Y
,
Kurnik
D
,
Markovits
N
, et al
.
Implementation of a clinical decision support system for computerized drug prescription entries in a large tertiary care hospital.
Isr Med Assoc J
.
2014
;
16
(
5
):
289
294
[PubMed]
17
Seidling
HM
,
Storch
CH
,
Bertsche
T
, et al
.
Successful strategy to improve the specificity of electronic statin-drug interaction alerts.
Eur J Clin Pharmacol
.
2009
;
65
(
11
):
1149
1157
[PubMed]
18
Zheng
K
,
Fear
K
,
Chaffee
BW
, et al
.
Development and validation of a survey instrument for assessing prescribers’ perception of computerized drug-drug interaction alerts.
J Am Med Inform Assoc
.
2011
;
18
(
suppl 1
):
i51
i61
[PubMed]
19
Hoffman
JM
,
Baker
DK
,
Howard
SC
,
Laver
JH
,
Shenep
JL
.
Safe and successful implementation of CPOE for chemotherapy at a children’s cancer center.
J Natl Compr Canc Netw
.
2011
;
9
(
suppl 3
):
S36
S50
[PubMed]
20
Harper
MB
,
Longhurst
CA
,
McGuire
TL
,
Tarrago
R
,
Desai
BR
,
Patterson
A
;
Children’s Hospital Association CDS Working Group
.
Core drug-drug interaction alerts for inclusion in pediatric electronic health records with computerized prescriber order entry.
J Patient Saf
.
2014
;
10
(
1
):
59
63
[PubMed]
21
Slight
SP
,
Seger
DL
,
Nanji
KC
, et al
.
Are we heeding the warning signs? Examining providers’ overrides of computerized drug-drug interaction alerts in primary care.
PLoS One
.
2013
;
8
(
12
):
e85071
[PubMed]
22
Nanji
KC
,
Seger
DL
,
Slight
SP
, et al
.
Medication-related clinical decision support alert overrides in inpatients.
J Am Med Inform Assoc
.
2018
;
25
(
5
):
476
481
[PubMed]
23
McDaniel
RB
,
Burlison
JD
,
Baker
DK
, et al
.
Alert dwell time: introduction of a measure to evaluate interruptive clinical decision support alerts.
J Am Med Inform Assoc
.
2016
;
23
(
e1
):
e138
e141
[PubMed]
24
Schreiber
R
,
Gregoire
JA
,
Shaha
JE
,
Shaha
SH
.
Think time: a novel approach to analysis of clinicians’ behavior after reduction of drug-drug interaction alerts.
Int J Med Inform
.
2017
;
97
:
59
67
[PubMed]
25
Horsky
J
,
Phansalkar
S
,
Desai
A
,
Bell
D
,
Middleton
B
.
Design of decision support interventions for medication prescribing.
Int J Med Inform
.
2013
;
82
(
6
):
492
503
[PubMed]
26
Jung
M
,
Riedmann
D
,
Hackl
WO
, et al
.
Physicians’ perceptions on the usefulness of contextual information for prioritizing and presenting alerts in computerized physician order entry systems.
BMC Med Inform Decis Mak
.
2012
;
12
(
1
):
111
[PubMed]
27
Phansalkar
S
,
Desai
A
,
Choksi
A
, et al
.
Criteria for assessing high-priority drug-drug interactions for clinical decision support in electronic health records.
BMC Med Inform Decis Mak
.
2013
;
13
(
1
):
65
[PubMed]
28
Cullen
DJ
,
Bates
DW
,
Small
SD
,
Cooper
JB
,
Nemeskal
AR
,
Leape
LL
.
The incident reporting system does not detect adverse drug events: a problem for quality improvement.
Jt Comm J Qual Improv
.
1995
;
21
(
10
):
541
548
[PubMed]
29
Meyer-Massetti
C
,
Cheng
CM
,
Schwappach
DL
, et al
.
Systematic review of medication safety assessment methods.
Am J Health Syst Pharm
.
2011
;
68
(
3
):
227
240
[PubMed]
30
Dexheimer
JW
,
Kirkendall
ES
,
Kouril
M
, et al
.
The effects of medication alerts on prescriber response in a pediatric hospital.
Appl Clin Inform
.
2017
;
8
(
2
):
491
501
[PubMed]
31
Ahn
EK
,
Cho
SY
,
Shin
D
,
Jang
C
,
Park
RW
.
Differences of reasons for alert overrides on contraindicated co-prescriptions by admitting department.
Healthc Inform Res
.
2014
;
20
(
4
):
280
287
[PubMed]
32
Ahn
EK
,
Kam
HJ
,
Park
DK
,
Jung
EY
,
Lee
Y
,
Park
RW
.
Differences among admitting departments in alerts and alert overrides for drug-drug interaction.
Pharmacoepidemiol Drug Saf
.
2014
;
23
(
4
):
390
397
[PubMed]

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: Dr Flynn is a consultant for Merck & Co; the other authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data