BARRIERS to Early Detection of Deterioration in Hospitalized Infants Using Predictive Analytics

The Institute of Medicine report entitled “Digital Infrastructure for the Learning Health System” set a goal for 90% of clinical decisions to be supported by digital clinical information by 2020.¹  This goal has not explicitly been achieved, in large part because of the lack of integration and translation from digital analytics to clinical practice.²  Despite progress in research and development of predictive analytics and artificial intelligence in health care, numerous challenges remain. These obstacles hinder the integration, adoption, and clinical application of advanced statistical models as digital infrastructure for early warning of illness, especially within the pediatric context.

Few populations have as much potential for benefit from early detection and treatment of illnesses as hospitalized infants (Table 1, Babies). Unrecognized clinical deterioration can have dire, lifelong consequences for children who suffer long-term adverse neurologic and functional outcomes.^3,4  Furthermore, our youngest patients cannot voice subjective complaints or call the attention of their medical team. Several algorithms have been developed for early warning of sudden deterioration in hospitalized infants and children.^5–10  One approach to help clinicians recognize physiologic trends before decompensation has been to use a track-and-trigger score, such as the Pediatric Early Warning Score (PEWS). Several versions of PEWS have been developed and validated in multicenter studies, but many use subjective assessments as parameters, making reliability difficult. Other challenges include the variability in standard use (Table 1, Reactors) and the lack of integration within existing electronic health records.¹¹  When de Vries and colleagues¹²  asked clinicians about their perceptions of PEWS, they responded that PEWS did enhance situational awareness, but they suggested additional risk stratification measures would make the response to a high PEWS score more valuable. The many interconnected variables driving clinical actions and the rare incidence of measurable outcomes, such as mortality, make it difficult to reveal the impact of scores such as PEWS in clinical trials.¹³ 

Another approach to early warning technology is in predictive analytics, which uses high dimensional data and multivariable or machine learning models to detect physiologic patterns that indicate a rising risk of clinical deterioration before warning signs that might not yet be clinically apparent.^14,15  Machine learning and complex mathematical models face challenges with implementation and adoption. Whether by modeling design or user perception, such algorithms may be viewed as a “black box.” The lack of understanding in what inputs changed a model’s output causes uncertainty for users in what the risk score means for the individual patient and how they should appropriately intervene. Metrics that typically measure model performance, such as the area under the receiver operating characteristic curve, or AUC, do not translate well to clinically meaningful parameters (Table 1, Analytics).¹⁶  Model performance measures with more intuitive clinical applications, such as positive predictive value and number needed to treat, require thresholds rather than continuous trends.⁸  Thresholds are prone to errors of inclusion or exclusion by nature. On the one hand, a low risk estimate from a predictive score can reassure the medical team and promote de-escalation or watchful waiting. On the other hand, waiting for the predicted risk to rise above a threshold could diminish vigilance in clinical monitoring and lead to delayed action.

Often, early warning scores and predictive analytics set thresholds to translate the score into a risk category or for decision support to initiate a clinical action. Therefore, those involved with the development and implementation of a novel analytic must understand the implications of setting thresholds for action. The challenge with this approach is twofold: (1) sudden deterioration in infants and children is a rare event, so false alarm rates are high, and (2) the approach relies on alerts for reaction instead of monitoring the continuous trend of the analytic.

Despite the acceleration of predictive analytic development, attention to successful implementation and long-term adoption of analytics among bedside clinicians remains lacking.¹⁷  For successful adoption of a novel analytic, clinicians must integrate the process of interpreting, documenting, and acting on the information it provides with the rest of the clinical data in a way that is synergistic with their already complex workflows.¹⁴  Paradoxically, the alert systems created to enhance patient safety can compromise patient safety if alarm fatigue develops from nonactionable alarms (Table 1, Integration).¹⁵  Guidelines for responding to predictive analytics monitoring must (1) be embedded within workflows, (2) be actionable, and (3) sustain user buy-in through clinician engagement.

Long-term adoption also relies on education of clinicians. Interpretation of early warning and predictive analytic risk scores can challenge novice and expert clinicians alike. Often, considerable effort goes into education on implementation, but ongoing education should be equally robust (Table 1, Reeducation). Users with varying experience, education, and clinical roles might react to the same analytic in different ways. For example, bedside nurses will follow the trends of a few patients closely, whereas attending physicians must monitor the entire unit. Predictive analytics monitoring must be intuitive for all users and offer decision support while not supplanting clinical judgment.¹⁸  Providers are more likely to communicate a change in risk score if they trust and understand the system.¹⁵  Predictive analytics and risk scores require a less intuitive response than an abnormal vital sign alarm. For example, when a low heart rate alarm fires, most clinicians react automatically. In contrast, a rising risk score warns of a future state, requiring a proactive response. Initial and ongoing education plus common clinical pathways can help reinforce key concepts for clinicians.¹⁴  Leadership at all levels is necessary for successful integration, particularly among clinicians with the most experience to improve engagement from the system.

For novel analytics to impact patient outcomes, the built environment and the culture of the care environment must align.¹⁹  Current monitoring equipment displays, at most, a few moments of individual vital signs (Table 1, Equipment). Most warning systems reveal to clinicians a change from baseline by allowing the user to view compiled trends. The successful adoption of predictive analytics relies on the thoughtful integration of human and technical elements. The analytic or risk score must be readily visible, easily interpreted, and interoperable within the electronic medical record (EMR) to be adopted by users.¹⁴  Additionally, space is limited because monitors and equipment surround the patient’s bedside (Table 1, Space). The EMR is dense with information. Adding an analytic that requires visual attention can compete for space physically or mentally.²⁰  Whether it is displayed, for example, on a central monitor, within the EMR, or on a separate screen at the bedside should be thoughtfully determined to encourage optimal use without additional burden. Implementation approaches using user-centered design and ergonomics can promote uptake and minimize negative impacts to clinician users.

Hospitalized infants who deteriorate because of illnesses such as sepsis often have insidious and nonspecific early signs. Early warning scores and predictive analytics have potential to improve patient outcomes. Researchers develop complex risk models from large data sets with the hope of warning the medical team of an imminent deterioration in an individual patient. Although promising in theory, in practice, these tools face many obstacles. Successful adoption of predictive analytics in newborn and pediatric units requires addressing the inherent BARRIERS (Table 1) to sustained use and desired outcomes. We recommend working with key stakeholders to create a standardized clinical response to a rising risk score: one that starts with a thorough assessment of the patient. Novel analytics do not replace clinician judgment but can bring the clinician to the right patient at the right time.