Adapting a Risk Prediction Tool for Neonatal Opioid Withdrawal Syndrome

Neonatal opioid withdrawal syndrome (NOWS) affects nearly 1 in 100 births nationwide, resulting in longer and more complicated hospital stays.^1,2 Infants diagnosed with NOWS spend, on average, 14 more days in the hospital compared with infants without NOWS.^1–3 Despite this, there are few tools to guide early treatment of infants at risk for developing NOWS given that the severity is multifactorial and can be difficult to assess at birth.^4,5 

The American Academy of Pediatrics (AAP) recommends at least 3 days of observation for infants exposed to short-acting opioids and up to 7 days for infants exposed to long-acting opioids, buprenorphine, or methadone.⁴ However, characterizing opioid exposure can be difficult on screening, especially in the milieu of stigma and legal complications, potentiating exposure from tobacco and certain prescription medications, and variation of infant metabolism.⁴ Furthermore, many opioid-exposed infants could be safely discharged without extended hospital observation because they will never be diagnosed with NOWS or only have relatively mild clinical signs.⁶ The decision to observe, perform further tests, or initiate treatment occurs soon after birth and often depends on balancing the anticipated severity of NOWS with the burden of treatment.⁴ Failing to identify infants at risk or idle monitoring of infants at mistakenly low risk can delay interventions and assessments, such as breastfeeding and toxicology.^4,5,7 If we can accurately assess the likelihood of developing severe NOWS soon after birth, then we can tailor the intensity of interventions and anticipate the duration of in-hospital treatment.

Given the multifactorial nature of NOWS severity and the limitations of guideline screening that relies on characterizing opioid exposure, prognostic models have been developed.^8–11 In prior work, we developed a NOWS prediction model using state-wide administrative claims.⁹ Although this model had good performance, it was limited as a clinical tool because it primarily relied on diagnostic codes and did not use infant treatment data. This could hamper the ability to assess the severity of NOWS and can lead to model miscalibration.¹² Moreover, the model did not incorporate potentially important predictors available at the time of birth to assess risk, such as the Apgar score. Therefore, we aimed to develop a more robust clinical tool by adapting our prior model and externally validating the approach with electronic health record (EHR) data. ^8,9 Our objective was to adapt and validate a prognostic model that will help clinicians identify infants at the time of birth who are at risk for developing severe NOWS. We also sought to assess the benefit of the model relative to an EHR definition of chronic opioid exposure based on guideline screening criteria.

In this prognostic study, we used EHR data from Vanderbilt University Medical Center, a tertiary medical center in the Southeast United States. The cohort included births in the medical center between November 2017 and January 2024. We included all live infants with a gestational age of more than 33 weeks and excluded infants who were deceased on discharge or were transferred from another hospital. We extracted data for 33 991 infants and 29 001 mothers (mothers could have more than 1 birth). This study was approved by the Vanderbilt University Medical Center Institutional Review Board, and we used the TRIPOD reporting guidelines for prediction models (Supplemental Figure 1).¹³ 

We adapted a model to predict severe NOWS, which was defined as administration of oral morphine. Oral morphine is the standard pharmacotherapy for NOWS at our institution, and we identified administration within 7 days of birth.⁴ On chart review, we found that some infants who developed severe NOWS would be missed by limiting the cohort to those with high-risk exposure (ie, hepatitis C or medications for opioid use disorder [OUD]); therefore, we fit the model with the entire cohort of 33 991 births, which aligns with our prior general population model that used administrative data.⁹ By including the full cohort according to our inclusion criteria, the model can be used to screen infants among the general population and not just those known to be at high risk.¹⁰ To assess for oral morphine use with indications other than NOWS, we reviewed charts for 27 infants who received oral morphine without the previously validated International Classification of Diseases, Tenth Revision, Clinical Modification P96.1; all were found to have clinical diagnoses of NOWS.¹⁴ In 2022, our institution transitioned from the Finnegan Score to Eat, Sleep, Console to guide treatment. Throughout the study period, however, our institution has emphasized nonpharmacologic measures, prior to pharmacotherapy, with care outside the neonatal intensive care unit (NICU) and keeping the maternal-infant dyad intact.¹⁵ 

We identified relevant predictors a priori from our previous model and additional studies.^{4,8–10,12,16} In total, we extracted 37 candidate predictors for mothers and infants, including medications, infectious disease, social history, and physiologic measures at birth. As proxy indicators of illicitly manufactured fentanyl use, we included intravenous (IV) drug use and diagnosis of hepatitis C and OUD. Supplemental Table 1 provides candidate predictors and definitions. Race and insurance at delivery were used to characterize the cohort but excluded as predictors.^17,18 Chosen predictors were routinely collected, available at the time of birth, and amenable to automated extraction and processing in the EHR. We tested for redundancy by examining the variance in each candidate predictor that could be explained by other candidate predictors using flexible parametric additive models.¹⁹ Exposure to any prescription opioids, which grouped short- and long-acting opioid prescriptions together, was removed because of redundancy. We included, however, the individual predictors for short- and long-acting opioid prescriptions separated.

We summarized cohort characteristics descriptively using means and standard deviations for continuous variables and counts with percentages for categorical variables. To predict severe NOWS, we used logistic regression with a least absolute shrinkage and selection operator (LASSO) approach, which allowed us to avoid overfitting the model by using the most influential predictors. Predictors were standardized (scaled and centered) prior to running the LASSO regression. LASSO introduces a penalty term to the standard regression model, which forces some of the regression coefficients to shrink toward zero, effectively performing predictor selection.²⁰ We chose to use LASSO regression, rather than backward selection with our prior model, to better handle the high-dimensional EHR data with a relatively low number of NOWS observations.⁹ To design for clinical use, missing numeric measures for mother’s age as well as the infant’s gestational age, birth weight, and head circumference were to be imputed with the cohort median for when measures become available.²¹ 

We evaluated model discrimination and calibration using the area under the curve (AUC), Brier score, calibration slope and intercept, integrated calibration index (ICI), and visual examination of calibration curves.^22,23 The Brier score is an overall accuracy metric with both discrimination and calibration elements and is calculated by the squared difference between the prediction (0 to 1) and outcome (0 = no NOWS and 1 = NOWS).²² The ICI is a numeric summary of model calibration across the predicted probabilities.²³ It is the weighted average of the absolute difference between the observed and predicted probabilities; therefore, a lower ICI indicates better calibration. A slope greater than 1 indicates potential underfitting and a slope less than 1 indicates potential overfitting.²³ The model was validated using bootstrap cross-validation with replacement.²⁴ 

We conducted a decision curve analysis to assess the clinical utility of the model compared to an EHR definition of chronic opioid exposure based on the AAP-recommended screening criteria.^25–28 Aligning with the AAP guideline and our prior work defining opioid exposure, we defined chronic opioid exposure as the mother receiving a diagnosis of OUD at any time before delivery or receiving a prescription for long-acting opioids, buprenorphine, or methadone between 90 and 2 days before delivery.¹² A decision curve analysis can evaluate the consequences of diagnostic tests, biomarkers, and prognostic models by incorporating the benefits and harms associated with testing. ^25–28 The net benefit combines the number of true positives (eg, treating infants with NOWS) and false positives (eg, treating infants without NOWS) at different threshold probabilities representing the cutoff point for treatment. For instance, if the predicted probability of severe NOWS is 5% or higher, the cutoff point assumes that an infant should receive treatment (eg, prolong observation or administer oral morphine). That is, clinicians would prefer to treat 20 infants (who may or may not develop severe NOWS) to prevent severe NOWS complications in 1 infant. To assess the relative utility of our model compared to guideline criteria, we plotted the net benefit across a range of threshold probabilities. All analyses were performed in R version 4.3.3.

Of 33 991 births, the mean (SD) age was 29.5 (5.7) years for mothers and 37.9 (2.8) weeks for infants. Most infants were characterized as either white (64.9%) or Black (18.3%). A total of 1243 (4%) infants were considered to have chronic opioid exposure according to our EHR-based definition of the guideline criteria, and 108 (0.3%) infants were treated with oral morphine for NOWS (Table 1). Of the 108 infants, 18 (17%) did not meet the criteria for chronic opioid exposure and 4 (4%) were given oral morphine after the birth hospitalization.

Among 37 candidate predictors, the LASSO approach helped identify 7 that were necessary to accurately predict severe NOWS. Included predictors in order of decreasing strength were OUD, cigarette smoking, 5-minute Apgar score (0–10; lower score indicates more severe illness at 5 minutes of life), and prescriptions for selective serotonin reuptake inhibitor (SSRI), antinausea, long-acting opioid, and gabapentin medications (Figure 1). Among the strongest predictors, the association with severe NOWS was significant for OUD (adjusted odds ratio [aOR], 47.0; 95% CI, 26.7–82.7; P < .001), cigarette smoking (aOR, 2.8; 95% CI, 1.8–4.6; P < .001), 5-minute Apgar score (aOR, 0.82; 95% CI, 0.75–0.90; P < .001), and SSRI (aOR, 2.5; 95% CI, 1.5–4.3; P < .001). On validation of the model, we observed a high level of discrimination (AUC, 0.959; 95% CI, 0.940–0.976) and good calibration as measured by the ICI (0.001), Brier score (0.003), and inspection of the calibration curve (Figure 2).

We compared the AAP screening criteria to the prediction model with a decision curve analysis. Figure 3 shows the model provided a higher net benefit across all threshold probabilities, compared with screening with the criteria for chronic opioid exposure. At a risk threshold of 5%, the difference in net benefit suggests that the model compared to guideline criteria would identify 3 additional infants per 10 000 who develop severe NOWS. This would occur without resulting in unnecessary treatments, such as extended observation due to identifying false positive cases (Table 2). Moreover, compared to the guideline criteria, the model would reduce the number of false positives or unnecessary extended hospitalizations by 50%. For risk thresholds greater than 8%, the guideline criteria provided a negative net benefit primarily due to the increased number of false positives.

We adapted and validated a model to predict severe NOWS using EHR data that are routinely collected and available at the time of birth. The model incorporated multiple predictors beyond opioid exposure to accurately predict treatment of NOWS with pharmacotherapy. Rather than relying solely on guideline criteria for chronic opioid exposure, using multivariable prediction as a clinical tool may help clinicians identify exposed infants who are at low risk of severe NOWS for earlier discharge. Following clinical trials examining the NOWS prediction tool, this approach may guide extended observation periods and early nonpharmacologic measures for infants likely to develop severe NOWS and allow for discharge of infants who are unlikely to return to the hospital for pharmacologic treatment later.

We found that polysubstance exposure was strongly associated with NOWS. Similar to our prior model, we found exposure to cigarette smoking, SSRIs, antinausea medications, and gabapentin were important predictors of severe NOWS.⁹ We did not, however, find that exposure to short-acting opioids was an important predictor, which may reflect a decrease in opioid prescribing in the past decade.²⁹ Our model included OUD treatment with buprenorphine and methadone under “long-acting opioids,” with approximately 97% of mothers with this exposure being treated with buprenorphine.³⁰ Although cigarette smoking documentation in the social history section of the EHR can be inaccurate, we found it to be one of the strongest predictors.³¹ This study provides additional evidence that exposure to multiple substances, beyond opioids, may augment onset and severity of NOWS and should be considered in screening for in-hospital observation.

Few models to predict NOWS have been published and none have demonstrated the performance necessary for use as a clinical tool or have been implemented in routine patient care.^8–11 Isemann and colleagues developed a prediction tool for administration of pharmacotherapy within 36 hours from birth among 264 infants with in utero exposure to opioids.⁸ The tool incorporated type of opioid exposure and predictive signs from the Modified Finnegan Score, including increased muscle tone, tremors when disturbed, and excoriation. In addition to concerns with relatively low sensitivity (38%) associated with the tool, the Finnegan Score is subjective and may be decreasing in use as the Eat, Sleep, Console assessment is becoming more broadly adopted.^4,5 Our prior model used Medicaid data between 2009 and 2014 to predict diagnosis of NOWS during the birth hospitalization among 218 020 maternal-infant dyads, from which 3208 infants were diagnosed with NOWS.⁹ Although the discrimination was good (AUC, 0.89), calibration figures show that in infants with low and high risk levels, the model overestimated and underestimated risk, respectively. Furthermore, reliance on administrative data could limit the clinical utility, and use of data prior to 2014 may not reflect current opioid prescribing and widespread use of illicitly manufactured fentanyl.

The current model enhances clinical utility by predicting the need for NOWS pharmacotherapy at birth using routinely collected EHR data, making it a valuable clinical tool. Our previous model included diagnosis of hepatitis C as a proxy for illicitly manufactured fentanyl; however, we added maternal history of OUD and IV drug use to further address this important predictor. Additionally, infants’ head circumference and Apgar scores are incorporated, as they are recorded in the EHR at birth for all deliveries and can reflect variation in infant health and metabolism. We tested the model’s ability to predict NOWS diagnosis within our cohort and found calibration issues, indicating challenges with using diagnostic codes for outcomes, likely due to inconsistencies and temporal changes in coding practices (Supplemental Figure 2).³² We maintained a similar approach for assessing medication exposure, focusing on the 30 days prior to delivery.

Although we anticipate this model could support clinicians in identifying infants who may develop severe NOWS, informing clinical decision-making to extend birth hospitalization observation periods or safely discharging low-risk infants, implementing a clinical tool is not without implications for clinician workload and administrative oversight; therefore, it is important to first understand the value of the information the tool provides and then determine real-world effectiveness of the tool with a pragmatic trial.^33–35 A decision curve analysis is an underutilized method to evaluate the benefit of prognostic models before pursuing resource-intensive assessments. Core to the decision curve analysis is the threshold probability, which in our use case relates to the relative value of an infant receiving proactive treatment and possibly extended hospitalization if NOWS is present or avoiding these interventions if NOWS is not present. Although certain nonpharmacologic interventions, such as breastfeeding and parent coaching, can be considered minimally invasive, managing infants in the NICU or extending hospitalization for observation are both costly and invasive. For example, if the mother-infant dyad could remain together outside the NICU, the threshold probability would be relatively low, meaning the concern of overtreating is offset by the concern of delaying early interventions. Meshing evidence-based treatment with hospital protocols will require a participatory approach with partners including clinicians, health information technology administrators, and health system leaders.^36–38 We implemented the model in the EHR, separate from patient care, to gather prospective data and initiate partner discussions, prepare for a pragmatic trial, and plan for a sustainable intervention.²¹ Supplemental Table 2 provides the equation and predictor coefficients we used to implement the model in the EHR.

We used multiple approaches to develop the NOWS prediction model so it can be disseminated to other health care institutions. First, we used medication and diagnostic code sets created by the EHR vendor (Epic) and based on the National Library of Medicine, Value Set Authority Center.^39,40 Second, we used data that are common among institutions, structured, and available without additional user input. Third, we used a regression approach that is readily interpretable by users and programable in the EHR. Although we had sufficient sample size to include important features without concern of overfitting the model, the NOWS event rate was low. Among our cohort, we found the incidence rate of NOWS was 3.2 cases per 1000 births, which is within range of what others have observed (2.5–16.2 cases per 1000 births).^3,4,6 We anticipate the lower observed rate is due to a subset of NOWS that is treated with pharmacotherapy. A low event rate may overestimate performance of the model due to accurately identifying many negative cases, which is why routine maintenance is crucial for clinical prediction tools.^41,42 Furthermore, inclusion of preterm infants aged 35 weeks or younger may require additional investigation for confounding variables that impact NOWS severity and management. We may have missed medication exposure if patients received prescriptions outside the institution. Because the cohort was from a comprehensive regional pediatric center with a neonatal intensive care unit, the model may not generalize to other institutions with different services or protocols, geographic coverage, OUD treatment coverage, or EHR system.

We adapted a model to accurately predict severe NOWS at the time of birth using EHR data. As a clinical tool, the model has the potential to improve the accuracy of screening infants for guideline-recommended observation and guide nonpharmacologic and pharmacologic treatment immediately after birth.

AAP

American Academy of Pediatrics

AUC

area under the curve

EHR

electronic health record

ICI

integrated calibration index

IV

intravenous

LASSO

least absolute shrinkage and selection operator

NICU

neonatal intensive care unit

NOWS

neonatal opioid withdrawal syndrome

OUD

opioid use disorder

SSRI

selective serotonin reuptake inhibitor