Hospital Quality Indicators for Opioid-Exposed Infants: Results From an Expert Consensus Panel

Study procedures were approved as exempt by the RAND Institutional Review Board.

We recruited expert panelists from multiple stakeholder groups, including health care providers, parents in recovery, quality experts, public health experts, and policymakers, starting with member lists of professional organizations and investigator professional networks and allowing those who were contacted to suggest additional names. Participants received a $100 gift card or prepaid debit card for each of the 3 rounds completed.

To identify candidate quality indicators for panel consideration, we conducted a review of (1) measures used by hospitals in quality improvement programs for the care of opioid-exposed infants, including state perinatal collaboratives, and (2) measures used in observational cohort studies or randomized controlled trials for the effectiveness of specific treatments or therapies for the management of opioid-exposed infants. Our sources included the published literature, publicly available resources and toolkits identified in a search of state perinatal quality collaborative websites identified through the Centers for Disease Control and Prevention Division of Reproductive Health¹⁴ in August 2022, and national recommendations. We identified an initial list of 972 quality indicators, which we organized by predefined domains of clinical care, including birthing parent screening, prenatal intervention, pharmacologic treatment, nonpharmacologic treatment, toxicology testing, infant scoring, documentation and communication, transition of care, education, and provider sensitivity. Two study team members collaboratively reviewed the list, grouping overlapping indicators and removing indicators deemed highly infeasible to collect from existing data sources. Overlapping indicators were summarized to produce a final list of 49 indicators. We grouped indicators into 8 broad categories. Five categories focused on structures of care (ie, hospital policies and staff training), and 3 focused on processes and outcomes of care. We refined the consensus panel protocol through cognitive testing with 2 clinicians (an obstetrician–gynecologist and addiction medicine specialist and a pediatrician specializing in newborn care and NOWS) who were not part of the research team.

We solicited expert input on the quality indicators using ExpertLens, an online modified-Delphi platform.¹⁵ The panel consisted of 3 rounds conducted between June and July 2023, including 2 rating rounds with a discussion round in between, with each round open for 1 to 3 weeks. Participants were assigned numeric identifiers, and their identities were concealed throughout the panel. Participants were consented when they first logged into ExpertLens.

In Round 1, participants reviewed, rated, and commented on the 49 quality indicators, rating each indicator on 2 criteria: importance (How important is it to use this information as a measure of a hospital’s quality of care for an opioid-exposed infant?) and feasibility (How feasible is it to collect this information?). Participants provided ratings on a 9-point Likert scale, in which numeric scores corresponded with descriptive ratings of “low” (1–3), “uncertain” (4–6), or “high” (7–9) importance or feasibility. Participants were encouraged to provide comments to explain their ratings.

Round 2 focused on discussing the 14 indicators rated “uncertain” on importance or feasibility in Round 1. Participants reviewed bar charts showing their own responses in relation to the distribution of the group’s Round 1 responses (Supplemental Information). We asked participants to discuss Round 1 results by responding to prepopulated discussion threads or starting new threads. Based on Round 2 discussion, we revised 4 indicators before Round 3 to improve clarity and/or clinical appropriateness (Supplemental Table 2).

In Round 3, participants had the opportunity to revise their original ratings, using the same Likert scales, on the basis of the Round 2 discussion. As in Round 2, we showed only the 14 indicators initially rated “uncertain” on importance or feasibility.

We analyzed ratings of importance and feasibility for each indicator using the RAND/UCLA Appropriateness Method to determine if the panelists reached consensus,¹⁶ calculating the median score for importance or feasibility if consensus was reached. We used data visualization methods developed in previous ExpertLens studies to present the distribution of scores for each indicator.^17–19 

We conducted a rapid qualitative analysis of comments and discussion threads to understand the rationale for participant ratings, grouping comments corresponding with numeric ratings of low (1–3), uncertain (4–6), or high (7–9).^20–22 Two researchers reviewed and summarized the comments in each group to identify emergent themes that could explain ratings of importance and feasibility.

Of the 36 experts who agreed to participate, 32 completed at least 2 rounds, and 22 completed all 3 rounds. Most experts identified as women (78%) and non-Hispanic white (81%). The majority (69%) had a professional or doctorate degree, and 50% were health care providers (Table 1). Health care providers included physicians and advanced practice providers in neonatal–perinatal medicine, hospital pediatrics, and obstetrics and gynecology with expertise in substance-affected mothers and infants. Quality improvement and public health experts represented federal agencies, national associations, not-for-profit organizations, and state perinatal quality collaboratives addressing maternal child health, child welfare, and substance use.

The distribution of Round 3 ratings is shown in Fig 1. All indicators except one reached consensus. In total, 36 indicators were rated high on both importance and feasibility, and the rest were rated uncertain on importance and/or feasibility (Supplemental Table 2). Of the 14 indicators reevaluated in Round 3, one had a change in consensus decision from uncertain importance to high importance (C2).

For indicators focused on structures of care, experts generally agreed it was feasible to identify the relevant information from review of hospital policies. Several experts commented that it would be less feasible to evaluate whether these policies were followed.

Experts agreed that hospitals should have a standardized approach to screen pregnant people for substance use disorder, although some commented that parents may not disclose substance use because of stigma and fear of child welfare involvement. Opinions differed on hospital policy for toxicology testing as a quality indicator; the importance was rated “uncertain.” Some experts believed universal toxicology testing could prevent discrimination or profiling and could be beneficial to detect late-term exposures. Others commented that universal toxicology testing is not the standard of care and raises ethical issues related to the criminalization of pregnant people. Several experts raised concerns about potential negative consequences of universal toxicology testing, including discriminatory reporting to Child Protective Services, increased bias and stigma, further distrust in the health care system, and increased costs.

Round 2 discussion highlighted differences of opinion regarding an indicator to assess completion of pediatric prenatal consultations before delivery for patients with substance use disorder. Experts generally agreed that these consultations can be helpful to prepare families for delivery, although some did not think this indicator was useful to measure hospital quality. Several experts noted that consultations ideally should occur before the birth hospitalization to establish a rapport between providers and families. The indicator was revised accordingly before Round 3 (Supplemental Table 2). Although this indicator scored “high” on importance, feasibility was rated “uncertain.” Experts raised concerns that the information could be difficult to capture for patients receiving care from various prenatal clinics.

Experts agreed on the importance of staff training on assessment for NOWS, regular assessment of staff competencies for scoring infants at risk for drug withdrawal, and trauma-informed training. Multiple experts emphasized the importance of ongoing assessment of staff competencies to maintain consistent standards of care, including assessment of interrater reliability for scoring tools.

Experts agreed on the importance of a standardized protocol to score opioid-exposed infants for NOWS and the use of appropriate scoring and observation periods depending on the type of pharmacologic treatment or type of exposure. In the Round 2 discussion, experts noted that infant assessment for signs of withdrawal should begin before 12 hours of life and that “assessment” is a better term than “scoring” because it encompasses non-scoring approaches, such as eat, sleep, console. The indicator was revised to reflect this feedback (“Infant assessment for signs of withdrawal begins before 12 hours of life”), resulting in a change in consensus rating from uncertain importance to high importance.

Experts agreed on the importance of a standardized NOWS scoring tool and the use of this tool to make decisions for weaning pharmacotherapy. Experts generally agreed that hospital policy should not require admission to a neonatal intensive care unit (NICU) for infants treated with pharmacotherapy.

Experts agreed on the importance of hospital policies such as a standardized protocol guiding use of breastmilk and supplemental formula, policies to promote skin-to-skin contact after delivery, and the use of nonpharmacologic approaches (eg, rooming-in, environmental controls) from birth to discharge. Experts generally agreed that hospitals should provide cluster care to minimize disturbance to infants. For breastfeeding, the initial indicator included information about the hospital’s baby friendly status. Experts noted many hospitals may not have the resources to obtain this designation. The indicator was revised accordingly (“Hospital has resources in place to support breastfeeding, eg, lactation support in morning and evening, availability of hospital grade pumps, supplementing with donor breastmilk”). The importance for this indicator was rated “uncertain.”

For indicators focused on processes and outcomes of care, experts saw feasibility as largely dependent on the information routinely documented in the electronic medical record (EMR) or hospital billing records. Experts generally agreed that details of pharmacologic treatment were feasible to obtain from the EMR, whereas information on nonpharmacologic approaches and transitions of care may require manual abstraction or chart review, resulting in additional administrative burden for hospitals.

Experts agreed on the importance of a quality indicator to assess whether the primary medication used to treat infant opioid withdrawal was an opioid. Several experts commented that use of a secondary medication such as clonidine or phenobarbital depends on the clinical situation and is not necessarily an indicator of quality.

Experts had differing opinions regarding an indicator to assess whether the infant was observed at least 24 hours after weaning off medications before discharge, although this indicator scored “high” on importance and feasibility. Some experts commented that continued monitoring is important to prevent readmission for continued withdrawal symptoms. Others commented that the timeframe for monitoring should be individualized depending on the type of treatment.

“Duration of pharmacotherapy” and “percent of opioid-exposed infants treated with pharmacotherapy” scored “high” on importance and feasibility, although several experts commented that these indicators are imperfect measures of quality and should be evaluated in the context of multiple other indicators. Experts noted that optimal care may reduce the duration of treatment for most infants, but longer treatment does not necessarily reflect poor quality of care. Similarly, experts commented that the percentage of opioid-exposed infants treated with pharmacotherapy may vary depending on patient acuity and location. Several experts raised concerns about unintended consequences, such as incentivizing hospitals to discharge infants before they are medically ready or keep pharmacologic treatment rates low. However, experts noted these indicators may be useful for comparison with national benchmarks.

Experts rated importance “high” for nonpharmacologic approaches such as provision of breastmilk, mother and infant rooming-in, social work consults, lactation consults, and provision of high-calorie formula or fortified breastmilk for infants with rapid weight loss. Experts rated the importance as “uncertain” for consultations from physical therapy, occupational therapy, and child life. Experts commented that these services are potentially helpful but not essential and not available in all hospitals. Although the feasibility of indicators for nonpharmacologic approaches was rated “high” overall, several experts commented that this information may not be documented in the EMR and would likely require chart review.

Experts agreed on the importance of education and referrals before discharge, such as safe sleep education and referrals to Early Intervention Services, outpatient lactation support, and infant testing for hepatitis C virus. Experts agreed on the importance of birthing parents’ receipt of a naloxone prescription before discharge, and several commented that naloxone should be dispensed before discharge. Although the feasibility of these indicators was rated “high” overall, several experts commented that information related to transitions of care may not be documented in the EMR and would likely require chart review.

Experts generally agreed that duration of the birth hospitalization in days (ie, length of stay) is a relevant, yet imperfect, indicator of quality. Although feasibility was rated “high,” the importance of this indicator was rated “uncertain.” Experts commented that although excessive pharmacologic intervention can lead to a prolonged length of stay, many factors unrelated to quality of care can also impact length of stay. Several experts raised concerns that use of this indicator as a standalone measure could incentivize hospitals to discharge infants before they are medically ready. One expert suggested measuring hospital readmissions or emergency department visits post-discharge as an alternative to length of stay.

One indicator did not reach consensus: “The infant was discharged home with their biological mother.” Experts disagreed on the importance of this indicator as a measure of hospital quality. Multiple experts commented that this decision is made by social services, and therefore hospitals cannot be held accountable.

A panel of experts reached consensus on a range of quality indicators to assess the structures, processes, and outcomes of hospital-based care for opioid-exposed infants, rating most indicators high on importance and feasibility but noting some key considerations for their use in assessing hospital quality. First, experts noted that indicators such as “hospital length of stay” may vary by factors such as patient acuity and case mix, making these indicators imperfect measures of quality. Experts also cautioned that quality indicators used as “standalone” measures of hospital performance may result in unintended consequences for clinical practice (eg, patients discharged too soon, overuse or inappropriate use of treatment), suggesting that a composite measure that summarizes quality across a range of indicators may provide a more complete picture of hospital quality. Experts also cautioned against quality measures that create a substantial administrative burden for hospitals to implement (eg, measures that require manual chart abstraction).

Expert opinions differed in some areas. For example, some experts saw potential advantages of universal toxicology testing for pregnant people, whereas others thought the negative consequences, such as increased child welfare involvement, may outweigh benefits. Several experts also commented that hospitals should not be held accountable for policies or outcomes beyond their control, such as decisions about whether infants are discharged from the hospital with their biological parents.

Much of the literature on quality of care for opioid-exposed infants focuses on pharmacologic treatment and hospital length of stay. Notably, experts had strong consensus on the importance of quality indicators to assess other aspects of care, including the universal screening of pregnant people for substance use disorder, hospital staff training, standardized assessment for NOWS, nonpharmacologic interventions, and transitions of care. Our initial search identified numerous quality indicators assessing structures of care in hospitals but fewer indicators assessing patient-level outcomes and processes of care. The limited number of process and outcome measures limits aspects of care that can be used for benchmarking hospital performance or evaluating the effectiveness of interventions. Further work is needed to evaluate the feasibility, validity, and reliability of candidate process and outcome indicators using EMR and claims data.

This study had several limitations. Like many Delphi panels, there was some attrition over the rounds that could have influenced our results. We conducted a single panel, and therefore we are unable to assess the replicability of our findings. Our participant group was relatively homogeneous with regard to demographics (eg, education, race and ethnicity) and did not include policymakers. A different group of participants (eg, fewer health care providers, more parents in recovery) may have yielded different consensus ratings. The RAND/UCLA Appropriateness Method requires responses from at least 9 participants to determine consensus within each stakeholder group; thus, we were unable to compare consensus ratings across stakeholder types.

The opioid epidemic remains a public health crisis with far reaching consequences. Continued efforts are needed to understand and improve the quality and outcomes of care for opioid-exposed infants, given the wide variation across hospitals in care practices and outcomes.^9,10 The quality indicators deemed important and feasible by this panel have potential for use in national benchmarking, intervention studies, and hospital performance measurement. Continued engagement of expert stakeholders throughout the measure development process will help to ensure that quality measures are patient-centered and based on rigorous evidence.

We wish to acknowledge Ashley Martin for contributions to qualitative data analysis.

EMR

electronic medical record

NOWS

neonatal opioid withdrawal syndrome