Use of Electronic Health Record-Based Measures to Assess Quality of Care for Pediatric Agitation

We retrospectively evaluated performance on quality measures for pediatric ED acute agitation management at 1 children’s hospital and 2 nonchildren’s hospitals in a large metropolitan area. The hospitals belong to different health systems and are each located approximately 30 miles apart.

The children’s hospital has a 12-bed inpatient psychiatric unit serving children ages 3 to 17 and approximately 56 000 annual ED visits. The nonchildren’s hospitals are community hospitals with academic affiliations. One has an on-site inpatient psychiatric unit with 8 adolescent beds; the other is affiliated with an off-site inpatient psychiatric facility with 12 adolescent beds. They have 11 000 and 7000 annual pediatric ED visits, respectively. Each ED has a separate space to care for children that is open 12 and 16 hours per day, respectively. Children are seen by pediatric emergency medicine physicians when the pediatric space is open, and otherwise are seen by general emergency medicine physicians.

The children’s hospital and 1 nonchildren’s hospital have an algorithm in place to guide the management of acute agitation in children.¹⁴  The children’s hospital and the same nonchildren’s hospital require ED nurses to complete formal training in de-escalation techniques. The children’s hospital and the same nonchildren’s hospital have child life specialists available for 10.5 and 7 hours per day, respectively. All 3 hospitals use an EHR developed by Epic Systems Corporation (Verona, Wisconsin). Pro re nata orders are not placed at any of the participating EDs; all orders are placed as 1-time. The study was deemed exempt or as quality improvement by each hospital’s institutional review board.

Twenty quality measures for pediatric ED acute agitation management were previously ranked by multidisciplinary stakeholders as highly important and potentially feasible.¹⁰  We worked with quality improvement specialists and hospital data analysts to specify each proposed quality measure by defining data sources, numerators, and target population denominators. We considered quality measures as feasible to measure if they could be defined using existing structured data fields in the EHR, rather than unstructured data (eg, free text) requiring manual review, as this would facilitate use of the measures repeatedly over time for their intended purpose of quality improvement work. Minor changes were made to definitions of the following 5 measures to enhance feasibility: safe room, timely vitals, medication given, restraints, and child life involved. For example, the proposed denominator for the safe room measure, “patients at risk for agitation,” was changed to “mental health ED visits” to enable measurement using existing structured EHR data elements.

The population denominator of “mental health ED visits” was defined as ED visits by children 5 to 17 years old with a psychiatric chief complaint (“ED PSYCH” selected from an EHR picklist) or a primary International Classification of Diseases, 10^th Revision, Clinical Modification (ICD-10-CM) diagnosis code for a mental health condition, based on the previously validated Child and Adolescent Mental Health Disorders Classification System (CAMHD-CS).^15–17 

We developed a definition for the denominator “ED visits with medication given for agitation” by iteratively comparing combinations of structured EHR data elements against documentation of prespecified keywords in ED notes as the gold standard criterion (Supplemental Fig 1). The keywords included: agitation or agitated, aggression or aggressive, combative, violent, yelling, biting, punching, threatening, or risk of harm to self or others.¹⁸  We selected the combination of structured EHR data elements that maximized sensitivity and specificity (100% and 98.1%, respectively) for medication given for acute agitation (Supplemental Table 5), as follows: (1) administration of diphenhydramine IM, lorazepam IM, olanzapine IM, haloperidol IM, or chlorpromazine IM, or (2) a psychiatric chief complaint and administration of diphenhydramine per os (PO), lorazepam PO, or olanzapine orally dissolving tablet (ODT). The medications included in this definition are recommended in national consensus guidelines by child and adolescent psychiatrists for pediatric acute agitation management and were also listed in the agitation care algorithm used in 2 of the participating EDs.^5,14,19 

We defined the population denominator of “ED visits with physical restraints used” as ED visits with a physician order of physical restraint application for violent or self-destructive behavior.

We assessed overall quality measure performance during ED visits by children 5 to 17 years old who presented for a mental health condition, received medication for agitation, or received physical restraints from July 2020 to June 2021 at the 3 hospitals.

We assessed performance by the following patient characteristics: sex, age group (5–11, 12–17),^20,21  race and ethnicity (based on EHR documentation and categorized as Hispanic, non-Hispanic Asian, non-Hispanic Black, non-Hispanic white, and other or missing), insurance status (private, public, uninsured, other, or missing), and the presence or absence of each of 4 mental health diagnosis groups (autism spectrum disorder [ASD]; substance-related and addictive disorders; disruptive, impulse control, and conduct disorders; and developmental delay or intellectual disability). The 4 diagnosis groups were selected based on prior literature demonstrating increased risk for acute agitation among children in medical settings,^3,15,22,23  and the diagnosis groups were defined using CAMHD-CS, a validated ICD-10-CM classification system for pediatric mental health diagnoses.^15,16  Race and ethnicity were examined as social constructs rather than biologic determinants,²⁴  and were included in the analysis because prior literature has demonstrated disparities in ED mental health care by race and ethnicity.^9,25 

We used bivariate and multivariable models to examine measure performance by patient characteristics and by hospital. We assessed differences in measure performance using χ² test or Fisher’s exact test for categorical measures and analysis of variance for continuous measures. The multivariable analysis included patient characteristics as covariates and used a fixed effect variable for hospitals to account for hospital-level systematic differences in care. Some quality measures had insufficient sample size for fully-adjusted models; in these cases, we performed partially-adjusted models adjusted for age group, sex, and fixed effect for hospital only; or for age group, sex, and variables significant on bivariate analysis only. Analysis was performed using SAS/STAT version 15.1 (SAS Institute, Inc, Cary NC).

Across the 3 EDs, we identified 2785 ED mental health visits, 275 visits with medication given for agitation, and 35 visits with physical restraint use during the study period. Of mental health ED visits, 81.4% were by adolescents 12 to 17 years old, 46.4% were by non-Hispanic white children, 31.7% by Hispanic children, and 13.1% by non-Hispanic Black children. Insurance was distributed across private (46.2%) and public (44.5%) payors. One-quarter (25.5%) of children were admitted or transferred to a psychiatric unit and 17.0% were admitted to a medical or surgical unit (Table 1).

Of the proposed quality measures, 10 were feasible to measure using existing structured data elements in the EHR (Table 2). Supplemental Table 6 describes reasons that the remaining quality measures were not feasible to measure using EHR data.

Overall quality measure performance and performance variation by hospital are presented in Table 3. Of the 2785 ED mental health visits, 6.4% had medication given for acute agitation and 1.2% involved physical restraint use, with a mean time in restraints of 105 minutes. Of 275 ED visits with medication given for acute agitation, involvement of child life specialists and documentation of a patient-specific plan for agitation occurred in 5.1% and 3.3%, respectively. Of 139 ED visits with medication given for acute agitation resulting in discharge, 6.5% had a return ED visit within 7 days. Performance on 4 quality measures varied significantly by hospital: placement in a safe room, obtaining timely vitals, physical restraint use, and length of stay (P < .01 for each). In particular, physical restraint use varied from 0.5% to 3.3% of mental health ED visits across hospitals.

Bivariate analyses of quality measure performance by patient characteristics are presented in Supplemental Table 7. In multivariable analyses (Table 4), the adjusted odds of receiving medication for agitation were higher among ED visits by males than females (adjusted odds ratio [aOR] 1.71, 95% confidence interval [CI] 1.20 to 2.44); during visits by children who identified as other race and ethnicity (aOR 2.21, 95% CI 1.07 to 4.56) compared with non-Hispanic white children; and during visits with versus without the following diagnoses: ASD (aOR 3.88, 95% CI 2.31 to 6.54); substance-related and addictive disorders (aOR 2.46, 95% CI 1.35 to 4.48); disruptive, impulse control, and conduct disorders (aOR 1.82, 95% CI 1.13 to 2.95); and developmental delay or intellectual disability (aOR 9.03, 95% CI 5.02 to 16.24). The adjusted odds of restraint use were higher among ED visits by males than females (aOR 2.67, 95% CI 1.26 to 5.65), children with than without ASD (aOR 4.87, 95% CI 1.28 to 18.54), and children with than without substance-related and addictive disorders (aOR 5.48, 95% CI 2.20 to 13.64). The adjusted time from medication order to administration was 13.8 minutes shorter (95% CI −26.9 to −0.7) during ED visits by children with ASD than without ASD. The adjusted odds of child life specialist involvement were lower during ED visits by adolescents 12 to 17 years old than children 5 to 11 years old (aOR 0.26, 95% 0.08 to 0.84). The adjusted odds of documenting a patient-specific plan for agitation were higher for ED visits by children with than without developmental delay or intellectual disability (aOR 10.95, 95% CI 2.30 to 52.11). The adjusted ED length of stay was 4.9 hours longer (95% CI 1.2 to 8.7) for visits with than without a diagnosis of substance-related and addictive disorders.

We determined that assessment of care quality for pediatric acute agitation management in the ED was feasible using EHR-derived quality measures at a children’s hospital and 2 nonchildren’s hospitals. Of 20 quality measures previously prioritized as important by multidisciplinary stakeholders, 10 visit-level quality measures were feasible to assess using existing structured EHR data elements. We identified variation in care by patient characteristics and by hospital across several measures, suggesting that opportunities may be available to improve care for this population.

Children with ASD, developmental delay, and intellectual disability are at particularly high risk for developing acute agitation during mental health ED visits. In particular, we found that ED visits by children with developmental delay or intellectual disability had 9 times higher adjusted odds of medication use for acute agitation, whereas ED visits by children with ASD had nearly 5 times higher adjusted odds of physical restraint use. Our findings are consistent with data from inpatient settings indicating that nearly 1 in 5 children with ASD experience acute agitation during hospitalization, and they have nearly twice the odds of receiving physical restraints in this setting compared with children without ASD.^22,26  We found that patient-specific agitation plans were documented more often for children with developmental delay or intellectual disabilities, suggesting that such plans may be particularly beneficial in this population. Further efforts are needed to design streamlined patient-specific agitation treatment plans that contain information considered valuable and relevant to clinicians and families.^27,38  Future work should also evaluate whether increased use of patient-specific agitation plans can prevent or reduce the severity of acute agitation episodes.

Using the quality measures, we evaluated whether care differed for pediatric acute agitation management by race and ethnicity. Prior studies on pediatric physical restraint use in the ED have been limited to single health systems, with 1 study finding higher rates of restraint use during ED visits for Black than white youth.^4,25,29  In contrast, in the 3 EDs we studied, we did not identify disparities in physical restraint use by race and ethnicity. However, we did find higher rates of medication administration for acute agitation during ED visits by children identifying as other races and ethnicities compared with non-Hispanic white children, a difference that is difficult to interpret given the group’s heterogeneity. Additional large-scale studies are needed to determine whether there are widespread racial and ethnic disparities in pediatric acute agitation care. If so, work is needed to understand the underlying mechanisms and to develop interventions to address these disparities.

To assess quality measure performance, our work relied on existing EHR data, which enhanced feasibility of measurement but lacked comprehensiveness. For instance, verbal de-escalation techniques were only documented in the free text of notes, and agitation severity was not documented objectively using a numeric agitation severity scale.^30,31  Also, there was no EHR marker to indicate which children were at increased risk of becoming agitated. Future steps could include predictive modeling with EHR data or the development of agitation risk screening questions,³⁶  so as to enable identification of children at elevated risk for agitation and facilitate targeted quality improvement efforts. Importantly, no patient-reported outcome measures were documented in the EHR. To achieve care that is patient-centered, the development and routine use of patient-reported outcome measures for this population will be critical.^32,33  Our process relied on hospital data analysts to extract data from the EHR. In the future, automated reports or EHR dashboards could be designed to extract and display quality measures instantaneously.

A strength of our study was the inclusion of 2 nonchildren’s hospitals, because most US children receive their emergency care at nonchildren’s hospitals.¹³  Prior work from the National Pediatric Readiness Project suggests that few EDs have dedicated policies for the care of children with mental health conditions.³⁴  In our study, we identified that 1 of the nonchildren’s hospitals had 4-fold lower odds of physical restraint use than the other; this hospital had an algorithm to guide pediatric acute agitation management¹⁴  and required nurses to complete formal training in de-escalation techniques. Although this study focused on visit-level quality measures, future work should also examine structural, site-level factors that influence care quality for children with acute agitation.

Next steps will include validation of the quality measure specifications at hospitals with different EHR systems and where clinicians use different medications to manage acute agitation, including medications recommended in guidelines that are not used locally (eg, clonidine, chlorpromazine, risperidone, quetiapine).^5,19  Quality measurement across additional sites may enable the establishment of achievable benchmarks of care.³⁵  Studies are also needed to determine whether improvements on these quality measures for pediatric acute agitation management result in meaningful changes to health outcomes. In local quality improvement initiatives, these quality measures can be tracked over time along with changes to outcome measures as specific interventions are undertaken, such as the implementation of agitation algorithms, multidisciplinary simulation to practice de-escalation techniques, and implementation of behavioral rapid response teams.^37–40  Using these measures to track progress, quality improvement collaboratives can facilitate simultaneous learning across institutions to enhance the quality of care provided for children who experience acute agitation in the ED.

With quality measures derived from EHR data, there is potential for misclassification. For instance, our definition of visits with medication given for agitation may have included some visits with medications given for another indication or as routine home medications; conversely, the definition may have missed some patients who received medication for agitation. Nevertheless, we chose a definition that demonstrated good sensitivity and specificity when compared against clinician notes. We conducted this study at a limited number of hospitals located in the same metropolitan area. Subsequent testing across more hospitals is needed to determine whether measurement is feasible across different EHR systems, to externally validate measure specifications, and to understand the full breadth of variation in care practices to establish achievable benchmarks for care. This study did not assess whether quality measure performance was associated with improved health outcomes, which remains an important future direction. For some quality measures (eg, minutes spent in physical restraints), our analysis was limited by small numbers of eligible patients, which may limit their use as quality measures unless more patients are determined to be eligible when measured across a larger sample of hospitals.^41,42 

We determined that it was feasible to assess the quality of care provided for pediatric acute agitation management using ten stakeholder-prioritized, EHR-derived quality measures. Performance on these quality measures varied across patient characteristics and hospitals, including children’s and nonchildren’s hospital settings. This variation suggests that opportunities are available to improve the quality of care provided for a vulnerable and high-priority population.