The number of youth presenting to hospitals with suicidality and/or self-harm has increased substantially in recent years. We implemented a multihospital quality improvement (QI) collaborative from February 1, 2018 to January 31, 2019, aiming for an absolute increase in hospitals’ mean rate of caregiver lethal means counseling (LMC) of 10 percentage points (from a baseline mean performance of 68% to 78%) by the end of the collaborative, and to evaluate the effectiveness of the collaborative on LMC, adjusting for secular trends.
This 8 hospital collaborative used a structured process of alternating learning sessions and action periods to improve LMC across hospitals. Electronic medical record documentation of caregiver LMC was evaluated during 3 phases: precollaborative, active QI collaborative, and postcollaborative. We used statistical process control to evaluate changes in LMC monthly. Following collaborative completion, interrupted time series analyses were used to evaluate changes in the level and trend and slope of LMC, adjusting for covariates.
In the study, 4208 children and adolescents were included—1314 (31.2%) precollaborative, 1335 (31.7%) during the active QI collaborative, and 1559 (37.0%) postcollaborative. Statistical process control analyses demonstrated that LMC increased from a hospital-level mean of 68% precollaborative to 75% (February 2018) and then 86% (October 2018) during the collaborative. In interrupted time series analyses, there were no significant differences in LMC during and following the collaborative beyond those expected based on pre-collaborative trends.
LMC increased during the collaborative, but the increase did not exceed expected trends. Interventions developed by participating hospitals may be beneficial to others aiming to improve LMC for caregivers of hospitalized youth with suicidality.
National organizations endorse counseling to reduce access to lethal means for youth hospitalized with suicidality and/or self-harm. However, little evidence exists to guide implementation of lethal means counseling in pediatric populations.
In this multihospital collaborative including children’s and nonchildren’s hospitals, teams implemented interventions to perform and document caregiver lethal means counseling; these resources may be beneficial to other hospitals aiming to implement this evidence-based process of care.
The substantial prevalence of mental health conditions among children and adolescents, coupled with a national shortage of mental health professionals, has contributed to increasing numbers of youth presenting to acute care hospitals in the United States with mental health crises.1–4 One in 5 children and adolescents in the United States has a mental health condition, yet only half receive treatment from a mental health professional.5 From 2007 to 2016, emergency department (ED) visits for mental health disorders increased by 60% and visits for self-harm increased by more than 300%.6 The proportion of hospitalizations for mental health conditions among youth have increased further during the coronavirus 2019 pandemic,7,8 yet few published quality improvement (QI) efforts have been disseminated to support this vulnerable pediatric population.
Among both adults and adolescents experiencing suicidality or self-harm, safety planning interventions have been shown to reduce suicidal ideation and subsequent suicide attempts.9,10 Psychoeducation about how to restrict access to lethal means, such as firearms and high-risk medications, is an integral component of these safety planning interventions and may be lifesaving.11,12 Recognizing both the lethality of firearms and the growing number of children in the United States who live in households with firearms – 30 million children in 2021 – counseling to reduce access to these highly lethal means is advocated by national organizations, including the American Academy of Children and Adolescent Psychiatrists and the American Academy of Pediatrics.13–15 QI efforts to improve the rate of counseling caregivers during their child’s hospitalization may be particularly beneficial, as we have shown in past research that caregiver receipt of lethal means counseling (LMC) is associated with significantly lower rates of hospital readmission,16 and because hospitalization is an opportune time for parents to increase home safety before their child returns home. Additionally, there is substantial variation across hospitals in rates of caregiver LMC, highlighting the need for QI interventions.16,17
To improve healthcare quality for children and adolescents admitted to a pediatric medical or psychiatric unit with self-harm or suicidality, we implemented a multihospital QI collaborative from February 1, 2018 to January 31, 2019 to increase performance and documentation in the electronic medical record (EMR) of caregiver LMC before hospital discharge. We specifically aimed for an absolute increase in hospitals’ mean rate of caregiver LMC of 10 percentage points (improvement from a baseline mean performance of 68% to 78%) by the end of the active QI collaborative period. Additionally, we aimed to evaluate the effectiveness of the collaborative on performance and documentation of LMC, adjusting for secular trends. We hypothesized that the hospital’s mean rate of caregiver LMC would increase significantly during the collaborative and would be sustained following the period of active intervention.
Methods
Context
This study was conducted as a component of the Pediatric Hospital Care Improvement Project (PHIP), an 8 hospital collaborative that aimed to disseminate, implement, and improve performance on transitions of care and mental health quality measures developed through the Pediatric Quality Measures Program funded by the Agency for Healthcare Quality and Research.18 The evidence-base supporting the feasibility, validity, and interrater reliability of these mental health quality measures has been published previously.16,17 The project was reviewed and approved by participating hospitals’ institutional review boards; it was considered “data only” human subjects research with no participant contact and underwent expedited review.
PHIP-participating hospitals were recruited from the Pediatric Research in Inpatient Settings (PRIS) network, a national voluntary pediatric hospital medicine research network.19 Hospitals were selected with a goal of representing diverse United States geographic regions and hospital types. Five hospitals were freestanding children’s hospitals and 3 were academic medical centers with pediatric units (Table 1). The intervention setting varied across hospitals, with some sites focusing the intervention within the ED and psychiatric unit only, and others focusing within the pediatric (nonpsychiatric) unit(s).
Funding for this study was received in September 2016, and data collection was divided into 3 phases:
The precollaborative period from September 2014 to August 2016 and December 2017 to January 2018 was phase 1. The time period of September 2014 to August 2016 preceded study funding and data were collected retrospectively to evaluate 5 PHIP mental health quality measures. From September 2016 to November 2017 data were not collected as we focused on analysis of these measures, selected a primary quality measure to focus on for the collaborative, and convened hospitals’ stakeholder teams. From December 2017 to January 2018, before active QI efforts, we extracted additional data for our LMC measure only.
The active QI collaborative period from February 2018 to January 2019 was phase 2.
The postcollaborative period from February 2019 to January 2020 was phase 3, during which time grant-funded support for QI efforts was no longer provided but data abstraction continued to evaluate maintenance of LMC.
The LMC measure was selected from the 5 PHIP mental health quality measures as the focus of this collaborative given variation in performance across sites during the precollaborative period and perceived importance and feasibility of QI in this arena. Details regarding the other measures collected during the precollaborative period have been published previously.16
Population
Eligible patients included children and adolescents 5 to 17 years of age (hereafter called youth) hospitalized at the 8 participating hospitals for suicidality or self-harm as identified by predefined principle or secondary ICD-9 and ICD-10 codes (Supplemental Table 4). Youth admitted to intensive care units were included if they received care on a medical or psychiatric unit before discharge. Only youth discharged to home or partial hospitalization (day treatment programs) were included; youth transferred to another inpatient or residential treatment facility were excluded. During the baseline period, up to 210 eligible youth per hospital were randomly selected per hospital-specific procedures for study inclusion. At hospitals with <210 eligible youth during this period, all were included. During the active QI collaborative and postcollaborative periods, up to 20 eligible youth per month at each site were randomly selected for study inclusion and hospitals with <20 youth per month included all eligible cases.
Interventions
The Institute for Healthcare Improvement Breakthrough Series model formed the operational framework for this collaborative.20 This learning system model uses a structured process of alternating learning sessions and action periods to facilitate change across multiple hospitals at the same time. Before the active QI collaborative period, a central team of QI experts conducted interviews with high performing hospitals at baseline (n = 4 collaborative hospitals; mean rate of LMC ≥ 78%) to identify best practices to inform development of a key driver diagram that modeled the drivers of the desired outcome and interventions (Fig 1). During the active QI collaborative period, teams developed 90-day aim statements to guide their improvement work and implemented multiple small rapid cycle tests of change (Plan-Do-Study-Act cycles), addressing hospital-specific barriers using locally available resources (Supplemental Table 5). Project facilitation included collaborative bimonthly webinars with all participating hospital QI teams, monthly meetings with individual hospital’s QI team, and other communication between these meetings to foster joint learning. Data collected by all teams was shared frequently in the form of both site-specific and collaborative-wide statistical process control (SPC) charts, and high performing teams were asked to share successful strategies with others. A change package was developed following the active QI collaborative based on successful strategies identified by sites; it was disseminated to support future QI interventions in other hospitals.21
Working within the above-described framework, each hospital convened a stakeholder team to implement local QI activities with teams selected based on local priorities and resources (summary provided in Table 1; additional details are provided in Appendix 1 in the Supplemental Information. Also building from local hospital needs and resources, teams developed unique intervention(s), including psychoeducational classes and written materials for caregivers. Additional intervention components included training of healthcare professionals, smart phrases for documentation of LMC in EMRs, and processes of monitoring and feedback for healthcare professionals.
Study of the Intervention
Our analytic approach was twofold. First, we used SPC methods to monitor for and identify improvements in LMC throughout the QI collaborative, both at the hospital level and collaborative (multihospital) level. Second, following project completion, we conducted unadjusted and adjusted analyses to examine whether there were significant differences in LMC performance over time. Adjusted analyses used interrupted time series (ITS) segmented regression to assess for differences in LMC beyond those expected based on the precollaborative period because of secular trends between the precollaborative, active QI collaborative, and postcollaborative periods.
LMC Quality Measure
The quality measure evaluated in this collaborative was a binary process of care measure (yes or no, which was coded as 100 or 0) evaluating EMR documentation of caregiver LMC before hospital discharge. These data were extracted by trained medical record abstractors using an electronic abstraction tool (Appendix 1 in the Supplemental Information; also available for download online22 ). If there was not clear documentation that caregivers had received LMC, documentation that a safety plan was discussed with the caregiver was also acceptable. In contrast, a safety contract with the patient was not considered acceptable documentation.
Analysis
a. Study Population and Hospital Characteristics
To characterize our study population, we examined sex, age, race, ethnicity, and season of hospitalization, all obtained from administrative data provided by the hospitals. Race and ethnicity were self-reported at the time of hospital registration per participating hospitals’ standard procedures and were included in analyses, given previous findings of racial and ethnic disparities in LMC.16 We examined the distribution of these characteristics by time period (precollaborative, active QI collaborative, and postcollaborative) and tested for differences using χ2 tests for categorical variables and ANOVA tests for continuous variables.
b. Statistical Process Control Evaluation
We assessed the proportion of caregivers with documented LMC over time at individual hospitals and combined across the 8 hospitals using SPC charts. For each participating hospital, we first determined the monthly percent of eligible cases who had documented LMC and plotted this value on a p-chart. To examine performance across all hospitals, we generated a p-chart using the arithmetic mean of the monthly hospital-specific LMC percentages; this approach weighted each hospital's contribution equally, consistent with our multihospital collaborative design. We followed established rules to identify special cause variation and determine when to move center-lines based on this analysis in conjunction with team member input. The observation period for center-line shifts began at the start of the active QI collaborative period.23 As a secondary analysis, we also created a p-chart using the percentage of LMC across all 8 hospitals by summing the monthly numerators and denominators at all sites; with this approach, hospitals with larger denominators contributed more to the combined chart.
c. Evaluation of Trends Over Time
Unadjusted analyses examined the proportion of caregivers with documented LMC within each time period and tested for differences over time using ANOVA tests. Adjusted analyses used ITS models where the independent variable was LMC performance. Because LMC data were not collected September 2016 to November 2017, the proportions of caregivers receiving LMC during this period were estimated based on precollaborative trends. At the beginning of the active QI collaborative, we expected interventions to be implemented gradually and thus did not allow for an intercept change at that time. In contrast, we allowed for an intercept change at the start of the postcollaborative period because grant support for QI efforts was no longer provided and an abrupt change in performance might be expected. We additionally tested for (1) a change in slope at the start of the active QI collaborative period and (2) a change in slope at the start of the postcollaborative period. The model adjusted for seasonality given the seasonal nature of hospitalizations for suicidality or self-harm,24 patient characteristics (age, sex, race, ethnicity), and hospital fixed effects. Missing values for patient characteristics were imputed using the mean within hospital. Analyses were conducted using SAS 9.4; statistical testing was 2-sided and Pvalues <.05 were considered statistically significant.
Results
Study Population
A total of 4208 youth were included in this analysis, including 1314 (31.2%) during the precollaborative period, 1335 (31.7%) during the active QI collaborative, and 1559 (37.0%) during the postcollaborative period (Table 2). 1The majority of participants were adolescents 13 to 17 years of age (n = 3289, 78.2%), and 39.8% (n = 1594) were members of racial and ethnic minority groups. During the active QI collaborative period, the number of eligible participants per hospital ranged from a low of 18 (1.3% of sample) to a high of 232 (17.4% of sample). There were significantly different distributions of age, race, ethnicity, number of participants per hospital, and discharge season.
Characteristics of Intervention Settings, Implementation Teams, and Quality Improvement Interventions
| Site ID | Hospital Description | Number of Pediatric Beds | Intervention Settings | Quality Improvement Team Composition | Brief Description of Intervention |
|---|---|---|---|---|---|
| A | Freestanding children’s hospital with several satellite locations | Medical-surgical: 635; psychiatric: 110 | Adolescent inpatient psychiatry units across 2 locations | Licensed social workers including 1 clinical manager, 2 lead coordinators, and 3 social workers providing direct patient care |
|
| B | Freestanding children’s hospital | Medical- surgical: 153, psychiatric: 41 | Pediatric inpatient units, psychiatry and behavioral medicine unit (PBMU) | Pediatric mental health specialist, inpatient psychiatry director, child psychiatrist, chief of psychosocial services, RN, medical director |
|
| C | Freestanding building connected to adult hospital that contains a pediatric ED and inpatient child psychiatry unit | Medical-surgical: 74, psychiatric: 15 | Pediatric ED, Pediatric inpatient units, inpatient psychiatry unit | Pediatric hospitalist, child psychiatrists, pediatric emergency department physicians, nurses, licensed social workers |
|
| D | Freestanding children’s hospital | Medical-surgical: 216, psychiatric: 18 | Pediatric inpatient units, Inpatient psychiatry unit | Pediatric hospitalist, child psychiatry clinical and patient safety director, behavioral health specialists, clinical research coordinators |
|
| E | Freestanding children’s hospital within academic medical center with pediatric ED | Medical-surgical: 131, psychiatric: 24a | Pediatric inpatient units, psychiatric hospital | Medical director, pediatric rheumatologist, hospitalist, child psychiatry fellow, child psychiatry, residents, study coordinator |
|
| F | Nested children’s hospital | Medical-surgical: 40, psychiatric: 17 | Pediatric psychiatric ED, pediatric psychiatric unit | Medical director pediatric quality and safety, medical director pediatric psychiatric unit, social workers, research coordinator |
|
| G | Freestanding children’s hospital within academic medical center with pediatric ED | Medical- surgical: 189, psychiatric: 0 | Inpatient pediatric unit | Clinical director, psychologist, hospitalist, data coordinator, behavioral emergency response team |
|
| H | Nested children’s Hospital | Medical-surgical: 95, psychiatric: 28a | Pediatric inpatient units, inpatient psychiatry unit | Pediatric hospitalist, child psychiatrist, nurses, licensed social workers, research assistant |
|
| Site ID | Hospital Description | Number of Pediatric Beds | Intervention Settings | Quality Improvement Team Composition | Brief Description of Intervention |
|---|---|---|---|---|---|
| A | Freestanding children’s hospital with several satellite locations | Medical-surgical: 635; psychiatric: 110 | Adolescent inpatient psychiatry units across 2 locations | Licensed social workers including 1 clinical manager, 2 lead coordinators, and 3 social workers providing direct patient care |
|
| B | Freestanding children’s hospital | Medical- surgical: 153, psychiatric: 41 | Pediatric inpatient units, psychiatry and behavioral medicine unit (PBMU) | Pediatric mental health specialist, inpatient psychiatry director, child psychiatrist, chief of psychosocial services, RN, medical director |
|
| C | Freestanding building connected to adult hospital that contains a pediatric ED and inpatient child psychiatry unit | Medical-surgical: 74, psychiatric: 15 | Pediatric ED, Pediatric inpatient units, inpatient psychiatry unit | Pediatric hospitalist, child psychiatrists, pediatric emergency department physicians, nurses, licensed social workers |
|
| D | Freestanding children’s hospital | Medical-surgical: 216, psychiatric: 18 | Pediatric inpatient units, Inpatient psychiatry unit | Pediatric hospitalist, child psychiatry clinical and patient safety director, behavioral health specialists, clinical research coordinators |
|
| E | Freestanding children’s hospital within academic medical center with pediatric ED | Medical-surgical: 131, psychiatric: 24a | Pediatric inpatient units, psychiatric hospital | Medical director, pediatric rheumatologist, hospitalist, child psychiatry fellow, child psychiatry, residents, study coordinator |
|
| F | Nested children’s hospital | Medical-surgical: 40, psychiatric: 17 | Pediatric psychiatric ED, pediatric psychiatric unit | Medical director pediatric quality and safety, medical director pediatric psychiatric unit, social workers, research coordinator |
|
| G | Freestanding children’s hospital within academic medical center with pediatric ED | Medical- surgical: 189, psychiatric: 0 | Inpatient pediatric unit | Clinical director, psychologist, hospitalist, data coordinator, behavioral emergency response team |
|
| H | Nested children’s Hospital | Medical-surgical: 95, psychiatric: 28a | Pediatric inpatient units, inpatient psychiatry unit | Pediatric hospitalist, child psychiatrist, nurses, licensed social workers, research assistant |
|
Pediatric psychiatric beds affiliated with medical center in a separate building.
Characteristics of Participants, Hospital Volumes, and Seasonality of Admissions
| Precollaborative,an (%),dn = 1314 | Active QI Collaborative,bn (%),dn = 1335 | Postcollaborative,cn (%),dn = 1559 | Pe | |
|---|---|---|---|---|
| Demographic characteristics | ||||
| Age, y | <.001 | |||
| 5–12 | 240 (18.26) | 284 (21.27) | 395 (25.34) | |
| 13–15 | 573 (43.61) | 588 (44.04) | 692 (44.39) | |
| 16–17 | 501 (38.13) | 463 (34.68) | 472 (30.28) | |
| Sex | .814 | |||
| Female | 858 (65.70) | 854 (65.95) | 1035 (66.77) | |
| Male | 448 (34.30) | 441 (34.05) | 515 (33.23) | |
| Race and ethnicityf | .003 | |||
| Hispanic | 152 (12.19) | 159 (12.62) | 180 (12.04) | |
| Non-Hispanic Asian or Pacific Islander | 35 (2.81) | 31 (2.46) | 29 (1.94) | |
| Non-Hispanic Black or African American | 195 (15.64) | 198 (15.71) | 228 (15.25) | |
| Non-Hispanic white | 772 (61.91) | 763 (60.56) | 873 (58.39) | |
| Other | 93 (7.46) | 109 (8.65) | 185 (12.37) | |
| Hospitalization characteristics | ||||
| Hospital identifier | <.001 | |||
| A | 168 (12.79) | 230 (17.23) | 237 (15.20) | |
| B | 201 (15.30) | 188 (14.08) | 171 (10.97) | |
| C | 214 (16.29) | 140 (10.49) | 238 (15.27) | |
| D | 196 (14.92) | 232 (17.38) | 249 (15.97) | |
| E | 194 (14.76) | 205 (15.36) | 232 (14.88) | |
| F | 140 (10.65) | 150 (11.24) | 223 (14.30) | |
| G | 15 (1.14) | 18 (1.35) | 11 (0.71) | |
| H | 186 (14.16) | 172 (12.88) | 198 (12.70) | |
| Season | <.001 | |||
| Spring | 287 (21.84) | 350 (26.22) | 387 (24.82) | |
| Summer | 187 (14.23) | 321 (24.04) | 360 (23.09) | |
| Fall | 308 (23.44) | 325 (24.34) | 402 (25.79) | |
| Winter | 532 (40.49) | 339 (25.39) | 410 (26.30) | |
| Precollaborative,an (%),dn = 1314 | Active QI Collaborative,bn (%),dn = 1335 | Postcollaborative,cn (%),dn = 1559 | Pe | |
|---|---|---|---|---|
| Demographic characteristics | ||||
| Age, y | <.001 | |||
| 5–12 | 240 (18.26) | 284 (21.27) | 395 (25.34) | |
| 13–15 | 573 (43.61) | 588 (44.04) | 692 (44.39) | |
| 16–17 | 501 (38.13) | 463 (34.68) | 472 (30.28) | |
| Sex | .814 | |||
| Female | 858 (65.70) | 854 (65.95) | 1035 (66.77) | |
| Male | 448 (34.30) | 441 (34.05) | 515 (33.23) | |
| Race and ethnicityf | .003 | |||
| Hispanic | 152 (12.19) | 159 (12.62) | 180 (12.04) | |
| Non-Hispanic Asian or Pacific Islander | 35 (2.81) | 31 (2.46) | 29 (1.94) | |
| Non-Hispanic Black or African American | 195 (15.64) | 198 (15.71) | 228 (15.25) | |
| Non-Hispanic white | 772 (61.91) | 763 (60.56) | 873 (58.39) | |
| Other | 93 (7.46) | 109 (8.65) | 185 (12.37) | |
| Hospitalization characteristics | ||||
| Hospital identifier | <.001 | |||
| A | 168 (12.79) | 230 (17.23) | 237 (15.20) | |
| B | 201 (15.30) | 188 (14.08) | 171 (10.97) | |
| C | 214 (16.29) | 140 (10.49) | 238 (15.27) | |
| D | 196 (14.92) | 232 (17.38) | 249 (15.97) | |
| E | 194 (14.76) | 205 (15.36) | 232 (14.88) | |
| F | 140 (10.65) | 150 (11.24) | 223 (14.30) | |
| G | 15 (1.14) | 18 (1.35) | 11 (0.71) | |
| H | 186 (14.16) | 172 (12.88) | 198 (12.70) | |
| Season | <.001 | |||
| Spring | 287 (21.84) | 350 (26.22) | 387 (24.82) | |
| Summer | 187 (14.23) | 321 (24.04) | 360 (23.09) | |
| Fall | 308 (23.44) | 325 (24.34) | 402 (25.79) | |
| Winter | 532 (40.49) | 339 (25.39) | 410 (26.30) | |
The precollaborative period was September 2014 to August 2016 and December 2017 to January 2018.
The active quality improvement collaborative period was February 2018 to January 2019.
The postcollaborative period was February 2019 to January 2020.
Percentages are calculated excluding missing values.
χ2 tests (for categorical variables) and ANOVA tests (for continuous variables) were performed to test for differences across time categories; χ2 tests were performed among non-missing cases only
Each hospital provided an indicator of Hispanic ethnicity and a separate race variable. We created mutually exclusive race and ethnicity categories such that Hispanic patients of any race were classified as Hispanic and non-Hispanic patients were classified as non-Hispanic white, Black, Asian or Pacific Islander, or other.
Statistical Process Control Evaluation
Hospital-specific p-charts for the 8 sites are provided in Supplemental Fig 4. Across sites, the center-line varied from a low of 16% to a high of 97% during the baseline period. At all but 1 hospital, the center-line increased during the active QI collaborative, and 2 sites (hospitals B and C) had second center-line shifts during this time period.
Figure 2 shows the p-chart of caregiver LMC across 8 implementation sites using the arithmetic mean of hospital-specific percentages. In accordance with established rules for identifying special cause variation – specifically 8 consecutive points above or below the center-line23 - the center-line on the collaborative p-chart was shifted from 68.2% to 74.9% in February 2018 and was shifted again to 86.1% in October 2018. Supplemental Fig 5 shows the p-chart combining all numerators and denominators across the 8 hospitals. The results are similar to our primary hospital-level analysis, but the mean rate of caregiver LMC during the baseline period is 69.7% (versus 68.2% on the hospital-level chart). Like the hospital-level analysis, we observed 8 consecutive points above this center line in February 2018, but the shift at this time was to 83.1% (versus 74.9% on the hospital-level chart). Unlike the hospital-level p-chart, we did not observe a second shift later in the collaborative.
Statistical process control p-chart showing mean hospital-level rates of caregiver lethal means counseling over time.
Statistical process control p-chart showing mean hospital-level rates of caregiver lethal means counseling over time.
Evaluation of Trends Over Time
As shown in Table 3, unadjusted performance of LMC varied across hospitals, ranging from a low of 17.29% (SD = 37.90) to a high of 97.86% (SD = 14.53) during the precollaborative period, from 22.22% (SD = 42.78) to 97.83% (SD = 14.61) during the active QI collaborative period, and from 55.88% (SD = 49.76) to 98.73% (SD = 11.20) during the postcollaborative period. Unadjusted performance of LMC was significantly different across time periods for 6 of the 8 hospitals; in all but 1 of these hospitals, unadjusted mean scores increased during the active QI collaborative period compared with the precollaborative period.
Caregiver Lethal Means Counseling Performance Before, During, and Following Implementation of the Multisite Quality Improvement (QI) Collaborative (unadjusted)
| Hospital Identifier | Measure Performance Precollaborative,a Mean % (SD) | Sample Size During Precollaborative Perioda | Measure Performance Active QI Collaborative,b Mean % (SD) | Sample Size Active QI Collaborative Periodb | Measure Performance Postcollaborative,c Mean % (SD) | Sample Size Postcollaborativec (mean, SD) | P valued |
|---|---|---|---|---|---|---|---|
| A | 90.48 (29.44) | 168 | 97.83 (14.61) | 230 | 98.73 (11.20) | 237 | <.001 |
| B | 80.10 (40.02) | 201 | 94.15 (23.53) | 188 | 87.72 (32.92) | 171 | <.001 |
| C | 17.29 (37.90) | 214 | 40.71 (49.31) | 140 | 55.88 (49.76) | 238 | <.001 |
| D | 66.84 (47.20) | 196 | 77.16 (42.07) | 232 | 87.95 (32.62) | 249 | <.001 |
| E | 72.68 (44.68) | 194 | 91.71 (27.64) | 205 | 85.78 (35.01) | 232 | <.001 |
| F | 97.86 (14.53) | 140 | 96.00 (19.66) | 150 | 95.52 (20.74) | 223 | .505 |
| G | 40.00 (50.71) | 15 | 22.22 (42.78) | 18 | 90.91 (30.15) | 11 | <.001 |
| H | 97.85 (14.55) | 186 | 93.60 (24.54) | 172 | 93.43 (24.83) | 198 | .088 |
| Patient-level mean across hospitals | 72.07 (44.88) | 1314 | 85.02 (35.70) | 1335 | 86.14 (34.56) | 1559 | <.001 |
| Hospital Identifier | Measure Performance Precollaborative,a Mean % (SD) | Sample Size During Precollaborative Perioda | Measure Performance Active QI Collaborative,b Mean % (SD) | Sample Size Active QI Collaborative Periodb | Measure Performance Postcollaborative,c Mean % (SD) | Sample Size Postcollaborativec (mean, SD) | P valued |
|---|---|---|---|---|---|---|---|
| A | 90.48 (29.44) | 168 | 97.83 (14.61) | 230 | 98.73 (11.20) | 237 | <.001 |
| B | 80.10 (40.02) | 201 | 94.15 (23.53) | 188 | 87.72 (32.92) | 171 | <.001 |
| C | 17.29 (37.90) | 214 | 40.71 (49.31) | 140 | 55.88 (49.76) | 238 | <.001 |
| D | 66.84 (47.20) | 196 | 77.16 (42.07) | 232 | 87.95 (32.62) | 249 | <.001 |
| E | 72.68 (44.68) | 194 | 91.71 (27.64) | 205 | 85.78 (35.01) | 232 | <.001 |
| F | 97.86 (14.53) | 140 | 96.00 (19.66) | 150 | 95.52 (20.74) | 223 | .505 |
| G | 40.00 (50.71) | 15 | 22.22 (42.78) | 18 | 90.91 (30.15) | 11 | <.001 |
| H | 97.85 (14.55) | 186 | 93.60 (24.54) | 172 | 93.43 (24.83) | 198 | .088 |
| Patient-level mean across hospitals | 72.07 (44.88) | 1314 | 85.02 (35.70) | 1335 | 86.14 (34.56) | 1559 | <.001 |
The precollaborative period was September 2014 to August 2016 and December 2017 to January 2018.
The active QI collaborative period was February 2018 to January 2019.
The postcollaborative period was February 2019 to January 2020.
ANOVA tests were performed to test for differences across time categories.
In the adjusted ITS analyses, LMC increased significantly during the precollaborative period (Fig 3; parameter estimate: 0.34, confidence interval [CI]: 0.27, 0.41, P < .001). Based on precollaborative trends, ITS analyses predicted increasing rates of LMC from September 2016 to November 2017, during which time actual data were not collected. During the postcollaborative period, there was not a significant shift in scores over time (parameter estimate: −0.94, CI: −3.80 to 1.93, P = .52). We additionally tested for terms that added to this model (1) a change in slope at the start of the active QI collaborative period and (2) a change in slope at the start of the postcollaborative period, but neither were significant (results not shown), indicating that the trend in LMC was not significantly different during the active QI collaborative and postcollaborative periods beyond what would have been expected given the precollaborative trend.
Interrupted time series model of caregiver lethal means counseling over time. Graph shows predicted measure scores over time from the interrupted time series (ITS) models. The ITS model included parameters to test for a change in intercept at the start of the postcollaborative period, and additionally adjusted for seasonality, patient characteristics (age, sex, race and ethnicity), and hospital fixed effects. Missing values for patient characteristics were imputed using the mean within hospital. The time period during which data was not collected is shaded in grey. Note: the predicted score line is not linear because the model adjusts for season.
Interrupted time series model of caregiver lethal means counseling over time. Graph shows predicted measure scores over time from the interrupted time series (ITS) models. The ITS model included parameters to test for a change in intercept at the start of the postcollaborative period, and additionally adjusted for seasonality, patient characteristics (age, sex, race and ethnicity), and hospital fixed effects. Missing values for patient characteristics were imputed using the mean within hospital. The time period during which data was not collected is shaded in grey. Note: the predicted score line is not linear because the model adjusts for season.
Supplemental Table 6 summarizes the coefficients for covariates included in the final ITS model of caregiver LMC over time. In addition to significant differences across hospitals, female sex (relative to males), and Hispanic ethnicity (relative to non-Hispanic white) were associated with increased rates of LMC.
Discussion
In this multihospital QI collaborative, SPC analyses identified 2 center-line shifts, with the proportion of caregivers who received LMC increasing from a hospital-level mean of 68% during the precollaborative period to 75% and then 86% during the collaborative. Correspondingly, we achieved our a priori goal of 78% of caregivers counseled to reduced access to lethal means. In adjusted ITS analyses, we found that although LMC increased over time, performance during and following the collaborative did not exceed expected levels given the precollaborative trends.
SPC and ITS analyses share several common features: both are well-established methods to evaluate QI interventions, both involve analysis of time-series data, and both apply methods to differentiate random variation from “actual” change.25,26 Despite this, they are rarely used in tandem. Because SPC analyses can be conducted in real time, they are very well-suited to the prompt evaluation and modification of QI interventions. In contrast, adjusted ITS analyses are conducted following completion of data collection; by design, ITS analyses can only be performed once data has been collected from at least 2 time periods (eg, preintervention and intervention periods). As a result, ITS analyses are less well-suited to inform prompt decision-making. However, ITS analyses do allow for adjustments using relevant covariates, and can test for statistically significant differences in both level (intercept) and trend (slope) over time. This study demonstrates both the value and the challenge of using the 2 analytic approaches together.
Although the SPC analysis indicated an increase in the hospital-level mean proportion of caregivers receiving LMC over time, from the ITS analysis we conclude that the observed changes in scores during and following the collaborative did not exceed those expected based on precollaborative trends. A challenge with our ITS analyses, however, is the absence of LMC data from September 2016 to November 2017. During this time, actual LMC data were not collected and our models predicted increasing rates of LMC based on precollaborative trends. This study’s conclusions depend upon the accuracy of these predicted trends, which we’re unable to validate with available data; it is possible that results from the SPC and ITS would have been more concordant if data from this time period were available. Additionally, it is challenging to determine the reasons for the increasing rates of LMC observed during the precollaborative period. This data were collected retrospectively and collaborative participation was unlikely to have influenced LMC rates at this time because funding for project implementation was not received until September 2016. However, this collaborative was implemented during a period of rapidly increasing ED utilization and acute care hospitalization for suicidality and self-harm.1,6 It is possible that hospitals dedicated additional resources to the care of youth hospitalized with suicidality during this time, which cannot be ascertained from available data.
Despite these challenges, the results of this study have important implications for hospitals aiming to improve LMC for caregivers of youth hospitalized with suicide and/or self-harm. Safety planning interventions that incorporate LMC have been associated with substantial decreases in suicidal behaviors following hospital discharge, and the Joint Commission endorses such interventions before hospital discharge.9,11,27 However, relatively little evidence is available to guide effective implementation of LMC, particularly for children and adolescents. A recent systematic review of LMC in EDs identified only 9 studies across pediatric and adult populations,28 whereas publications describing pediatric safety planning interventions in inpatient settings are limited to pilot studies.29–31 The results of this collaborative demonstrate the feasibility of LMC in diverse hospital settings that care for children; resources developed during this collaborative are publicly available for implementation and adaptation.21,32
These results should be interpreted in the context of study strengths and limitations. Although the inclusion of 8t hospitals of varying sizes and geographic regions is a strength, rural and community hospitals were under-represented, as were hospitals without pediatric psychiatric units. Consistent with the Institute for Healthcare Improvement Breakthrough Series model, hospitals prioritized and implemented interventions based on local resources and needs. Although this is advantageous as it reflects “real world” opportunities, we are unable to determine which hospital-specific efforts were most effective in increasing LMC. Additionally, we had limited ability to adjust for disease severity in our models; payer and co-occurring mental health diagnoses were not available for this analysis. Furthermore, direct observation of LMC was not feasible, so this analysis assumes that EMR documentation accurately reflects LMC performance; this approach is consistent with prior studies.33,34 In our adjusted ITS analyses we observed that caregivers of Hispanic youth were more likely than caregivers of non-Hispanic white youth to receive LMC, with no other observed differences based on race or ethnicity. Mixed methods studies ascertaining reasons for this observed difference may generate valuable new knowledge. We were unable to assess for disparities in LMC by caregiver preferred language for medical communication because of the lack of reliable documentation of this variable in EMRs. Future studies examining LMC based on caregiver race, ethnicity, and preferred language may be particularly important to ensure that QI efforts do not worsen existing disparities.
In conclusion, hospitals participating in this collaborative implemented a variety of interventions to improve caregiver LMC. Although unadjusted rates of LMC increased significantly, in adjusted ITS analyses there were no significant differences beyond the trend observed during the precollaborative period. However, strategies employed in this collaborative may be valuable to hospitals aiming to improve their performance on this evidence-based quality measure.
Acknowledgments
We thank Ms Lowrie Ward, MPH, CHP, Q Burkhart, and members of the study team at our participating hospitals, including Monroe Carell Jr Children’s Hospital at Vanderbilt (Jonathan Boyd, MD; Alaina Davis, MD, MPH; Nicole Drawbridge, MD; Yasas Tanguturi, MD; Meg Benningfield, MD; Jakobi Johnson, BS), University of California San Francisco (Marisol Romero; Gabriela Bronson-Castain), and Children’s Hospital Colorado (Dr Jason Williams, PsyD); and we thank Mr Lloyd Provost for his guidance in how to create SPC charts appropriate for our multihospital collaborative design.
Dr Mange's current affiliation is the Center for Health Equity Research and Promotion (CHERP), Corporal Michael Crescenz VA Medical Center, Philadelphia, PA.
Ms Esporas' current affiliation is Vynamic, Durham, NC.
Dr Parast's current affiliation is Department of Statistics and Data Sciences, University of Texas at Austin, Austin, T X.
FUNDING: This study was done under funding from a cooperative agreement with the Agency for Healthcare Research and Quality and Centers for Medicare and Medicaid Services, grant number U18HS025291. The content is solely the responsibility of the authors and does not represent the official views of the funder. The funder had no role in the study design; in the collection, analysis, or interpretation of data; in the writing of the report; nor in the decision to submit the article for publication. Funded by the National Institutes of Health (NIH).
CONFLICT OF INTEREST DISCLOSURES: Dr Britto reported receiving support from the American Board of Pediatrics Foundation Board of Directors and Dr Leyenaar reported receiving consulting support from the American Board of Pediatrics Foundation, both unrelated to the submitted work. The remaining authors have no conflicts of interest relevant to this article to disclose.
Comments