OBJECTIVES:

Hospitals accredited by The Joint Commission (TJC) are now required to use a validated screening tool and a standardized method for assessment of suicide risk in all behavioral health patients. Our aims for this study were (1) to implement a TJC-compliant process of suicide risk screening and assessment in the pediatric emergency department (ED) and outpatient behavioral health clinic in a large tertiary care children’s hospital, (2) to describe characteristics of this population related to suicide risk, and (3) to report the impact of this new process on ED length of stay (LOS).

METHODS:

A workflow using the Columbia Suicide Severity Rating Scale was developed and implemented. Monthly reviews of compliance with screening and assessment were conducted. Descriptive statistics were used to define the study population, and multivariable regression was used to model factors associated with high suicide risk and discharge from the ED. ED LOS of behavioral health patients was compared before and after implementation.

RESULTS:

Average compliance rates for screening was 83% in the ED and 65% in the outpatient clinics. Compliance with standardized assessments in the ED went from 0% before implementation to 88% after implementation. The analysis revealed that 72% of behavioral health patients in the ED and 18% of patients in behavioral health outpatient clinics had a positive suicide risk. ED LOS did not increase. The majority of patients screening at risk was discharged from the hospital after assessment.

CONCLUSIONS:

A TJC-compliant process for suicide risk screening and assessment was implemented in the ED and outpatient behavioral health clinic for behavioral health patients without increasing ED LOS.

Suicide is the second leading cause of death in youth 10 to 24 years of age.1  Authors of recent studies report rising rates in youth. Centers for Disease Control and Prevention data revealed a 30% increase in suicide rates from 2000 to 2017 in youth.2  In all age groups, the majority of completed suicides was in boys.1,2  In 10- to 17-year-old youth, use of firearms was the second most common and the most lethal method.3,4 

Studies of risk factors have revealed that suicidal ideation and behaviors are the most salient predictors of suicide.57  Effective July 1, 2019, according to the National Patient Safety Goal (NPSG) 15.01.01,8  all The Joint Commission (TJC)–accredited hospitals must screen all patients with behavioral health concerns using a validated screening tool that asks about suicidal ideation and behaviors. Furthermore, all patients with positive risk must be further assessed by using an evidence-based process that asks about the severity of suicidal ideation and behaviors as well as other risk and protective factors.9  TJC further recommends safety planning for all patients identified as at risk before discharge from the hospital with evidence-based resources to achieve this goal, including Counseling on Access to Lethal Means (CALM).8,10  CALM is designed for mental health professionals to counsel families on how to reduce access to lethal means for patients at risk for suicide. Lethal means reduction counseling for parents and guardians is an effective safety intervention for reducing suicide risk in youth.4 

In September 2018, a mock review was completed at our tertiary care children’s hospital in preparation for an upcoming TJC visit. In the review, it was found that current practice did not meet TJC standards. The emergency department (ED) was using a validated tool to screen for suicide risk but lacked a standardized method to assess patients who screened positive, whereas the behavioral health outpatient clinic (OPC) was not using a validated tool for screening or assessment.

Brahmbhatt et al11  described an approach to development of a suicide risk screening clinical pathway for pediatric hospitals using a three-tiered approach: an initial screen to identify patients at risk with the Ask Suicide-Screening Questions (ASQ), a brief assessment to assess risk severity, and a full evaluation for patients at high risk. Roaten et al12  described the practical implementation of universal suicide risk screening in an adult safety net hospital using the Columbia Suicide Severity Rating Scale (C-SSRS); however, implementation in a freestanding pediatric hospital setting has not been described. Time constraints, delays in care, provider discomfort, and lack of education were the most commonly cited limitations to screening in the ED and outpatient settings.13  A review of feasibility studies revealed no change in length of stay (LOS) related to screening,14  and screening was found to be acceptable to patients and parents.15 

Our aims for this study were to implement a new suicide risk screening and assessment process in the ED and OPC for behavioral health patients in accordance with TJC guidelines, to describe characteristics of this patient population related to suicide risk, and to assess the impact of this new process on ED LOS.

A workgroup of key stakeholders was assembled in October 2018 in response to this review to develop a new suicide risk screening process. This group included representatives from the divisions of psychiatry, psychology, and emergency medicine; nursing informatics; and the department of social work and met weekly to review progress and assign tasks to each subgroup.

Relevant literature was reviewed to guide policy and procedure development. A projected time line was created (Fig 1) with the goal to initiate the new process in the ED and OPC before the anticipated TJC visit in July 2019 and allow a 30-day gap between phase 1 and phase 2 to address initial problems.

FIGURE 1

Project time line.

FIGURE 1

Project time line.

Close modal

The proposal was initially met with several concerns from hospital leadership, including increased burden on mental health clinicians in the ED, potential negative impact on ED LOS, lack of support staff to provide safety interventions, acceptability of screening to patients, and the lack of evidence that screening prevents suicide. Despite these concerns, buy-in was obtained because of the TJC policy (NPSG 15.01.01)9  on mandatory screening. To address the concern for delays in care and shortage of staff in the ED, a simulation model for how patients would be screened and assessed was developed. Notional values for the duration of each step of medical and behavioral health evaluation were used, incorporating current resources (such as staff and secure ED beds) available to patients with behavioral health complaints. Outcome estimates included overflow in the ED to non–behavioral health beds, mean wait times for these patients, and their mean ED LOS. These estimates revealed that a phased approach model12  targeting a smaller population at a time would have a less drastic impact on overflow and ED LOS. Approval for 2 additional positions for ED social workers, 5 additional behavioral health technicians for safety interventions, and 1 nursing informatics staff member to help with electronic health record (EHR) building was obtained.

The model had 3 phases: phase 1, screening of patients presenting to the ED with behavioral health complaints; phase 2, screening of patients presenting to the behavioral health OPC; and phase 3, screening all non–behavioral health patients presenting to any hospital setting. In this study, we describe phases 1 and 2.

A review of existing TJC guidelines at the commencement of this project revealed no mandate related to age (an update published by TJC in November 2019 now recommends using a validated screening tool for ages ≥12 years). Presuming a higher burden of suicide risk among the behavioral health population,16,17  a lower limit of 6 years was chosen for this subgroup (phases 1 and 2) compared to a lower limit of 10 years for future non–behavioral health screening (phase 3). Although suicide in children <12 years is rare, it has been reported as the 10th most common cause of death1,2  and is more likely to occur in Black children in the elementary school age group.18  In 2019, the National Violent Death Reporting statistics revealed a suicide rate of 5 of 1 000 000 for 5- to 9-year-olds.3 

The factors considered by the workgroup were (1) the length of the tool, (2) the strength of psychometric properties in the pediatric population, and (3) the ability to stratify risk to guide safety interventions. TJC R3 report on NPSG 15.01.01 for Suicide Prevention8  recommends the Suicide Behavior Questionnaire–Revised,19  the Patient Health Questionnaire–9 (PHQ-9), the Patient Safety Screener,20  the ASQ tool,21  or the C-SSRS22  as possible options for screening tools for hospitals to use. For risk assessment, TJC recommends the C-SSRS Risk Assessment version,22  The Beck Scale for Suicide Ideation,23  and the Scale for Suicide Ideation–Worst.24 

Final contenders for suicide risk screening tools were the PHQ-9, the ASQ, and the C-SSRS given that these are most commonly used in the pediatric population (Supplemental Table 3). The PHQ-9 was not chosen because it is not specific to suicide risk screening. Both the C-SSRS and ASQ are validated in children and in triage risk. The C-SSRS tool was chosen because it stratified risk to low, moderate, and high to guide safety interventions, and the TJC recommends the C-SSRS Risk Assessment version to guide assessment.

The C-SSRS screener is a structured 6-question tool that screens for suicide risk by asking questions about thoughts, intent, plan, and behaviors over the past month and about any attempts over the past 3 months or the lifetime. It scores risk as high, medium, or low depending on affirmative answers (Fig 2). For further assessment, the C-SSRS Full version asks about the intensity of ideation and the severity of behavior, including assessing actual or potential lethality, and the risk assessment page provides a checklist of risk and protective factors. Studies of internal validity, sensitivity, and specificity of the screener have been conducted in patients 11 years or older20,22 ; however, in several studies, the C-SSRS has been used for patients as young as 5 years old.25  Two versions of the C-SSRS screener, varying in time line for symptoms and both available online,26  were incorporated into the EHR: (1) the C-SSRS Recent, which assesses risk over the past month, and (2) the C-SSRS Since Last Asked, which assesses risk since last assessed. The latter was used for rescreening in OPCs and daily rescreening of patients admitted to psychiatry or medical inpatient units. A full assessment, which combined the C-SSRS Full and Risk Assessment pages to meet TJC standard, was also built into the EHR.

FIGURE 2

C-SSRS screener, recent logic.

FIGURE 2

C-SSRS screener, recent logic.

Close modal

The key stakeholders collaborated to draft a process guide, which included both a written component as well as a visual algorithm in line with consensus recommendations from the Pathways in Clinical Care Group.14  The Pathways in Clinical Care Group clinical pathway was modified, with permission, according to the tool and interventions used (Supplemental Fig 4).

The ED is a 90-bed level 4 trauma center with 4 beds in a dedicated, locked behavioral health section and 5 behavioral health overflow beds in an unlocked area. Approximately 2000 patients a year present with a behavioral health complaint to the ED. The behavioral health section was staffed 24/7 by 1 emergency physician, 2 psychiatry social workers during peak hours, and 1 psychiatry social worker during nonpeak hours. One child and adolescent psychiatrist was on-site from 8 am to 5 pm. In addition, the ED had limited coverage by 1 medical social worker who traditionally saw families of patients coming in for medical complaints. The psychiatry social workers had past training and experience in treating patients with behavioral health concerns, whereas the medical social workers did not. Patients presenting with behavioral health complaints received a face-to-face evaluation by the psychiatry social worker, supervised in person or over the phone by a child and adolescent psychiatrist. Constant observation for safety, as needed, was provided by behavioral health technicians who had a high school diploma and training in crisis de-escalation.27 

Phase 1 went live on May 2, 2019. The C-SSRS was administered in the EHR by the triage nurse, preferably with the patient alone. Patients who refused to answer were triaged as moderate risk for further assessment. Each risk level automatically triggered specific safety orders in the EHR. Patients at low or moderate risk received environmental safety interventions (ie, change to hospital gown and safe meal trays), whereas all patients at high risk also received 1:1 constant observation. General environmental safety interventions included addition of lockers for belongings and metal detectors for visitors to the ED. See Supplemental Fig 5 for safety guidelines.

Additional support was garnered from medical social workers for completion of the C-SSRS Full Assessment, which reduced the burden on psychiatry social workers. Patients at risk who were discharged from the hospital received a safety plan, which included lethal means reduction counseling by the medical or psychiatry social worker. Discharged patients received a follow-up phone call from social work 24 hours after discharge.

Each month, a report for problems, including missed screens, was generated from the EHR, and reasons were determined; 24/7 EHR support was made available to the ED behavioral health staff for the first 72 hours after implementation. Early glitches included the screener appearing for children <6 years old and “refused to answer” failing to populate at moderate risk. These errors were immediately identified by triage nurses and corrected in the EHR. Another issue highlighted was that patients younger than 8 years were screening false-positive on the screener. Therefore, the C-SSRS screener was modified to include the simplified language in the C-SSRS Very Young Children28  version.

The OPC houses clinics for psychiatry and psychology and cares for ∼16 000 patients a year. The clinical staff included 4.2 full-time equivalent (FTE) psychiatrists, 2.4 FTE child and adolescent psychiatry fellows, 2.0 FTE psychologists, and 1.0 FTE nurse. One FTE patient care technician position was added before phase 2. Patient care technicians differ from behavioral health technicians in that they do not have crisis de-escalation training. This position allowed the normal clinic process to flow if the clinic nurse had to provide 1:1 observation.

Phase 2 went live on June 27, 2019. Patients were screened by the patient care technician using the C-SSRS screen in the EHR at intake. The clinician was informed of the risk level by using a laminated color-coded card in addition to the risk level order in EHR. All patients screening at high risk were placed on 1:1 observation with the clinic nurse or patient care technician until assessed by the primary clinical provider (psychologist or psychiatrist). If the clinician assessment determined a need for inpatient admission, the existing procedure for admission to the hospital inpatient psychiatry unit was followed. One unique challenge in this setting was determining the frequency with which a C-SSRS screener should be completed for patients with weekly follow-up appointments. Because TJC does not provide specific guidelines for frequency, the following 3 criteria for rescreening were used:

  1. a concern for worsening clinical status (as assessed by a clinician);

  2. first appointment after discharge from inpatient psychiatry unit; and

  3. yearly.

Training videos from the Columbia Lighthouse Project Web site26  were assigned to staff involved in screening and assessment (social workers, psychiatrists, and psychologists) to be completed before implementation of phase 1. In addition, all ED social workers were required to complete an online course10,29  on counseling for reducing access to lethal means (CALM). All hospital nursing and medical providers were required to complete a module30  to increase awareness around suicide. Before implementation of each phase, educational flyers that outlined workflow and roles using screenshots from the EHR were widely distributed. Ongoing education was conducted via lectures and seminars.

The data for the study was obtained from the EHR. All ED chief complaints during the study period were queried. These were further categorized as behavioral health versus other by 4 members of the workgroup.

A retrospective cross-sectional analysis, by using SAS version 9.4 (SAS Institute, Inc, Cary, NC), of all patients 6 to 17 years presenting to the ED (n = 1053) for a behavioral health complaint or to the OPC (n = 571) for an initial behavioral health appointment over 5 months was conducted. Descriptive statistics were used to define the characteristics of our population, and a multivariable logistic regression was used to model factors associated with (1) high risk on the C-SSRS and (2) discharge from the ED. The average ED LOS and the proportion of boarders per behavioral health assessment were compared before and after implementation by using statistical process control charts. Boarders were defined as patients needing admission who waited in the ED for >24 hours because of the lack of bed availability.

Patient Demographics

The majority of patients were girls (n = 581; 55.2%). The mean age was 13.4 years (SD: 2.8). Almost two-thirds were of non-Hispanic Black ethnicity (n = 667; 63.4%) and were publicly insured (n = 691; 65.6%); 64.9% (n = 683) were discharged from the ED (Table 1).

TABLE 1

Characteristics of the ED and OPC Behavioral Health Population by Level of Suicide Risk on the C-SSRS

VariableAll Behavioral Health ComplaintsTotal Screened, n (%)No Risk, n (%)Low Risk, n (%)Moderate Risk, n (%)High Risk, n (%)
ED       
 Total 1053 879 (84) 248 (28)b 166 (19)b 117 (13)b 348 (40)b 
 Age group, y       
  6–12 412 (39) 347 (39) 114 (46) 67 (40) 48 (41) 118 (34) 
  13–17 641 (61) 532 (61) 134 (54) 99 (60) 69 (59) 230 (66) 
 Sex       
  Female 581 (55) 489 (56) 105 (42) 100 (60) 53 (45) 231 (66) 
  Male 472 (45) 390 (44) 143 (58) 66 (40) 64 (55) 117 (34) 
 Insurance       
  Private 271 (26) 227 (26) 61 (25) 42 (25) 18 (15) 106 (31) 
  Public 691 (66) 576 (66) 165 (67) 105 (63) 87 (74) 219 (63) 
  No insurance 67 (6) 54 (6) 19 (8) 14 (8) 6 (5) 15 (4) 
  Missing 24 (2) 22 (2) 3 (1) 5 (3) 6 (5) 8 (2) 
 Race and/or ethnicity       
  Non-Hispanic Black 667 (63) 563 (64) 163 (66) 99 (60) 84 (72) 217 (62) 
  Non-Hispanic white 137 (13) 100 (11) 33 (13) 25 (15) 10 (9) 52 (15) 
  Hispanic 153 (15) 122 (14) 26 (10) 27 (16) 16 (13) 53 (15) 
  Other 196 (9) 74 (8) 26 (10) 15 (9) 7 (6) 26 (7) 
 Disposition       
  Discharged 683 (65) 541 (62) 181 (73) 114 (69) 70 (60) 176 (51) 
  Admitted or transferred 364 (35) 334 (38) 65 (26) 51 (31) 47 (40) 171 (49) 
  Left AMA 6 (<1) 4 (<1) 2 (<1) 1 (<1) 1 (<1) 
OPC       
 Total 571 374 (65) 306 (82) 23 (6) 23 (6) 22 (6) 
 Age group, y       
  6–12 356 (62) 227 (61) 196 (64) 16 (70) 7 (30) 8 (36) 
  13–17 215 (38) 147 (39) 110 (36) 7 (30) 16 (70) 14 (64) 
 Sex       
  Female 225 (45) 171 (46) 130 (43) 10 (43) 15 (65) 16 (73) 
  Male 316 (55) 203 (54) 176 (58) 13 (57) 8 (35) 6 (27) 
 Insurance       
  Private 344 (60) 224 (60) 178 (58) 13 (56) 18 (78) 15 (68) 
  Public 209 (37) 137 (37) 116 (38) 10 (44) 4 (17) 7 (32) 
  Missing 18 (3) 13 (3) 12 (3) 1 (4) 
 Race and/or ethnicitya       
  Non-Hispanic Black 189 (33) 129 (34) 104 (34) 8 (35) 6 (26) 11 (50) 
  Asian 15 (3) 10 (3) 8 (3) 1 (4) 1 (5) 
  White 227 (40) 152 (40) 122 (40) 10 (44) 13 (57) 7 (32) 
  Multiple race 5 (1) 2 (<1) 2 (1) 
  Missing 135 (24) 81 (22) 70 (23) 4 (17) 4 (17) 3 (14) 
VariableAll Behavioral Health ComplaintsTotal Screened, n (%)No Risk, n (%)Low Risk, n (%)Moderate Risk, n (%)High Risk, n (%)
ED       
 Total 1053 879 (84) 248 (28)b 166 (19)b 117 (13)b 348 (40)b 
 Age group, y       
  6–12 412 (39) 347 (39) 114 (46) 67 (40) 48 (41) 118 (34) 
  13–17 641 (61) 532 (61) 134 (54) 99 (60) 69 (59) 230 (66) 
 Sex       
  Female 581 (55) 489 (56) 105 (42) 100 (60) 53 (45) 231 (66) 
  Male 472 (45) 390 (44) 143 (58) 66 (40) 64 (55) 117 (34) 
 Insurance       
  Private 271 (26) 227 (26) 61 (25) 42 (25) 18 (15) 106 (31) 
  Public 691 (66) 576 (66) 165 (67) 105 (63) 87 (74) 219 (63) 
  No insurance 67 (6) 54 (6) 19 (8) 14 (8) 6 (5) 15 (4) 
  Missing 24 (2) 22 (2) 3 (1) 5 (3) 6 (5) 8 (2) 
 Race and/or ethnicity       
  Non-Hispanic Black 667 (63) 563 (64) 163 (66) 99 (60) 84 (72) 217 (62) 
  Non-Hispanic white 137 (13) 100 (11) 33 (13) 25 (15) 10 (9) 52 (15) 
  Hispanic 153 (15) 122 (14) 26 (10) 27 (16) 16 (13) 53 (15) 
  Other 196 (9) 74 (8) 26 (10) 15 (9) 7 (6) 26 (7) 
 Disposition       
  Discharged 683 (65) 541 (62) 181 (73) 114 (69) 70 (60) 176 (51) 
  Admitted or transferred 364 (35) 334 (38) 65 (26) 51 (31) 47 (40) 171 (49) 
  Left AMA 6 (<1) 4 (<1) 2 (<1) 1 (<1) 1 (<1) 
OPC       
 Total 571 374 (65) 306 (82) 23 (6) 23 (6) 22 (6) 
 Age group, y       
  6–12 356 (62) 227 (61) 196 (64) 16 (70) 7 (30) 8 (36) 
  13–17 215 (38) 147 (39) 110 (36) 7 (30) 16 (70) 14 (64) 
 Sex       
  Female 225 (45) 171 (46) 130 (43) 10 (43) 15 (65) 16 (73) 
  Male 316 (55) 203 (54) 176 (58) 13 (57) 8 (35) 6 (27) 
 Insurance       
  Private 344 (60) 224 (60) 178 (58) 13 (56) 18 (78) 15 (68) 
  Public 209 (37) 137 (37) 116 (38) 10 (44) 4 (17) 7 (32) 
  Missing 18 (3) 13 (3) 12 (3) 1 (4) 
 Race and/or ethnicitya       
  Non-Hispanic Black 189 (33) 129 (34) 104 (34) 8 (35) 6 (26) 11 (50) 
  Asian 15 (3) 10 (3) 8 (3) 1 (4) 1 (5) 
  White 227 (40) 152 (40) 122 (40) 10 (44) 13 (57) 7 (32) 
  Multiple race 5 (1) 2 (<1) 2 (1) 
  Missing 135 (24) 81 (22) 70 (23) 4 (17) 4 (17) 3 (14) 

AMA, against medical advice.

a

The OPC EHR does not record race and/or ethnicity the same as the ED.

b

Proportions for each level of risk are reported for all screened patients.

Proportion of Patients With Suicide Risk

Of all behavioral health patients aged 6 to 17 years presenting to ED, 83.5% (n = 879) were screened. Of all patient screened, 71.8% (n = 631) had a positive suicide risk. Suicide risk level distribution among all screened patients was as follows: 18.9% (n = 166) were low risk, 13.3% (n = 117) were moderate risk, and 39.6% (n = 348) were high risk (Table 1). Only 0.46% of all patients screened (n = 4) left against medical advice.

Factors Associated With High Risk

After controlling for race and/or ethnicity, age group, sex, insurance status, and disposition, the adjusted odds of screening at high risk for suicide were twice as high for girls compared with boys (adjusted odds ratio [aOR] 1.93; 95% confidence interval [CI] 1.44–2.59), 32% lower for public insurance compared with private insurance (aOR 0.68; 95% CI 0.47–0.98), and twice as high in patients who were admitted compared with those who were discharged (aOR 2.04; 95% CI 1.53–2.73) (Table 2).

TABLE 2

Factors Associated With High Risk on the C-SSRS

VariableHigh Risk, aOR (95% CI)
ED  
 Age group, y  
  6–12 0.83 (0.61–1.11) 
  13–17 Referent 
 Sex  
  Female 1.93 (1.44–2.59)a 
  Male Referent 
 Insurance status  
  Public 0.68 (0.47–0.98)a 
  Self 0.55 (0.28–1.10) 
  Private Referent 
 Race and/or ethnicity  
  Non-Hispanic Black 1.17 (0.73–1.87) 
  Hispanic 1.45 (0.81–2.61) 
  Non-Hispanic white Referent 
 Disposition  
  Admitted or transferred 2.04 (1.53–2.73)a 
  Left AMA 0.59 (0.06–5.54) 
  Discharged Referent 
OPC  
 Age group, y  
  6–12 0.40 (0.16–1.08) 
  13–17 Referent 
 Sex  
  Female 3.15 (1.15–8.56)a 
  Male Referent 
 Insurance status  
  Public 0.63 (0.21–1.85) 
  Private Referent 
 Race and/or ethnicity  
  African American or Black 2.72 (0.88–8.37) 
  Asian American 2.43 (0.25–23.62) 
  White Referent 
VariableHigh Risk, aOR (95% CI)
ED  
 Age group, y  
  6–12 0.83 (0.61–1.11) 
  13–17 Referent 
 Sex  
  Female 1.93 (1.44–2.59)a 
  Male Referent 
 Insurance status  
  Public 0.68 (0.47–0.98)a 
  Self 0.55 (0.28–1.10) 
  Private Referent 
 Race and/or ethnicity  
  Non-Hispanic Black 1.17 (0.73–1.87) 
  Hispanic 1.45 (0.81–2.61) 
  Non-Hispanic white Referent 
 Disposition  
  Admitted or transferred 2.04 (1.53–2.73)a 
  Left AMA 0.59 (0.06–5.54) 
  Discharged Referent 
OPC  
 Age group, y  
  6–12 0.40 (0.16–1.08) 
  13–17 Referent 
 Sex  
  Female 3.15 (1.15–8.56)a 
  Male Referent 
 Insurance status  
  Public 0.63 (0.21–1.85) 
  Private Referent 
 Race and/or ethnicity  
  African American or Black 2.72 (0.88–8.37) 
  Asian American 2.43 (0.25–23.62) 
  White Referent 

aOR was adjusted for age, sex, race and/or ethnicity, insurance status, and suicide risk.

a

Statistically significant.

Factors Associated With Discharge

After controlling for race and/or ethnicity, age group, sex, insurance status, and level of suicide risk, the adjusted odds of being discharged were lower for moderate suicide risk (aOR 0.56; 95% CI 0.34–0.90) and high suicide risk (aOR 0.41; 95% CI 0.28–0.59). Adjusted odds of discharge were higher for patients who were self-insured (aOR 4.58; 95% 1.83–11.40) and non-Hispanic Black compared with those who were non-Hispanic white (aOR 1.72; 95% CI 1.08–2.76) (Supplemental Table 4).

Average LOS

There was no increase in average ED LOS for behavioral health patients or in the proportion of boarders, suggesting that our intervention did not prolong ED evaluation times (Fig 3).

FIGURE 3

A, ED LOS for behavioral health patients: January 2019 to December 2019. B, Proportion of ED boarders: January 2019 to December 2019. LCL, lower control limit; UCL, upper control limit.

FIGURE 3

A, ED LOS for behavioral health patients: January 2019 to December 2019. B, Proportion of ED boarders: January 2019 to December 2019. LCL, lower control limit; UCL, upper control limit.

Close modal

Compliance With Screening and Assessment

Overall compliance with completion of the C-SSRS screener was 83.5% (n = 879) (Supplemental Table 5). Compliance with full assessment for patients screening positive, which was previously not being done, was 88.0% (n = 773) The most common reason for missed assessment (n = 53) was “discharged by ED physician without social work consult.”

Patient Demographics

The majority was male (n = 316; 55.3%). The mean age was 11.2 years (SD 3.30). Unlike the ED sample, approximately one-third (33.10%; n = 189) were of non-Hispanic Black ethnicity, and the majority had private insurance (n = 344; 60.25%) (Table 1).

Proportion of Patients With Suicide Risk

Of all behavioral health patients aged 6 to 17 years presenting to the OPC for initial assessment, 65.5% (n = 374) received a C-SSRS screener. Of all screened patients, 18.2% (n = 68) had a positive suicide risk screen result. Suicide risk level distribution among all screened patients was as follows: 6.1% (n = 23) were low risk, 6.1% (n = 23) were moderate risk, and 5.9% (n = 22) were high risk (Table 1).

Factors Associated With High Suicide Risk

After controlling for age, race and/or ethnicity, and insurance status, the adjusted odds of screening at high risk for suicide were higher for girls compared with boys (aOR 3.15; 95% CI 1.15–8.56). The odds for screening high risk were not significantly different on the basis of age, insurance, or ethnicity in this sample (Table 2).

Compliance With Screening

Overall compliance with completion of the C-SSRS screener was 65%, with an initial rising trend up to 78% during the first 4 months, followed by a drop to 47% in month 5 (Supplemental Table 5).

Implementation of a standardized method of suicide risk screening and assessment in behavioral health patients by using the C-SSRS brought hospital practice in compliance with TJC standards without increase in ED LOS or boarding. Seventy-two percent of behavioral health patients presenting to the ED were identified as having a positive suicide risk, justifying the effort to improve the care of these patients. The frequency of high-risk screens was lower in the OPC (5.9%) compared with the ED (39.6%) likely because of overall lower acuity in the OPC.

Compliance with risk screening in the ED remained stable between 76% and 88%, averaging at 83.5% over a 5-month period. However, in the OPC, screening compliance dropped precipitously in month 5 after the patient care technician position became vacant, highlighting the importance of appropriate personnel.

Integration of the C-SSRS Full Assessment with risk factors into the EHR made it part of the social work workflow and standardized suicide risk assessment in the ED. The phased approach allowed for problem solving, for example, inclusion of the C-SSRS Very Young Children version in the EHR before going live in the OPC. Furthermore, aligning safety interventions and psychiatry social work evaluations to the risk level reduced the burden on mental health personnel. Anecdotally, the medical social workers reported growing comfort with use of the C-SSRS Full Assessment to guide their decision-making, and use of automated safety orders allayed provider concerns for missing safety interventions.

Delays in care and increased boarding in the ED were commonly cited concerns regarding wider screening for suicide risk.11  An important finding from this study was that the ED LOS did not increase and that the majority of patients screening positive for suicide risk were discharged from the hospital. Boarding rates also decreased after implementation, and the downtrend was sustained over time. This may indicate that conducting a comprehensive suicide risk assessment and safety planning in the ED improves provider comfort in discharging patients.31 

In both samples, female sex was associated with screening at high risk for suicide, consistent with reports of higher rates of suicidal thinking in girls.7,32,33 

There was no significant difference in suicide risk rates between non-Hispanic Black and non-Hispanic white youth in both samples; however, non-Hispanic Black youth had much higher odds of discharge from the hospital after controlling for sociodemographic risk factors and suicide risk. Further disparity was noted in patient demographics (ie, a majority of patients accessing outpatient care were white and had private insurance, whereas ED services were accessed by Black and publicly insured patients). Modifying public insurance policies by widening eligibility to slightly higher income brackets, increasing public insurance coverage of outpatient behavioral health care, and reducing co-pays might ensure that appropriate care is provided to this vulnerable population.34,35 

We did not report some key metrics to assess implementation. Measuring compliance with safety planning in phase 1 and compliance with standardized assessment in phase 2 required a chart review, which was out of the scope of this study. The presenting behavioral health chief complaint was entered as free text in the EHR; therefore, descriptive statistics could not be reliably reported. Additionally, analysis of full assessment items (ie, severity of ideation and behavior) was not conducted. Provider feedback regarding acceptability was not formally obtained.

The study has implications for hospital administrators and providers given the TJC mandate, and we find that a systematic approach can be implemented without overburdening the treatment system. The variability in practice systems and level of administration buy-in are the most likely hinderances to implementation of this process in other hospitals.

Phase 3 will expand screening and assessment to non–behavioral health patients presenting to the hospital. This will require further expansion of behavioral health services. Further research is also necessary to determine if standardized methods of risk assessment after screening will achieve the goal of reduction in national suicide rates, which was the driver for the revised NPSG 15.01.01. Given overall low rates of completed suicides, pooling data from other hospitals that have implemented similar processes may help answer this question.

We thank the members of the Children’s National Suicide Screening workgroup for their hard work on this project.

Ms Chan-Salcedo, Ms Badolato, and Dr McKinley completed initial data collection and analyses; Ms Newton designed and wrote the environmental safety checklist; Dr Robb critically reviewed the manuscript; Dr Latif drafted the initial manuscript; and all authors were involved in conceptualizing and designing the project and study, reviewed and revised the manuscript, and approved the final manuscript as submitted.

Deidentified individual participant data will not be made available.

FUNDING: No external funding.

1
Heron
M
.
Deaths: Leading Causes for 2015
.
Natl Vital Stat Rep
.
2017
;
66
(
5
):
1
76
2
Hedegaard
H
,
Curtin
SC
,
Warner
M
.
Suicide rates in the United States continue to increase
.
NCHS Data Brief
.
2018
;(
309
):
1
8
3
Ertl
A
,
Sheats
KJ
,
Petrosky
E
,
Betz
CJ
,
Yuan
K
,
Fowler
KA
.
Surveillance for violent deaths - National Violent Death Reporting System, 32 states, 2016
.
MMWR Surveill Summ
.
2019
;
68
(
9
):
1
36
4
Miller
M
,
Barber
C
,
White
RA
,
Azrael
D
.
Firearms and suicide in the United States: is risk independent of underlying suicidal behavior?
Am J Epidemiol
.
2013
;
178
(
6
):
946
955
5
Brown
GK
,
Beck
AT
,
Steer
RA
,
Grisham
JR
.
Risk factors for suicide in psychiatric outpatients: a 20-year prospective study
.
J Consult Clin Psychol
.
2000
;
68
(
3
):
371
377
6
Wenzel
A
,
Berchick
ER
,
Tenhave
T
,
Halberstadt
S
,
Brown
GK
,
Beck
AT
.
Predictors of suicide relative to other deaths in patients with suicide attempts and suicide ideation: a 30-year prospective study
.
J Affect Disord
.
2011
;
132
(
3
):
375
382
7
Bridge
JA
,
Goldstein
TR
,
Brent
DA
.
Adolescent suicide and suicidal behavior
.
J Child Psychol Psychiatry
.
2006
;
47
(
3–4
):
372
394
8
The Joint Commission
.
Suicide prevention. Available at: https://www.jointcommission.org/npsg_150101_suicide_prevention_resources/. Accessed January 23, 2020
9
The Joint Commission
.
Sentinel Event Alert 56: Detecting and Treating Suicide Ideation in All Settings
.
Oakbrook Terrace, IL
:
The Joint Commission
;
2019
10
Suicide Prevention Resource Center
.
CALM: Counseling on Access to Lethal Means
.
Waltham, MA
:
Suicide Prevention Resource Center
;
2018
11
Brahmbhatt
K
,
Kurtz
BP
,
Afzal
KI
, et al
;
PaCC Workgroup
.
Suicide risk screening in pediatric hospitals: clinical pathways to address a global health crisis
.
Psychosomatics
.
2019
;
60
(
1
):
1
9
12
Roaten
K
,
Johnson
C
,
Genzel
R
,
Khan
F
,
North
CS
.
Development and implementation of a universal suicide risk screening program in a safety-net hospital system
.
Jt Comm J Qual Patient Saf
.
2018
;
44
(
1
):
4
11
13
Habis
A
,
Tall
L
,
Smith
J
,
Guenther
E
.
Pediatric emergency medicine physicians’ current practices and beliefs regarding mental health screening
.
Pediatr Emerg Care
.
2007
;
23
(
6
):
387
393
14
Horowitz
L
,
Ballard
E
,
Teach
SJ
, et al
.
Feasibility of screening patients with nonpsychiatric complaints for suicide risk in a pediatric emergency department: a good time to talk?
Pediatr Emerg Care
.
2010
;
26
(
11
):
787
792
15
Ballard
ED
,
Bosk
A
,
Snyder
D
, et al
.
Patients’ opinions about suicide screening in a pediatric emergency department
.
Pediatr Emerg Care
.
2012
;
28
(
1
):
34
38
16
Miranda
R
,
Ortin
A
,
Scott
M
,
Shaffer
D
.
Characteristics of suicidal ideation that predict the transition to future suicide attempts in adolescents
.
J Child Psychol Psychiatry
.
2014
;
55
(
11
):
1288
1296
17
Horowitz
L
,
Bridge
JA
.
Analysis of multisource data establishes wide-ranging antecedents of youth suicide in England
.
Evid Based Ment Health
.
2017
;
20
(
3
):
e12
18
Sheftall
AH
,
Asti
L
,
Horowitz
LM
, et al
.
Suicide in elementary school-aged children and early adolescents
.
Pediatrics
.
2016
;
138
(
4
):
e20160436
19
Osman
A
,
Bagge
CL
,
Gutierrez
PM
,
Konick
LC
,
Kopper
BA
,
Barrios
FX
.
The Suicidal Behaviors Questionnaire-Revised (SBQ-R): validation with clinical and nonclinical samples
.
Assessment
.
2001
;
8
(
4
):
443
454
20
Serrani Azcurra
D
.
Psychometric validation of the Columbia-Suicide Severity Rating Scale in Spanish-speaking adolescents
.
Colomb Med (Cali)
.
2017
;
48
(
4
):
174
182
21
Horowitz
LM
,
Bridge
JA
,
Teach
SJ
, et al
.
Ask Suicide-Screening Questions (ASQ): a brief instrument for the pediatric emergency department
.
Arch Pediatr Adolesc Med
.
2012
;
166
(
12
):
1170
1176
22
Posner
K
,
Brown
GK
,
Stanley
B
, et al
.
The Columbia-Suicide Severity Rating Scale: initial validity and internal consistency findings from three multisite studies with adolescents and adults
.
Am J Psychiatry
.
2011
;
168
(
12
):
1266
1277
23
Steer
RA
,
Kumar
G
,
Beck
AT
.
Self-reported suicidal ideation in adolescent psychiatric inpatients
.
J Consult Clin Psychol
.
1993
;
61
(
6
):
1096
1099
24
Beck
AT
,
Brown
GK
,
Steer
RA
.
Psychometric characteristics of the Scale for Suicide Ideation with psychiatric outpatients
.
Behav Res Ther
.
1997
;
35
(
11
):
1039
1046
25
Flamarique
I
,
Santosh
P
,
Zuddas
A
, et al
;
STOP consortium
.
Development and psychometric properties of the Suicidality: Treatment Occurring in Paediatrics (STOP) Suicidality Assessment Scale (STOP-SAS) in children and adolescents
.
BMC Pediatr
.
2016
;
16
(
1
):
213
26
The Columbia Lighthouse Project
.
Free training for communities and healthcare. Available at: http://cssrs.columbia.edu/training/training-options/. Accessed January 28, 2020
27
Crisis Prevention Institute
.
28
The Columbia Lighthouse Project
.
Columbia Suicide Severity Rating Scale Very Young Children. Available at: https://cssrs.columbia.edu/the-columbia-scale-c-ssrs/cssrs-for-communities-and-healthcare/#filter=.general-use.english. Accessed xx xx, xxxx
29
Johnson
RM
,
Frank
EM
,
Ciocca
M
,
Barber
CW
.
Training mental healthcare providers to reduce at-risk patients’ access to lethal means of suicide: evaluation of the CALM Project
.
Arch Suicide Res
.
2011
;
15
(
3
):
259
264
30
Children’s Hospital Association
.
31
Baca-García
E
,
Diaz-Sastre
C
,
Resa
EG
, et al
.
Variables associated with hospitalization decisions by emergency psychiatrists after a patient’s suicide attempt
.
Psychiatr Serv
.
2004
;
55
(
7
):
792
797
32
Goldman-Mellor
S
,
Olfson
M
,
Lidon-Moyano
C
,
Schoenbaum
M
.
Association of suicide and other mortality with emergency department presentation
.
JAMA Netw Open
.
2019
;
2
(
12
):
e1917571
33
Boeninger
DK
,
Masyn
KE
,
Feldman
BJ
,
Conger
RD
.
Sex differences in developmental trends of suicide ideation, plans, and attempts among European American adolescents
.
Suicide Life Threat Behav
.
2010
;
40
(
5
):
451
464
34
Lê Cook
B
,
Barry
CL
,
Busch
SH
.
Racial/ethnic disparity trends in children’s mental health care access and expenditures from 2002 to 2007
.
Health Serv Res
.
2013
;
48
(
1
):
129
149
35
Cook
BL
,
Trinh
NH
,
Li
Z
,
Hou
SSY
,
Progovac
AM
.
Trends in racial-ethnic disparities in access to mental health care, 2004-2012
.
Psychiatr Serv
.
2017
;
68
(
1
):
9
16

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data