Positive Predictive Value of ICD-10 Codes to Identify Acute Suicidal and Self Harm Behaviors Free

We performed a secondary analysis using data from an observational study that evaluated a broad range of neuropsychiatric events, including suicidal behaviors.⁶  We performed a cross-sectional study of children 5 to 17 years of age hospitalized at 2 US children’s hospitals in [Nashville, TN and Aurora, CO] between April 1, 2016 and March 30, 2020 with a discharge ICD-10 diagnosis of a self-harm event. Cases of intentional self-harm, nonsuicidal self-injury, suicidal ideation, and suicide attempts were included in the study. We included both primary and secondary diagnoses of a self-harm event. For example, hospitalization with a primary diagnosis of ingestion or trauma with a secondary diagnosis of a self-harm event would be included in the study. The study included those transferred to psychiatric facilities and those discharged from the hospital. In-hospital deaths were excluded from the study. Individuals discharged from the emergency department were excluded because of concerns regarding misclassification of diagnosis codes. Individuals with multiple hospitalizations were included in the study.

A total of 376 encounters with a SI or SH diagnosis were included in this study. For reference, there were a total of 6312 self-harm encounters at the 2 study sites, and our random sample of 376 self-harm encounters in this study represents 6.0% of all self-harm encounters over the study period. Medical records were manually reviewed for each encounter. Data extracted for analysis included demographics, race and ethnicity, and concurrent diagnoses. Demographic data were included for descriptive purposes only.

To determine a true SI or SH event, a pediatrician reviewed each case identified to determine if there was a qualifying SI or SH event documented in the medical record (history and physical, progress and procedure notes, or discharge summaries). If an event was present, the physician reviewers determined if the event was (1) present on admission and related to hospitalization (eg, suicide attempt), (2) not present on admission but active hospital problem (eg, SI after prolonged hospitalization or ICU stay), or (3) a problem present on admission but not related to hospitalization (eg, history of SI or SH in an adolescent with depression hospitalized for unrelated reasons). Reviewers were blinded to the qualifying diagnoses for each encounter. A true positive was defined as an active hospital problem present on admission and related to hospitalization.

The PPV with 95% confidence interval SI or SH diagnosis was calculated overall and also stratified by groups: diagnosis order (primary SI or SH diagnosis, secondary SI or SH diagnosis, or any secondary diagnosis with a primary psychiatric diagnosis) and age (5 to 11 years vs 12 to 17 years). This study was approved by the Vanderbilt University Medical Center Institutional Review Board.

There were 376 children identified with an SI or SH hospitalization. The median age was 14 years, and the majority of individuals were female (58%), non-Hispanic White (69%), and privately insured (57%) (Table 1). Of these, 96 had an SI or SH primary diagnosis, 280 had SI or SH as a secondary diagnosis, and 244 had a secondary SI or SH diagnosis with a primary psychiatric diagnosis. A total of 332 confirmed SI or SH were identified, with a PPV of 0.88 (confidence interval 0.85–0.91). PPVs were similar when stratified by diagnosis order and age (Table 2).

Among the 44 false positive cases, 42 of the diagnoses were not directly related to hospitalization (eg, past diagnosis of SI), 1 was present on admission but not related to hospitalization (eg, endorsed SI but admitted for appendicitis), and 1 was not present on admission but an active hospital problem (eg, admitted for severe burns and developed depression and SI during prolonged hospitalization).

In this retrospective study, we determined the accuracy of ICD-10 codes to identify hospitalizations for suicidal and self-harm events. Accurate identification of these events in children is important to minimize misclassification when using administrative claims data, as well as developing and evaluating interventions to improve outcomes in children. The authors of a 2019 review of 34 adult and pediatric studies validating International Classification of Disease, 9th Revision codes identified several potential algorithms for capturing SI events. In pediatric-specific studies in the review, PPVs ranged from 43.4% to 88.3% in identifying SI in children and adolescents.⁴  Studies using diagnosis codes alone without hospitalization or exclusionary diagnosis requirements may have resulted in misclassification and inaccurate estimates of SI or SH behavior.^4,5 

The authors of the current study evaluated the widely adopted ICD-10 classification system⁷  to identify suicidal and self-harm behaviors. Utilizing primary diagnoses alone, compared with the inclusion of secondary SI or SH diagnoses, increased the PPV while substantially reducing the sample size of true positives identified. There were no significant differences in PPV when stratifying by age. Recent studies have revealed a spike in suicidal and self-harm behaviors in children 12 to 18 years of age, and accurately identifying these cases as targets for intervention could address public health concerns.⁸ 

PPV, rather than specificity, in identifying an outcome is most critical when evaluating the relationship between exposure and outcome.⁹  Our study findings may inform future studies to investigate factors related to SH and SI hospitalizations in children, such as infections,¹⁰  medications,¹¹  or underlying conditions.^12–14  With such a robust PPV, investigators and public health professionals relying on this set of ICD-10 codes to identify SI and SH can be confident that the identified hospitalizations represent true cases of SI or SH that were present on admission and directly related to hospitalization. However, among the false positive cases, the most likely reason for the misclassification of cases was carryover diagnoses, meaning diagnoses that were present on admission but not directly related to hospitalization. The authors of future studies may focus on enhancing the identification of cases through the use of algorithms incorporating data elements present in the electronic health record.^6,15 

In our study, we were not able to determine if this set of ICD-10 codes missed any SI or SH events. Therefore, we could not determine the sensitivity or negative predictive value of the ICD-10 codes. Applying these codes for epidemiologic or surveillance studies should be used with caution given that the amount of self-harm hospitalizations that are missed using these codes is unknown. The authors of future studies should focus on determining measures of sensitivity and negative predictive value to better inform real-world studies using ICD-10 codes.

Study limitations included the use of data from only 2 children’s hospitals, which may limit the generalizability of our findings, although these hospitals were from different geographic and demographic areas of the United States. Additionally, we did not assess SI or SH in the ambulatory or emergency department setting or events that did not result in health care encounters, limiting applicability to milder cases. Finally, we did not include those who died in the hospital, and our study may have missed patients with severe injuries resulting in out-of-hospital and in-hospital death. For example, an individual with a completed suicide attempt (such as a self-inflicted firearm injury) would not have been included in this evaluation.

In summary, hospitalizations for serious suicidal and self-harm behaviors in children and adolescents can be accurately identified by using ICD-10 discharge diagnosis codes. The use of these validated code sets when using real-world data may minimize misclassification in identifying SI or SH events. Future studies may use these findings to further investigate factors related to SI or SH hospitalizations in children, such as clinical and patient characteristics.¹⁴  These ICD-10 codes may be especially useful for evaluating real-world effectiveness of interventions when using administrative or claims data, electronic health records, or large observational databases.: