Video Abstract

Video Abstract

Close modal
BACKGROUND AND OBJECTIVES:

Patient and family violent outbursts toward staff, caregivers, or through self-harm, have increased during the ongoing behavioral health crisis. These health care-associated violence (HAV) episodes are likely under-reported. We sought to assess the feasibility of using nursing notes to identify under-reported HAV episodes.

METHODS:

We extracted nursing notes across inpatient units at 2 hospitals for 2019: a pediatric tertiary care center and a community-based hospital. We used a workflow for narrative data processing using a natural language processing (NLP) assisted manual review process performed by domain experts (a nurse and a physician). We trained the NLP models on the tertiary care center data and validated it on the community hospital data. Finally, we applied these surveillance methods to real-time data for 2022 to assess reporting completeness of new cases.

RESULTS:

We used 70 981 notes from the tertiary care center for model building and internal validation and 19 332 notes from the community hospital for external validation. The final community hospital model sensitivity was 96.8% (95% CI 90.6% to 100%) and a specificity of 47.1% (39.6% to 54.6%) compared with manual review. We identified 31 HAV episodes in July to December 2022, of which 26 were reportable in accordance with the hospital internal criteria. Only 7 of 26 cases were reported by employees using the self-reporting system, all of which were identified by our surveillance process.

CONCLUSIONS:

NLP-assisted review is a feasible method for surveillance of under-reported HAV episodes, with implementation and usability that can be achieved even at a low information technology-resourced hospital setting.

What’s Known on This Subject:

Health care-associated violence negatively impacts both patients, families, and health care workers. Escalating during and after the coronavirus disease 2019 pandemic, and with current behavioral health crisis, violence reporting is not mandatory, and surveillance is limited to self-reporting and occupational health injury logs.

What This Study Adds:

We provide evidence that health care-associated violence episodes are recorded in nursing hand-off notes and can be mined using natural language processing models. Furthermore, we provide proof of concept that a thin and effective framework can be transferrable to non-academic settings.

Verbal and physical violence directed toward health care workers (HCWs) have reached concerning levels worldwide. A systematic review and meta-analyses published at the end of 2019 found a high prevalence of workplace violence by patients and visitors against nurses and providers.1 In this review, which included 253 eligible studies (with a total of 331 544 participants), 61.9% of the participants reported exposure to any form of workplace violence, 42.5% reported exposure to nonphysical violence, and 24.4% experienced physical violence in the past year.2 Verbal abuse (57.6%) was the most common form of nonphysical violence, followed by threats (33.2%) and sexual harassment (12.4%). The prevalence of violence against HCWs was particularly high in Asian and North American countries. Our study is United States based, yet violence directed toward HCWs is not unique to the United States. In Germany, severe aggression or violence has been experienced by 23% of primary care providers.3 The consequences of violence against HCWs can be very serious: deaths or life-threatening injuries, reduced work interest, job dissatisfaction, decreased retention, more leave days, impaired work functioning,4 depression, post-traumatic stress disorder,5 decline of ethical values, and increased practice of defensive medicine.6 Workplace violence is associated directly with higher incidence of burnout, risks to patient safety, and more frequent adverse events.7 

HCWs experiencing violence develop a variety of secondary psychiatric and stress-related conditions affecting their personal lives and professional performance.4,6–8 Escalating during the coronavirus disease 2019 (COVID-19) pandemic9,10 and continuing during the post-COVID-19 era,11 hospitals are burdened with a behavioral health crisis typified by patients boarding in emergency departments (EDs)12–15 and inpatient beds,16,17 all of which adds stress to both patients and HCWs.

We define an episode of health care-associated violence (HAV) as an intentional action by a patient or caregiver in the setting of care delivery with potential to cause harm or insult, to self or to others. This definition is consistent with available regulations regarding the reporting of workplace violence events in the health care setting, and with the Joint Commission definition of workplace violence that went into effect January 1, 2022: An act or threat occurring at the workplace that can include any of the following: verbal, nonverbal, written, or physical aggression; threatening, intimidating, harassing, or humiliating words or actions; bullying; sabotage; sexual harassment; physical assaults; or other behaviors of concern involving staff, licensed practitioners, patients, or visitors. HAV episodes are not included in the hospital-acquired conditions (HACs)18 that are a primary focus of patient safety efforts. However, HCW safety is an important and recognized aspect of a safe health care environment.19 Providers may report these events to occupational health if injured, and employees are encouraged to report these events through dedicated hospital safety reporting systems when available. However, in general, data related to self-reported clinical events reveal frequent under-reporting,20–22 suggesting that HAV episodes are also likely under-reported.

Artificial intelligence (AI) techniques, such as machine learning, can provide clinical risk prediction to improve patient safety. Most AI projects identify or mitigate health care harms such as: health care-associated infections,23 adverse drug events,24 venous thromboembolism,25 surgical complications,26 pressure ulcers,27,28 falls,25 insufficient decompensation detection,29 and diagnostic errors—including missed and delayed diagnoses.30,31 Many of these events have structured data elements that can be easily retrieved, such as culture results for central line infections or imaging studies for venous thromboembolisms. Unstructured data, such as text or clinical narrative, provides granular data of some events and at times is the only data source other than self-reporting logs, as in the case of peripheral intravenous infiltration or pressure injury surveillance.32,33 To the best of our knowledge, HAV episode data are primarily derived from self-reporting or survey data, rather than AI or natural language processing (NLP)-based surveillance.

We hypothesize that text-based clinical notes, particularly nursing notes, are a rich data source for identification of HAV episodes. We chose nursing documentation as we consider nurses to be the HCW that spend the most time with patient and family during inpatient stay, as well as being a major target or victim of HAV episodes. These events are likely documented by frontline staff to describe the patient’s mindset and distress, and to provide situational awareness to other providers of possible volatile interactions with patients, families caregivers, and visitors.

We further hypothesize that these events can be identified by a NLP-assisted process,34 one we have used successfully for clinical events35–38 and patient safety event surveillance.32,33 We have shown in prior work that an NLP-assisted process, implemented by a provider with medical expertise but no computer science or clinical informatics expertise, effectively and efficiently identified specific clinical events of interest.38–40 In this setting, we aim to establish a surveillance dashboard to monitor incidence of HAV episodes, leveraging an NLP model that is continuously retrained with updated narrative data and enhanced with additional structured data elements.

We sought to assess the feasibility and performance of an NLP-assisted process to screen nursing notes in an effort to improve identification and surveillance of HAV episodes, particularly unreported events, in the inpatient hospital setting.

We performed a retrospective cross-sectional study of consecutive patients admitted at 2 hospitals: an urban academic tertiary care pediatric hospital and an urban general community-based hospital that serves both adults and children. Our main outcome was a documented HAV episode. We built, trained, and internally validated an NLP model using the tertiary care data; then, we externally validated and established proof-of-transferability using data from the community hospital. Finally, we implemented a prospective surveillance system for July 2022 to December 2022 at the community hospital. Violence against staff is reported internally via the Quantors© system. We reported identified HAV episodes to hospital leadership and cross-referenced our surveillance cases with reported incidents.

The study protocol was approved by our institution’s Institutional Review Board, contingent on a requirement from the tertiary hospital to use data that were at least 3 years old for added privacy protection. We, therefore, reviewed electronic health record data of patients admitted to both sites from January 2019 to December 2019 for training and validation. Our Institutional Review Board approval at the community hospital allowed use of current data.

We included notes of all patients <21 years of age admitted to inpatient units, including the PICU. For the community hospital, which does not have a PICU, we included NICU notes to mirror our tertiary care cohort by including patients with high clinical acuity or severity and high stress for families and caregivers. Although patients in the PICU and NICU have different clinical characteristics, in both units the medical condition of the patients precludes many patients from being the main actor in HAV episodes.

We have obtained nursing documentation that included: handoff documentation, pain treatment reports, and event reports. In total, 70 981 nursing notes from the tertiary care center were available for review, representing 5484 unique admissions involving 4220 patients. These notes were used to build a model and to validate it internally. For the community hospital data, we had 19 332 nursing handoff notes, representing 3670 unique admissions involving 2827 patients. These notes were used to validate the model with external data. All notes were extracted from hospital electronic health records (EHRs) into our processing platform to assure that the process is EHR platform independent, as 1 center uses CRNER and the other uses EPIC.

Case identification was conducted in 3 phases: (1) establishing a training set for document classifier, while at the same time harnessing domain expertise; (2) creating and applying an N-gram NLP models and manually reviewing their output; and (3) combining the methods into 1 cohesive model. The initial screening of cases was performed by a student and an experienced registered nurse (J.B., K.F.), and 7% of the data were reviewed by an experienced Pediatric and Pediatric Emergency Medicine specialist (A.K.), to assure adequate inter-rater agreement.

Our primary outcome is an HAV episode documented by a HCW in the clinical notes. An example of a “positive” case, ie, an HAV episode, is documentation of a patient throwing a urine sample at a nurse. An example of a “negative” case would be documentation of a patient violently thrashing while emerging from medical sedation.

A preliminary review of collected notes revealed that most episodes were documented in brief and unequivocal language, without granular data, eg, “assaulting security” or “went on to kicking this nurse.” To assess subjectivity in the ascertainment of outcomes, we adjudicated 7% of positive cases by a second reviewer to calculate inter-rater agreement. Since our outcome is an event rather than a document describing an event, we consolidated documents describing the same event to identify each unique HAV episode.

Since HAV episodes subcategories are not mutually exclusive, we devised a subcategory hierarchy. For a verbal threat and physical assault occurring during the same interaction, we considered physical assault to be higher on the scale and, therefore, this HAV episode as assault. We considered throwing bodily fluids (urine or blood specimens) and spitting to be physical assault, as this act of violence exposes HCWs to the risk of encountering infections.

Post hoc, after an exploratory manual review of 150 potential events, we added additional subcategories to include cases that are informative or important for safe delivery of health care, even when harm to a HCW was not part of the event. These include HAV episode acted toward other family caregivers and/or involving self-harm, and inappropriate or concerning family-related events (for example a parent using corporal punishment on a sibling of an admitted child). Although there may some subjectivity in these secondary categories, we felt it helpful to flag them for surveillance follow-up with conviction that if the provider felt an event was important enough to convey an event in documentation (for colleagues, for keeping record), it is likely important enough for safety surveillance.

A detailed description of our NLP-assisted review process is published in prior work [cite] and depicted with a workflow map (Fig 1). Very briefly, we created a computer-assisted screening tool called Document Review Tool (DrT)41 with Graphical User Interface (GUI) to accelerate and scale the human review of clinical documents (Fig 2). The use of DrT by clinical chart reviewers, who typically have no formal training in NLP or computer science, produces a training set of documents that are manually labeled as positive or negative cases of HAV events. DrT incorporates tools from NLP, such as Regular Expressions (RegEx) and distributional semantics. For example, in a clinical note the word “security” may be strongly associated with other terms such as “officer,” “restrain,” and “BRT” (Behavioral Response Team). We illustrate the list of distributional semantics terms based on the term “security” (Fig 3).

FIGURE 1

Workflow of training and validating NLP models by a lay NLP user.

FIGURE 1

Workflow of training and validating NLP models by a lay NLP user.

Close modal
FIGURE 2

RegEx-assisted document labeling.

FIGURE 2

RegEx-assisted document labeling.

Close modal
FIGURE 3

Distributional semantics assisted document labeling, building new RegEx.

FIGURE 3

Distributional semantics assisted document labeling, building new RegEx.

Close modal

The initial data set is used to train an N-gram classifier and Support Vector Machine (SVM) model (Fig 4), which is applied to the larger set of clinical notes to predict which notes document HAV events. Documents that have a predicted probability above a specified threshold are subjected to additional human review for case ascertainment.

FIGURE 4

List of RegEx.

For the note classifier, we used an N-gram SVM to generate a continuous score for each document describing a potential HAV episode. We used individual RegEx expressions as predictors in a binary logistic regression model to identify documents likely to indicate a HAV event. We evaluated 2 methods to combine the RegEx binary logistic regression model and the SVM model: a random forest model, and a joint scoring model in which a positive prediction from either the RegEx model or the SVM model is considered a positive case.

We used Bayesian credible intervals to calculate percentages and confidence intervals for the prevalence of HAV episodes, and a κ statistic with 95% confidence interval to assess inter-rater agreement. Data were analyzed using IBM SPSS for Windows V27 (Chicago, 2019).

We sought a model with high sensitivity (lower boundary of 95% confidence interval >90%), and thus we manually reviewed a minimum of 100 notes flagged as positive. Of these, 60% were used for training of a model and 40% for internal validation. Notes flagged by different RegEx were used to assure variability in the identification method. In the first stage of screening, our output flagged 8395 notes (11.8% of total) for manual review. Our use of RegEx made manual review of this substantial number of notes feasible, and we identified 324 notes (0.5% of total) as probable notes that were eventually mapped to 34 events.

Inter-rater agreement between 2 reviewers classifying notes as positive cases was substantial (κ = 0.62, 95% confidence interval 0.53–0.74).

For the N-gram classifier model, we specified a cutoff score that achieved a sensitivity >95%.

Using RegEx as predictors of a positive note, we ran a binary logistic regression and estimated an adjusted odds ratio for each RegEx, then calculated the receiver operating characteristic area under the curve as an additional measure of model performance (Table 1). The joint model demonstrated the best sensitivity (Table 2).

TABLE 1

The Final List of RegEx Used to Flag Potential HAV Episodes

List of Regular Expressions (RegExp)
\bbang\w*\b \bthreat\w*\b \bslur\w*\b 
against\s*staff|toward[s]?\s*staff \bagitat\w*\b \w*PPV|\bvent\b|pacifier 
\bspit\w*\b \bthr[eo]w\w*\b infant\w*|newborn|baby|pacifier|pump|delivery|\bnbs\b 
\bscream\w*\b \bescalat\w*\b \baggress\w*\b 
\bpunch\w*\b \bsettle\w*\b \bh[ie][rs]\b\s(mom|nurse|RN|dad) 
(\bhit\w*\b)|(\bscratch\w*\b) \bat\b\s*(\bh[ie][rs]\b\s)?(staff|security|nurse|\bRN\b|mom|dad|monther|father) \bgrab\w*\b 
\bkick\w*\b|\bbites?\b|biting|\bbitt en\b \brestraint\w*\b \bthrash\w*\b 
security \bgrab\w*\b \binappro\w*\b 
\byell\w*\b \btowards?\b  
List of Regular Expressions (RegExp)
\bbang\w*\b \bthreat\w*\b \bslur\w*\b 
against\s*staff|toward[s]?\s*staff \bagitat\w*\b \w*PPV|\bvent\b|pacifier 
\bspit\w*\b \bthr[eo]w\w*\b infant\w*|newborn|baby|pacifier|pump|delivery|\bnbs\b 
\bscream\w*\b \bescalat\w*\b \baggress\w*\b 
\bpunch\w*\b \bsettle\w*\b \bh[ie][rs]\b\s(mom|nurse|RN|dad) 
(\bhit\w*\b)|(\bscratch\w*\b) \bat\b\s*(\bh[ie][rs]\b\s)?(staff|security|nurse|\bRN\b|mom|dad|monther|father) \bgrab\w*\b 
\bkick\w*\b|\bbites?\b|biting|\bbitt en\b \brestraint\w*\b \bthrash\w*\b 
security \bgrab\w*\b \binappro\w*\b 
\byell\w*\b \btowards?\b  
TABLE 2

Tertiary Care Center Model Performance Using Internal Validation

ModelSensitivitySpecificityAccuracyROC Area Under the Curve
SVM – support vector machine 97.0% (91.4%–100%) 63.3% (55.7%–73.9%) 73.5% (65.3%–81.6%) 0.93 
RegEx binary logistic regression 94.1% (86.2%–100%) 76% (66.5%–85.4%) 81.42% (74.2%–88.6%) 0.91 
Joint model score (cutoff = 1 [either or]) 100% (87.4% –99.7%) 52% (40.4%–63.2%) 66.4% (57.7%–75.1%) 0.91 
ModelSensitivitySpecificityAccuracyROC Area Under the Curve
SVM – support vector machine 97.0% (91.4%–100%) 63.3% (55.7%–73.9%) 73.5% (65.3%–81.6%) 0.93 
RegEx binary logistic regression 94.1% (86.2%–100%) 76% (66.5%–85.4%) 81.42% (74.2%–88.6%) 0.91 
Joint model score (cutoff = 1 [either or]) 100% (87.4% –99.7%) 52% (40.4%–63.2%) 66.4% (57.7%–75.1%) 0.91 

ROC, receiver operating characteristic.

Next, we applied the models developed in the first phase to the community hospital data. We used each model without any adjustment, and then recalculated performance after optimizing the cutoff (Table 3).

TABLE 3

External Validation of NLP Model at the Community Hospital Site

ModelSensitivitySpecificityAccuracyROC Area Under the Curve
SVM – using tertiary care model without adjustment 67.7% (51.3%–84.2%) 87.8% (82.9%–92.7%) 84.7% (79.8%–89.7%) 0.85 
SVM – adjusted cutoff 96.8% (90.6%–100%) 47.1% (39.6%–54.6%) 56.7% (47.8%–61.5%) — 
RegEx regression model without adjustment 74.3% (59.8%–88.8%) 82.1% (76.4%–87.9%) 80.1% (75.4%–86.2%) 0.86 
RegEx regression model adjusted cutoff 94.3% (86.6%–100%) 49.0% (41.8%–57.0%) 57.1% (50.3%–64%) — 
Joint scoring system (cutoff 1 [either or]) 100% (86.3%–99.7%) 33.1% (26.3%–40.8%) 43.5% (36.5%–50.2%) 0.87 
ModelSensitivitySpecificityAccuracyROC Area Under the Curve
SVM – using tertiary care model without adjustment 67.7% (51.3%–84.2%) 87.8% (82.9%–92.7%) 84.7% (79.8%–89.7%) 0.85 
SVM – adjusted cutoff 96.8% (90.6%–100%) 47.1% (39.6%–54.6%) 56.7% (47.8%–61.5%) — 
RegEx regression model without adjustment 74.3% (59.8%–88.8%) 82.1% (76.4%–87.9%) 80.1% (75.4%–86.2%) 0.86 
RegEx regression model adjusted cutoff 94.3% (86.6%–100%) 49.0% (41.8%–57.0%) 57.1% (50.3%–64%) — 
Joint scoring system (cutoff 1 [either or]) 100% (86.3%–99.7%) 33.1% (26.3%–40.8%) 43.5% (36.5%–50.2%) 0.87 

ROC, receiver operating characteristic.

Since our IRB approval at the community hospital was free of the 3-year gap, we were able to apply the adjusted model to inpatient notes for the second half of 2022. We screened 133 363 notes and identified 40 notes (0.3% of total) describing 31 events among 14 different patient encounters and 9 subcategories of event type (Table 4). The 6 most frequent event categories meet community hospital criteria for self-reporting. Inappropriate family interaction examples included: parents being intoxicated, physical or verbal abuse toward patient or siblings, or assisting patients who elope from involuntary admission.

TABLE 4

Categories of Episodes Captured

Event TypeNumber of Events Captured by NLP ScreenNumber of Self-reported EventsNumber of Self-reported Events Not Identified by NLP Screen
Physical assault on staff (not specified) 
Verbal assault staff (not specified) 
Escalation requiring medical or chemical restraints 
Escalation requiring physical restraints 
Physical assault nurse 
Verbal assault nurse 
Assaulting family members during hospital stay NA NA 
Inappropriate family interaction NA NA 
Physical harm to self NA NA 
Violent events not mapped to a patient encounter NA NA 
Event TypeNumber of Events Captured by NLP ScreenNumber of Self-reported EventsNumber of Self-reported Events Not Identified by NLP Screen
Physical assault on staff (not specified) 
Verbal assault staff (not specified) 
Escalation requiring medical or chemical restraints 
Escalation requiring physical restraints 
Physical assault nurse 
Verbal assault nurse 
Assaulting family members during hospital stay NA NA 
Inappropriate family interaction NA NA 
Physical harm to self NA NA 
Violent events not mapped to a patient encounter NA NA 

NA, not applicable.

Our prospective surveillance study demonstrated a positive predictive value of 29%. Most false positives were because of the model incorrectly flagging cases, description of a past, or nonspecified patient agitation.

We assessed the workload for routine and ongoing surveillance of HAV episodes. If we assumed standard nursing shifts to be 10 to 12 hours, resulting in a maximum of 5 nursing notes per patient every 48 hours, and HAV episode containing note prevalence of 0.3%, a human reviewer will identify 1 to 2 cases for each 5 notes flagged. Of note, our system allows specification of cutoffs based on other metrics, such as positive predictive values, that would allow an administrator to customize the level of manual review with corresponding trade-off against sensitivity.

During the study period there were 11 inpatient events logged into the system. Seven events, involving 3 patients, had corresponding documentation within the EHR (ie, description within the nurse’s notes), all captured by our system. These data support our hypothesis that HAV episodes are often documented in nursing notes.

Of the remaining 4 events, 2 involved parents, 1 involved a visitor, and 1 involved a staff member; these events were not linked to a specific patient medical record number or encounter number, and so it was not possible to assess whether they match cases identified in our cohort. We had 22 reportable cases identified in our NLP screen. After subtracting the 7 reported or known cases, we are left with 15 cases mapped to encounters on the NLP arm and 4 self-reported cases that are not mapped an encounter. Thus, at a minimum, we more than doubled our verified cases of HAV episodes for 2022.

We present a novel method to perform surveillance of HAV episodes in a hospital setting, providing proof-of-concept that nursing notes contain data that can identify these events. Furthermore, we demonstrate that provider-driven NLP-assisted modeling is both feasible and transferrable and can be applied in settings similar to our community hospital validation site that has low-IT resources. Finally, we provided some evidence that these events are under-reported in 1 of the systems where reporting action is available to providers.

Our surveillance activity focused on nursing notes, since data suggest that nurses are most often the victims of HCW assault.42,43 In a 2021 meta-analysis of ED data, Aljohani et al have shown that among the ED workers involved in violence against HCW, 2112 (36.5%) were physicians, 3225 (55.7%) were nurses, and 455 (7.8%) other ED staff. This is also why our study focused on inpatient units rather than EDs. In the ED setting, physician notes were typically complete documents, whereas nursing narratives often consisted of brief comments and short sentences within structured forms rather than generating a formal document with detailed narrative (this held true across 2 different EHRs). A modified surveillance process for the ED setting is currently under development.

Although surveillance of HAV episodes can be achieved in part by using structured data elements, such as self-reporting logs, physical and chemical restraint orders, and occupational health logs, it is likely limited and biased. For example, restraint orders are limited to severe acts of violence and do not cover visitor and family assault, whereas occupational health reports likely will not capture assaults resulting in minor physical injuries or throwing specimens at HCWs, events which can result in emotional distress yet no major physical harm. For these reasons, accessing clinical notes as a novel data source for HAV episode surveillance, and using NLP as a method to extract information from these data, represents an important enhancement to existing surveillance approaches.

We foresee the use of such a system as an approach for enhanced surveillance. We can envision a system administrator (nurse, patient safety leadership) using this NLP tool to identify cases that merit further safety review and/or prompting a self-report to provide further details. Action plans for specific episodes will rely on the type of episode. Appropriate responses might include communication and resources to assure provider physical and emotional wellbeing, identification of high-risk settings or risk factors associated with HAV events, or development of protocols and processes to mitigate harm. Further study of the patient safety factors associated with HAV episodes is beyond the scope of this paper, yet improved surveillance data make these studies possible and thus represent our vision for this system.

Our study has several limitations: first, as a retrospective screen, the data lag hours and possibly days after the event itself. This is a limitation of surveillance based on self-report as well, given that HCWs will often continue to provide care until the end of shift. Second, the positive predictive value of our models requires manual review of the output. Our approach involves machine learning, which means its performance will improve as more human reviewed output is provided to the system through continuous training. Although this represents a modest ongoing investment of time and resources, manual review and debriefing after reports are not unique to our surveillance approach, and continual safety and improvement efforts are expected in any system intending to deliver high quality care. The investment to operate our system and GUI in fact makes routine quality improvement efforts more efficient, thus offering a potential for a net reduction in resource costs. Finally, not all HAV episodes are equal. A visitor assaulting an HCW because of frustration with waiting time presents a different safety risk from a behavioral health teenage patient forcibly pushing providers in an attempt to elope. We feel that both events are relevant to the safety culture and environment, and thus of value to safety leadership. More complete reporting of these events also provides opportunities to mitigate harm to the provider affected and address HCW physical and emotional well-being.

We are currently developing a surveillance system specific to the ED setting, a refined system for self-harm events, and methods to expand our approach beyond English language EHRs.

Our findings suggest that nurses are documenting elements they feel are pertinent to safe care, yet these events remain underreported in the existing safety infrastructure, similar to under-reporting of patient safety events and medical errors.20,22,38–40 HAV may contribute to undetected staff harm and burnout, and enhanced surveillance using NLP may help to identify strategies to improve reporting, advance the study of HAV events, and ultimately to improve safety for patients and staff. As the field of NLP is making huge strides forward with new AI techniques, implementation should not lag behind.

Drs Waltzman, Milliren, and Ozonoff conceptualized and designed the study, and drafted the initial manuscript; Dr Kimia and Mr Landschaft designed the data collection instruments, including natural language processing-based graphical user interface, supervised the data collection, and conducted the initial analyses; Ms Fournier and Mr Bulis did the manual review of data and assessment for inter-rater agreement; and all authors critically reviewed and revised the manuscript, approved the final manuscript as submitted, and agree to be accountable for all aspects of the work.

COMPANION PAPER: A companion to this article can be found online at https://www.pediatrics.org/cgi/doi/10.1542/peds.2024-066108.

GUI

graphical user interface

HAV

health care-associated violence

HCW

health care workers

NLP

natural language processing

RegEx

regular expressions

SVM

support vector machine

1
Phillips
JP
.
Workplace violence against health care workers in the United States
.
N Engl J Med
.
2016
;
375
(
7
):
e14
2
Liu
J
,
Gan
Y
,
Jiang
H
, et al
.
Prevalence of workplace violence against healthcare workers: a systematic review and meta-analysis
.
Occup Environ Med
.
2019
;
76
(
12
):
927
937
3
Vorderwülbecke
F
,
Feistle
M
,
Mehring
M
,
Schneider
A
,
Linde
K
.
Aggression and violence against primary care physicians—a nationwide questionnaire survey
.
Dtsch Arztebl Int
.
2015
;
112
(
10
):
159
165
4
Magnavita
N
,
Heponiemi
T
,
Chirico
F
.
Workplace violence is associated with impaired work functioning in nurses: an Italian cross-sectional study
.
J Nurs Scholarsh
.
2020
;
52
(
3
):
281
291
5
D’Ettorre
G
,
Pellicani
V
,
Vullo
A
.
Workplace violence against healthcare workers in emergency departments. A case-control study
.
Acta Biomed
.
2019
;
90
(
4
):
621
624
6
Toraldo
DM
,
Vergari
U
,
Toraldo
M
.
Medical malpractice, defensive medicine and role of the “media” in Italy
.
Multidiscip Respir Med
.
2015
;
10
(
1
):
12
7
Berlanda
S
,
Pedrazza
M
,
Fraizzoli
M
,
de Cordova
F
.
Addressing risks of violence against healthcare staff in emergency departments: the effects of job satisfaction and attachment style
.
BioMed Res Int
.
2019
;
2019
:
5430870
8
D’Ettorre
G
,
Pellicani
V
,
Mazzotta
M
,
Vullo
A
.
Preventing and managing workplace violence against healthcare workers in emergency departments
.
Acta Biomed
.
2018
;
89
(
4-S
):
28
36
9
Ramzi
ZS
,
Fatah
PW
,
Dalvandi
A
.
Prevalence of workplace violence against healthcare workers during the COVID-19 pandemic: a systematic review and meta-analysis
.
Front Psychol
.
2022
;
13
:
896156
10
Saragih
ID
,
Tarihoran
DETAU
,
Rasool
A
,
Saragih
IS
,
Tzeng
HM
,
Lin
CJ
.
Global prevalence of stigmatization and violence against healthcare workers during the COVID-19 pandemic: a systematic review and meta-analysis
.
J Nurs Scholarsh
.
2022
;
54
(
6
):
762
771
11
McKay
D
,
Heisler
M
,
Mishori
R
,
Catton
H
,
Kloiber
O
.
Attacks against health-care personnel must stop, especially as the world fights COVID-19
.
Lancet
.
2020
;
395
(
10239
):
1743
1745
12
Hooton
A
,
Bloom
BM
,
Backus
B
.
Violence against healthcare workers at the emergency department
.
Eur J Emerg Med
.
2022
;
29
(
2
):
89
90
13
Kim
I
,
Bukhman
AK
,
Chen
PC
, et al
.
Psychiatric care in the emergency department: converting boarding time to treatment time
.
Acad Emerg Med
.
2022
;
29
(
12
):
1512
1514
14
Kraft
CM
,
Morea
P
,
Teresi
B
, et al
.
Characteristics, clinical care, and disposition barriers for mental health patients boarding in the emergency department
.
Am J Emerg Med
.
2021
;
46
:
550
555
15
Yoon
J
,
Bui
LN
,
Govier
DJ
,
Cahn
MA
,
Luck
J
.
Determinants of boarding of patients with severe mental illness in hospital emergency departments
.
J Ment Health Policy Econ
.
2020
;
23
(
2
):
61
75
16
Lepping
P
,
Lanka
SV
,
Turner
J
,
Stanaway
SE
,
Krishna
M
.
Percentage prevalence of patient and visitor violence against staff in high-risk UK medical wards
.
Clin Med (Lond)
.
2013
;
13
(
6
):
543
546
17
Rodríguez-Acosta
RL
,
Myers
DJ
,
Richardson
DB
,
Lipscomb
HJ
,
Chen
JC
,
Dement
JM
.
Physical assault among nursing staff employed in acute care
.
Work
.
2010
;
35
(
2
):
191
200
18
CMS.gov
.
Hospital-acquired condition reduction program
. Available at: https://www.cms.gov/medicare/payment/prospective-payment-systems/acute-inpatient-pps/hospital-acquired-condition-reduction-program-hacrp. Accessed January 2024
19
PSNet
.
Workplace safety in health care
. Available at: https://psnet.ahrq.gov/perspective/workplace-safety-health-care. Accessed January 2024
20
Burlison
JD
,
Quillivan
RR
,
Kath
LM
, et al
.
A multilevel analysis of U.S. hospital patient safety culture relationships with perceptions of voluntary event reporting
.
J Patient Saf
.
2020
;
16
(
3
):
187
193
21
Rutledge
DN
,
Retrosi
T
,
Ostrowski
G
.
Barriers to medication error reporting among hospital nurses
.
J Clin Nurs
.
2018
;
27
(
9-10
):
1941
1949
22
Skutezky
T
,
Small
SS
,
Peddie
D
,
Balka
E
,
Hohl
CM
.
Beliefs and perceptions of patient safety event reporting in a Canadian emergency department: a qualitative study
.
CJEM
.
2022
;
24
(
8
):
867
875
23
Umscheid
CA
,
Mitchell
MD
,
Doshi
JA
,
Agarwal
R
,
Williams
K
,
Brennan
PJ
.
Estimating the proportion of healthcare-associated infections that are reasonably preventable and the related mortality and costs
.
Infect Control Hosp Epidemiol
.
2011
;
32
(
2
):
101
114
24
Bates
DW
,
Cullen
DJ
,
Laird
N
, et al
;
ADE Prevention Study Group
.
Incidence of adverse drug events and potential adverse drug events. Implications for prevention
.
JAMA
.
1995
;
274
(
1
):
29
34
25
Jha
AK
,
Larizgoitia
I
,
Audera-Lopez
C
,
Prasopa-Plaizier
N
,
Waters
H
,
Bates
DW
.
The global burden of unsafe medical care: analytic modelling of observational studies
.
BMJ Qual Saf
.
2013
;
22
(
10
):
809
815
26
Linnebur
M
,
Inaba
K
,
Chouliaras
K
, et al
.
Preventable complications and deaths after emergency nontrauma surgery
.
Am Surg
.
2018
;
84
(
9
):
1422
1428
27
Gardiner
JC
,
Reed
PL
,
Bonner
JD
,
Haggerty
DK
,
Hale
DG
.
Incidence of hospital-acquired pressure ulcers - a population-based cohort study
.
Int Wound J
.
2016
;
13
(
5
):
809
820
28
Landrigan
CP
,
Parry
GJ
,
Bones
CB
,
Hackbarth
AD
,
Goldmann
DA
,
Sharek
PJ
.
Temporal trends in rates of patient harm resulting from medical care
.
N Engl J Med
.
2010
;
363
(
22
):
2124
2134
29
Kuo
LE
,
Kaufman
E
,
Hoffman
RL
, et al
.
Failure-to-rescue after injury is associated with preventability: the results of mortality panel review of failure-to-rescue cases in trauma
.
Surgery
.
2017
;
161
(
3
):
782
790
30
Sundberg
M
,
Perron
CO
,
Kimia
A
, et al
.
A method to identify pediatric high-risk diagnoses missed in the emergency department
.
Diagnosis (Berl)
.
2018
;
5
(
2
):
63
69
31
Singh
H
,
Meyer
AN
,
Thomas
EJ
.
The frequency of diagnostic errors in outpatient care: estimations from three large observational studies involving US adult populations
.
BMJ Qual Saf
.
2014
;
23
(
9
):
727
731
32
Ozonoff
A
,
Milliren
CE
,
Fournier
K
, et al
.
Electronic surveillance of patient safety events using natural language processing
.
Health Informatics J
.
2022
;
28
(
4
):
14604582221132429
33
Milliren
CE
,
Ozonoff
A
,
Fournier
KA
,
Welcher
J
,
Landschaft
A
,
Kimia
AA
.
Enhancing pressure injury surveillance using natural language processing
.
J Patient Saf
.
2023
;
20
(
2
):
119
124
34
Kimia
AA
,
Savova
G
,
Landschaft
A
,
Harper
MB
.
An introduction to natural language processing: how you can get more from those electronic notes you are generating
.
Pediatr Emerg Care
.
2015
;
31
(
7
):
536
541
35
Kimia
A
,
Lee
L
,
Shannon
M
, et al
.
Holiday ornament-related injuries in children
.
Pediatr Emerg Care
.
2009
;
25
(
12
):
819
822
36
Kimia
AA
,
Rudloe
TF
,
Aprahamian
N
, et al
.
Predictors of a drainable suppurative adenitis among children presenting with cervical adenopathy
.
Am J Emerg Med
.
2019
;
37
(
1
):
109
113
37
Kimia
R
,
Voskoboynik
B
,
Hudgins
JD
, et al
.
Is lymphangitic streaking associated with different pathogens?
Am J Emerg Med
.
2021
;
46
:
34
37
38
Waltzman
ML
,
Lee
LK
,
Ozonoff
A
,
Kupiec
JK
,
Landschaft
A
,
Kimia
AA
.
Treadmill injuries in children
.
Am J Emerg Med
.
2021
;
46
:
495
498
39
El Helou
R
,
Landschaft
A
,
Harper
MB
,
Kimia
AA
.
Bacteremia in children with fever and acute lower extremity pain
.
Pediatrics
.
2023
;
151
(
5
):
e2022059504
40
Rudloff
JR
,
El Helou
R
,
Landschaft
A
,
Harper
MB
,
Ahmad
FA
,
Kimia
AA
.
Bacteremia in patients with fever and acute lower extremity pain in a non-Lyme endemic region
.
Pediatrics
.
2024
;
153
(
1
):
e2023064095
41
Kimia
A
.
DrT - documentreviewtools.com
. Available at: https://documentreviewtools.com/. Accessed January 2024
42
Aljohani
B
,
Burkholder
J
,
Tran
QK
,
Chen
C
,
Beisenova
K
,
Pourmand
A
.
Workplace violence in the emergency department: a systematic review and meta-analysis
.
Public Health
.
2021
;
196
:
186
197
43
Stafford
S
,
Avsar
P
,
Nugent
L
, et al
.
What is the impact of patient violence in the emergency department on emergency nurses’ intention to leave?
J Nurs Manag
.
2022
;
30
(
6
):
1852
1860
44
Friedl
JEF
.
Mastering Regular Expressions
, 3rd ed.
O’Reilly
;
2006
45
Goyvaerts
J
,
Levithan
S
.
Regular Expressions Cookbook
. 2nd ed.
O’Reilly
;
2012
46
Bernardy
J-P
,
Blanck
R
,
Chatzikyriakidis
S
,
Lappin
S
,
Maskharashvili
A
.
Probabilistic Approaches to Linguistic Theory
.
Center for the Study of Language and Information
;
2022
47
Cohen
T
,
Widdows
D
.
Empirical distributional semantics: methods and biomedical applications
.
J Biomed Inform
.
2009
;
42
(
2
):
390
405

Competing Interests

CONFLICT OF INTEREST DISCLOSURES: The authors have indicated they have no potential conflicts of interest to disclose.

Supplementary data