Opportunities for Diagnostic Improvement Among Pediatric Hospital Readmissions Free

This study was conducted at a freestanding academic children’s hospital with ∼29 000 annual discharges, including 8000 annual discharges from the general pediatrics service. Patient charts were reviewed if a patient was discharged from the general pediatrics service between October 1, 2018, and September 30, 2020, and had an unplanned readmission to the hospital (to any service) within 15 days of discharge. Planned readmissions (eg, scheduled surgery) were excluded. We evaluated 15-day readmissions to align with the pediatric readmission literature, which has found readmissions within this time frame to be more frequently modifiable than readmissions after 15 days.^8,15,17–19  This study was reviewed by the institutional review board at our institution and met criteria for exemption.

To leverage the expertise of practicing inpatient pediatric physicians, the study team was composed of 13 hospital-based pediatricians, including 1 general pediatric resident, 1 pediatric infectious disease fellow (with board certification in general pediatrics), and 11 pediatric hospital medicine attending physicians (board-certified in general pediatrics). Reviewers were offered maintenance of certification credit for participation. To foster a common approach to case review, all reviewers participated in a 1-hour training in identification of MOIDs led by a faculty member with expertise in QI, specifically diagnostic quality and safety, before commencing chart reviews. In keeping with our goals of fostering dialogue about diagnosis and building capacity in utilizing novel improvement tools, reviewers were encouraged to discuss cases as a group during weekly sessions, which served to further develop reviewer skills and comfort in identifying MOIDs.

Each case was independently reviewed by 2 study team members. Case reviews occurred asynchronously, and responses were recorded in REDCap, a secure, web-based data collection platform.²⁰  Reviewers were asked to consider all documentation in the electronic health record (EHR) pertaining to the episodes of care under review, including inpatient note documentation for the index admission and readmission, documentation from interval or preceding primary care and outpatient specialty visits, emergency department (ED) documentation, laboratory or radiology results, telephone encounters, vital sign data, medication administration, and prescription records. Reviewers documented the International Classification of Diseases, Clinical Modification (ICD-CM), codes for the index admission and readmission final diagnoses. Reviewers did not evaluate cases in which they had provided direct clinical care. To assess feasibility of the review process, reviewers were also asked to estimate time spent on each case review.

Our use of standardized tools and language to identify cases with potential diagnostic delays and error were for the purposes of learning and continuous improvement, both locally and to contribute to the diagnostic medicine literature. These cases were reviewed by clinicians trained in analytic tools for possible opportunities in the diagnostic process, without consideration for any determination of a deviation in the standard of care.

For each case, reviewers completed SaferDx, a widely adopted instrument designed to guide and operationalize evaluation for presence or absence of MOIDs.^9,21,22  SaferDx contains 13 statements relevant to the patient encounter. Reviewers indicate agreement with each statement on a scale from 1 (strongly disagree) to 7 (strongly agree). The final (13th) item states that “the episode of care under review has a missed opportunity to make a correct and timely diagnosis”⁸  on the basis of all the preceding 12 items. Similarly, reviewers indicate agreement with this concluding statement on a 1 (strongly disagree) to 7 (strongly agree) scale. For our study, cases in which reviewers assigned a score of >4 to the concluding SaferDx statement were classified as containing a MOID.⁹  A neutral score of 4 was presented to reviewers as an option only if the reviewer could not decide about the presence of a MOID. Discordant reviews (in which 1 reviewer assigned a score <4 and the other reviewer assigned a score ≥4) and nondefinitive reviews (in which both reviewers assigned a score of 4) were referred to a third reviewer within the study team for final determination. The case assignment process was performed at random, with an equal number of cases assigned to each study review team member. Open forum discussion by the study team was employed to achieve consensus in cases where the third reviewer assigned a neutral score of 4.

Central to operationalizing the identification of MOIDs is the explicit cognizance (and avoidance of) hindsight bias. In both reviews and case discussions, there was deliberate attention paid to:

1. whether there was opportunity to improve diagnosis on the basis of the information available to clinicians at the time; and

2. whether delays in diagnosis were present rather than progression or evolution of disease.

All cases with a MOID were discussed by the study team to identify potential diagnostic process improvement opportunities, mapped to domains in the Modified DE and Evaluation Research (DEER) taxonomy, which classifies “failure points”²³  (or opportunities for improvement) according to the phase in the diagnostic process in which they occur. The DEER taxonomy instrument illustrates the following phases of the diagnostic process: Access to care, history, physical exam, testing, hypothesis generation, consultation, and follow-up. The instrument also delineates potential opportunities for improvement within each phase. The tool allows for designation of multiple opportunities for each case. Case discussions occurred during weekly virtual conferences in which reviewers presented a brief case synopsis and their corresponding SaferDx scores, followed by identification of pertinent modified DEER taxonomy diagnostic process domains. A minimum of one-third of the study team was present for each discussion.

We calculated the incidence of MOIDs among unplanned general pediatric readmissions. Modified DEER taxonomy diagnostic process opportunities were tabulated, and the median number of opportunities per instance of MOID was calculated.

Descriptive statistics were used to compare demographic and clinical characteristics of the readmission study population with and without MOIDs. We looked at demographic characteristics that have been previously studied in the adult readmission context,⁴  including patient age, sex, preferred language, insurance, and a proxy equity measure using Social Vulnerability Index category.²⁴  Diagnostic disparities have been identified across conditions ranging from acute appendicitis²⁵  to depression,²⁶  and therefore we included race and ethnicity in our evaluation. To identify populations at increased risk for MOIDs and potentially refine future review criteria, we evaluated clinical characteristics including medical complexity defined in this study by presence of a complex chronic condition,²⁷  admission and discharge ICD-CM codes, admission source (ED or hospital-to-hospital transfer), ICU admission during either index or readmission, time to readmission (0–7 vs 8–15 days), median time to readmission, and hospital length of stay. ICD-CM codes were grouped using ICD, 10th Revision, categories and description keywords. Because the study period spanned the beginning of the coronavirus disease 2019 pandemic, with resultant changes to patient volumes and staffing which could have impacted diagnosis, we categorized episodes of care with an index admission date on or after March 1, 2020, as “during the pandemic” and evaluated this period separately. Differences were evaluated using Pearson’s χ² test. For continuous variables, we computed the median and interquartile range (IQR). Analyses were completed at both the patient and the readmission level, but are presented at the readmission level in keeping with our objective of identifying clinical systems’ opportunities for improvement. Finally, reviewer estimates of review time per case were averaged. Data analysis was performed using R version 4.0.3 (Foundation for Statistical Computing, Vienna, Austria) and Stata 17 (StataCorp LP, College Station, TX).

During the study period, there were 353 readmissions by 321 unique patients, including 5 planned readmissions excluded from review (Fig 1). Of the 348 cases reviewed with SaferDx, 22 (6.3%) MOIDs were identified (Table 1). Of the cases with MOIDs, potential diagnostic process improvement opportunities were identified in every modified DEER taxonomy domain except access to care (Fig 2). Cases with MOIDs had a median of 5 areas for potential diagnostic improvement. The most frequently identified diagnostic process improvement opportunities included: clinician assessment (weighing of a piece of history [n = 10, 45%]), decision-making (ordering needed test[s] [n = 10, 45%]), and hypothesis generation or diagnostic reasoning (considering the correct diagnosis [n = 11, 50%]; weighing of the correct diagnosis [n = 10, 45%]). Example cases of each DEER taxonomy domain are illustrated in Table 2.

Readmitted patients who experienced MOIDs were older (median age 3.8 years, IQR 1.5–11.2) than those without MOIDs (median age 1.0 year, IQR 0.3–4.9), but otherwise demographically similar (Table 1). Patients with MOIDs experienced longer readmission length of hospital stay (median 4.3 days, IQR 2.0–11.9) than cases without (median 2.3 days, IQR 1.5–4.8) MOIDs. Although most (82%) MOIDs were associated with readmissions occurring within 0 to 7 days from discharge, the probability of having a MOID did not vary significantly among those readmitted within 0 to 7 days compared with those readmitted from 8 to 15 days (P = .13). We evaluated demographic characteristics at both the encounter and the patient level; no notable differences in significance or magnitude existed.

MOIDs occurred across a range of conditions. Final readmission diagnoses included infectious processes such as septic arthritis, lymphadenitis with abscess, and cellulitis (n = 6, 22.7%); upper airway anomalies such as tracheal stenosis and laryngeal cleft (n = 3, 13.6%); and cardiac etiologies (n = 2, 9.1%). No index admission diagnoses or diagnostic groups were significantly associated with MOIDs. However, lower respiratory tract conditions (including bronchiolitis, asthma exacerbation, and pneumonia) accounted for 91 (26%) of index admission diagnoses but only 1 (4.5%) case of MOID (P = .02). Among those admitted for nonlower respiratory tract conditions (n = 257), the incidence of MOIDs did not differ by age group (P = .3).

Reviewers reported that using the SaferDx tool to reach a conclusion about the presence or absence of MOIDs took an average of 30 minutes per case. Because each case was reviewed by 2 reviewers and 46 cases (13%) required a third review, this represented ∼371 review hours (or 29 hours per reviewer over the study period). Limiting review to readmissions within 7 days of discharge and to nonlower respiratory tract conditions would have reduced the review burden by nearly half (from 348 charts to 179 charts) and identified 17 (78%) of 22 MOIDs, increasing the rate of MOID to 9.5%.

Using a systematic review process, we identified MOIDs in 6.3% of children who were discharged from the general pediatrics service and required readmission within 15 days of discharge. Potential diagnostic process improvement opportunities within our cohort were those related to clinician decision-making and consideration or prioritization of the correct diagnosis. We did not identify conditions at increased risk for MOIDs; however, readmissions for lower respiratory tract infections were less likely to experience a MOID.

The identified rate of MOIDs among general pediatric readmissions in our study is similar to the DE rate identified in adult internal medicine readmissions (5.6%).⁴  It also aligns with rates identified among pediatric inpatients in the United Kingdom (5.0%)²⁸  and among pediatric patients discharged from the ED who were subsequently hospitalized (4.8%).²⁹  Given that nearly 3 million children experience hospitalization in the United States every year, these rates indicate an urgent need to develop diagnostic improvement strategies for acute care pediatrics. Our approach identified novel opportunities to improve diagnosis and potentially avert the harms associated with readmission in a subset of readmitted patients.

We used DEER taxonomy to classify potential diagnostic process improvement domains. Echoing the adult readmissions literature,⁴  the most frequently identified opportunities for improvement pertained to clinician assessment (ordering tests and hypothesis generation/diagnostic reasoning). This aligns with findings from a 2014 study of a British pediatric community hospital cohort, which identified cognitive factors (data gathering and formulation of a different diagnosis) as primary drivers of DE.²⁸  Although crafting specific interventions was not the focus of this study, our findings suggest that QI efforts targeting diagnostic reasoning (eg, adoption of standardized processes to label diagnostic uncertainty³⁰  or development of note templates that support recognition of alternative diagnoses) may be helpful in reducing MOIDs. Identification of these potential improvement domains provides focus for the systems-change efforts needed to promote and support diagnostic quality and safety at an individual, team, and systems level.¹⁰ 

We sought to identify demographic and clinical characteristics of readmitted children associated with increased risk of MOIDs. We observed overall age differences between readmitted patients who experienced MOIDs and those who did not, though the frequency of admissions because of respiratory conditions among infants (especially for admissions related to lower respiratory tract conditions) appears to drive these differences and is consistent with previous literature among unplanned pediatric admissions identified by an e-trigger.²⁹  Although we did not identify sociodemographic factors significantly associated with MOIDs, it is notable that 18% of MOIDs occurred in the 29 (8%) encounters in which patients or families identified as using a primary language other than English.³¹  Previous studies have noted that patients with language barriers are at particular risk for medical error (including DE),^32–36  and we suggest the importance of further surveillance of MOIDs among this population.

One of our secondary objectives was to support dialogue about diagnostic improvement within a diverse group of practicing pediatricians. Mindful of the psychological safety of clinicians and to promote a focus on systems improvement, we shifted from error nomenclature to “potential missed opportunities.” We observed that, although we used a standardized review strategy, 13% of cases required a third reviewer because of discordant or nondefinitive scoring. Even among cases that were ultimately identified as having a MOID, many (17 of 22) required a third reviewer. This likely represents a combination of complexity or nuance in the particular cases, in addition to the relative newness of this work. It also reflects the reality that, even with a standardized tool like SaferDx, it can be difficult for individual reviewers to classify cases as having a MOID. In response to reviewer uncertainty, we established open discussion forums to come to group consensus about cases. We anticipate that, as concepts of MOID become more familiar within pediatrics and at our institution, reviewers will be more definitive in their scoring.

Finally, we explored readmission factors that might improve the yield of our case review process. Pediatric autopsy data from studies in pediatric and neonatal critical care units have found high rates of missed sepsis and vascular events^37,38 ; however, these conditions occur less frequently in general pediatrics and did not occur in our sample. EHR or administrative diagnostic, code-based triggers for diagnostic review have been proposed as a strategy for identifying opportunities to improve diagnosis.^29,39,40  In the absence of clear diagnostic groups associated with pediatric inpatient MOIDs, there may be a subset of pediatric hospital readmissions that are high yield in identifying opportunities to improve diagnosis (and could serve as an e-trigger for review). Limiting future reviews to 7-day readmissions and excluding readmissions for lower respiratory tract infections would have reduced the case review burden by nearly half, while identifying 78% of MOIDs. Narrowing the cohort of reviewed pediatric hospital readmission cases may be a pragmatic strategy for ongoing MOID surveillance, acknowledging the tradeoff of cases missed using narrower review criteria.

One strength of our study is the application of a standardized, 2-reviewer process which used an established instrument (SaferDx) to identify MOIDs coupled with case review via DEER taxonomy. Since Warwick et al²⁵  published initial exploration of DE among pediatric inpatients in 2014, tools such as SaferDx have been developed to operationalize identification of MOID. Others have used SaferDx to evaluate pediatric diagnosis in critical care settings^37,38,41  and ED settings^29,42 ; we used SaferDx and DEER taxonomy to identify process improvement opportunities among general pediatric inpatients. Our standardized review approach is readily replicable in other settings. Our 13-member review panel provided varied perspectives and robust clinical experiences; this review model allowed for the identification of a range of potential improvement opportunities and promoted dialogue within the practice community. Reflecting on 2 years of general pediatric readmissions allowed for evaluation of MOIDs across time, hospital care delivery model, and clinician.

Our findings should be contextualized within the limitations of our approach. First, we detected relatively few MOIDs and thus were limited in our ability to assess differences between groups. Identifying populations at low risk for MOIDs should allow for future review efforts to be more efficient, perhaps allowing for detection of a cohort large enough for us to evaluate drivers of MOIDs. Similarly, future work exploring approaches to pediatric hospital MOID identification in community hospital settings or across institutions will be important in terms of generalizability. We sought to mitigate the risk of hindsight bias from retrospective case review by engaging in multiclinician discussions in which this potential hazard was explicitly named and considered. Our reviews were limited by the information available in the EHR; important factors such as clinician cognitive load, family perspectives, and verbal communication were outside our scope. Although we had access to records from our tertiary health care system, we were unable to detect MOIDs in which patients did not return to care, did not require readmission, or were readmitted at a different institution.

Using a standardized approach with SaferDx and DEER taxonomy, we identified MOIDs among general pediatric readmissions. We found common areas for diagnostic process improvement opportunities, namely clinician decision-making and diagnostic reasoning that confirm data trends in adult populations. We identified conditions at lower risk for MOIDs that will inform ongoing improvement work. Future studies across institutions and care settings can further describe populations of children at risk for MOIDs.

We thank Kristi McNaughton, MHS, for editing assistance.