Predictive Modeling to Identify Children With Complex Health Needs At Risk for Hospitalization

We conducted this study at an academic health system that includes a tertiary children’s hospital with more than 6000 annual discharges and provides care across urban and rural communities in the southern United States. The health system has used an enterprise-wide EHR (Epic; Verona, WI) since 2014. A large pediatric primary care practice in the health system that sees >24 000 children and conducts >59 000 visits/y was selected a priori as a primary care testing site for the developed CDS tool.

Patient eligibility for the study cohort for development and testing of the clinical predictive model (referred to as the “model”) included age 0 to 20 years old and attribution to a primary care physician within our health system. We abstracted EHR data on included patients from January 1, 2016 to March 31, 2020. To reflect how the tool would be used in practice to risk-stratify patients on a monthly basis, every month a patient was study-eligible, their EHR data contributed a unique person-month of patient-level data analyzed by the model. Person-months were excluded if the individual was pregnant to reduce confounding on model predictions introduced by including routine pregnancy-related hospitalizations. Children without clinical encounters in our health system in the previous 18 months were excluded to focus model predictions on patients more likely to have established care within our health system.

The primary outcome predicted by the model was all-cause hospitalization within 6 months of the current eligible patient-month. Model output was the probability (0%–100%) for each patient to be hospitalized in the next 6 months.

We abstracted patient demographics (age, sex, race, ethnicity, payer), diagnosis codes, and health service utilization data (outpatient, emergency, inpatient) from the EHR. We grouped International Classification of Diseases, 10th revision (ICD-10) diagnosis codes by organ systems as defined by the PMCA.¹³  We supplemented clinical, demographic, and health service utilization data with Area Deprivation Index (ADI)²¹  scores for each patient’s EHR-documented home ZIP code. The ADI is a composite measure of community-level disadvantage and socioeconomic status (SES) based on 17 US census–based measures of education, housing, employment, and poverty.²¹  The ADI score served as a proxy for individual SES, with higher ADI scores correlating with higher community-level disadvantage and lower SES. We tabulated service utilization counts based on the previous 365 and 30 days from the first day of the current person-month. We included a total of 50 variables in the final predictive model (Supplemental Table 3). Model features were selected through an iterative process led by a team, including data scientists, an informaticist, and pediatricians.

To facilitate use of the model in clinical care, we developed an online dashboard (Fig 1; Tableau; Seattle, WA)²²  to allow clinicians to visualize the model’s risk predictions. We created a workflow that identified all eligible patients on a monthly basis, applied the model to calculate 6-month hospitalization risk, and transferred data for all high-risk patients into the dashboard. The dashboard also displayed all model features at the individual patient level, categorized as sociodemographic, clinical, and health care utilization, so clinicians could see the contribution of each variable to summative risk scores. The layout and functionalities, including the ability to view patient-level details and query risk scores, were codesigned with clinicians to optimize real-world usability.

For eligible patients in the model development cohort, we divided included person-months between January 1, 2016 and September 30, 2018, into training and validation datasets; data were split in a 70:30 (training/validation) ratio at the patient level to maintain distinct data without overlap of training and validation datasets. Next, we analyzed eligible person-months between October 1, 2018 and September 30, 2019, as a testing cohort (Fig 2) followed by 6 months of follow-up through March 31, 2020, for outcomes analyses.

We used gradient boosting machines (GBMs) to build the model. GBM is an ensemble machine learning algorithm that combines shallow trees, known as stumps.²³  As a tree-based algorithm, it is effective for modeling nonlinear and heterogenous effects. Based on internal cross-validation, optimal model depth was 3 and consisted of 500 stumps. We used validation data to compare performance of LASSO regularized logistic regression and GBM and found that GBM had the best performance. We evaluated the model’s performance on testing data using area under the receiver operator curve (AUROC), area under the precision recall curve (AUPRC), and calibration slope. To account for individual repeated measurements, we calculated monthly performance metrics and computed a weighted average based on the sample size. Model parameters were not retrained beyond the initial training and validation period.

Translation of the model’s risk predictions into a CDS rule for identification of high-risk CCHN was informed by user-centered design methods.^24,25  Because of the clinical relevance of care coordination (CC) to supporting CCHN, we considered CC as a use case and focused CDS rule development on the model’s potential to identify high-risk CCHN who could be prioritized for CC services. The first step in the CDS rule development process was partnership with primary care and health system clinicians to understand baseline CC workflows. As a result, we learned that the number of CCHN in need of CC was greater than the workload capacity of existing clinical staff, underscoring the role of a CDS rule to inform allocation of limited available CC services. Next, to align with potential workload capacity of clinical staff to prioritize CC delivery at the primary care test clinic to high-risk patients, a high-risk threshold and CDS rule for the model was established at $≥$10% risk for future hospitalization.

Performance of the model-based CDS rule was assessed using sensitivity, specificity, and precision (measured as positive predictive value [PPV]). To contextualize performance of the model-based CDS rule, we compared it with Two other decision rules used in research and practice that are available to identify high-risk patients in need of CC: (1) PMCA classification as a patient with complex chronic disease (tier 3) and (2) patients with previous high hospital utilization, defined as $≥$2 hospitalizations in the past 12 months. The PMCA tier 3 and previous high hospital utilization criteria were used as alternative thresholds to which we compared the model’s high-risk threshold of $≥$10% risk for future hospitalization. Model development was performed in Python 3.7.6. This work was reviewed and approved by our institutional review board.

The model development cohort assembled between January 1, 2016 and September 30, 2018, included 116 799 patients accounting for 2 353 262 person-months (Fig 2). To assess baseline characteristics of the study sample, we stratified a cross-section of the model development cohort (n = 94 720 patients as of September 30, 2018) into those with and without hospitalizations, respectively, in the previous 12 months. Among all patients in this cross-section of the model development cohort, 2.2% (n = 2061) had $≥$1 hospitalization in the past 12 months. In general, a higher proportion of children with previous hospitalizations were younger, male, publicly insured, and had higher medical complexity (eg, more organ systems involved, higher prevalence of technology dependence) (Table 1).

Evaluation of the predictive model’s overall performance using the testing dataset demonstrated a weighted AUROC = 0.79 and a weighted AUPRC = 0.13 (Figs 3A,3B). Analysis of the testing dataset demonstrated that 1.7% (n = 456) of all children with primary care attributed to the test clinic site (n = 27 087) were ever hospitalized during the 18-month testing and outcomes assessment period (October 1, 2018–March 31, 2020). The model’s average sensitivity and positive predictive value were 24.9% and 19%, respectively (Table 2). To assess potential for health inequities, we evaluated model performance based on individual age, race, sex, and insurance status and found that the model performed similarly across all demographic strata (Supplemental Table 4).

A total of 27 087 patients with primary care attributed to the test clinic site were included in the testing cohort, accounting for 274 678 person-months. Within this testing cohort, 0.8% (n = 2322), 17.3% (n = 47 444), and 0.6% (n = 1524) of the included person-months were classified as high risk by the data model, PMCA classification, and previous high hospitalization use, respectively. In a cross-sectional comparison (as of September 1, 2019; Fig 4), 3856 (14%) of all patients attributed to the primary care test clinic were classified as high risk by at least 1 of the CDS rules. The PMCA score classified the most patients as high risk (n = 3785; 98% of all patients classified high risk by any of the 3 CDS rules), followed by the data model (n = 177; 4.6%) and the previous high hospital utilization rule (n = 102; 2.6%). A total of 86% of patients classified as high risk by the data model were also identified by the PMCA; 28% of high-risk patients classified by the data model were also identified by previous high hospital utilization.

Comparative evaluation of the predictive accuracy of the Three CDS rules is shown in Table 2. On average throughout the model testing period (October 1, 2018–March, 31, 2020), 17.3% of all children were classified as high risk by PMCA, 0.5% by previous high hospital utilization, and 0.8% by the model. High-risk classification using a PMCA-based CDS had the highest sensitivity (52.4%) and lowest PPV (1.9%), whereas the model demonstrated moderate sensitivity (24.9%) and the highest PPV (19%). Compared with the model, CDS based on previous high hospital utilization demonstrated lower sensitivity (13%) and PPV (15%). CDS based on the PMCA classified 17.3% of children as having complex, chronic disease but only 4.7% of those children were actually hospitalized in the next 6 months. Differences between high-risk classification and actual hospitalization rates by CDS based on the data model and previous high hospital utilization, respectively, were smaller (Supplemental Table 4).

We applied machine learning methods to the development and evaluation of a novel EHR-based clinical predictive model that identified CCHN at high risk for future hospitalization. Model evaluation demonstrated moderate to high predictive performance with an AUROC of 0.79. Approximately 1 in 7 children (14%) within a primary care population was classified as high risk by at least one of the two CDS rules. Among the two CDS rules evaluated, sensitivity was low (13%–52%), specificity was high (83%–99%), and precision was highest for the model (PPV 19%).

Performance of the model should be interpreted in the context of the model’s data source, predicted outcome, and other published predictive models. For example, although precision for our model was relatively low compared with PPV for typical laboratory diagnostic tests, the model’s precision was more than 11-fold higher than other EHR data-based predictive models (median PPV among 26 previously published models was 1.7%).¹⁷  The high accuracy of the model was similar to other EHR data–based models predicting hospitalization (median C-statistic 0.71 for 10 published models)¹⁷  and comparable to published administrative data-based models predicting readmissions among hospitalized children.^26,27  Low sensitivity of our model implied a higher proportion of high-risk patients “missed.” However, accuracy of administrative and EHR data–based risk models to predict hospitalizations is limited.^17,28  These challenges are magnified for a complex population like CCHN with clinical heterogeneity and varying hospitalization risks.

The number of patients identified and missed are only one part of the assessment of a model’s overall value. A number needed to benefit (NNB) framework is one approach to estimate a predictive model’s value in practice.²⁹  For our model, one component of NNB, the number needed to screen (number of patients flagged as high risk by the model to find one patient who will actually be hospitalized) – was five. A second component of NNB – number needed to treat (NNT), depends on the intervention linked to an implemented model’s predictions. In this case, we hypothesized the model could be applied to inform prioritization of care coordination for identified high-risk patients. If applied in practice, the NNT might be estimated to be between 5 and 9 patients based on published trials of pediatric complex care coordination interventions.^15,30  However, NNT estimates of care coordination for CCHN are imprecise because of the small number of trials and variation in child complex care interventions³¹ ; sites also might apply the model to non-care coordination interventions, thereby altering NNT estimates and limiting the ability to accurately estimate overall NNB for our model at this time. Furthermore, costs of the model and intervention are site-specific factors for NNB and overall value that should be considered if the model is ultimately implemented in practice.

Comparable complexity classification tools¹¹  available for application as CDS rules, such as PMCA and previous high hospital utilization, are associated with health service use^27,32 ; however, these tools were not designed to predict future hospitalization. The challenge with using PMCA alone to prospectively identify children at high risk for hospitalization was demonstrated in our study, in which for every 20 patients classified as high risk by the PMCA, one was actually hospitalized. Nonetheless, use of PMCA is expanding at the population level as a tool to identify children who might benefit from additional resources.¹⁴  Growth in its use along with PMCA’s established validity informed its selection as a pragmatic CDS tool to compare with our data model.

Although the model was not clinically implemented in this study, our plan is to apply model predictions as CDS to inform delivery of CC and evaluate the potential benefits of using this EHR data–based model approach. If applied clinically in the future to allocation of CC, higher relative precision gained by use of the model may translate into more patients correctly classified as high risk (true positives) who receive CC services. Because CC is resource intensive, a CDS rule with higher PPV may yield large population-level cost savings by more precisely matching limited CC services to CCHN at highest risk for negative outcomes. Future prospective testing of this approach is warranted to clarify the impacts of a model-based strategy in clinical practice.

This study had several strengths. First, the model was co-developed with clinicians to enhance usability and potential for clinical translation.³³  Second, we developed the model using multiple years of EHR data and designed it to be broadly applicable to children across the complexity spectrum. In contrast to adults with complex needs whose illnesses and health service use are often well established,³⁴  childhood chronic illnesses often take time to establish (eg, genetic syndromes diagnosed after years of symptomatology, high health service use). Regular, monthly updates to our model’s source data (eg, diagnoses, encounters) aligned with this longer course of pediatric chronic illness and enabled understanding the child’s longitudinal health trajectory. Third, we accounted for health-related social needs via inclusion of ADI as a model variable. Despite limited pediatric studies that integrate measures like ADI within EHR-based models, recent studies demonstrated associations between multidimensional measures of neighborhood-level socioeconomic factors (eg, Child Opportunity Index) and health service outcomes,^35,36  thus supporting relationships between neighborhood context and child health. We hypothesize that our model’s inclusion of medical and social risk factors contributed to more precise identification of children who were hospitalized (ie, highest PPV among three CDS rules evaluated). Finally, we conducted silent testing over an 18-month period to evaluate the model’s performance with real-world data before implementing the model within routine workflows.

Several limitations of this study should be acknowledged. First, generalizability of our findings is limited by the single-center, single-EHR study design. Operationalization of our site-specific EHR data demonstrated the potential value of the EHR data modeling approach, not necessarily the need to replicate the exact model and its features. More work is needed to adapt our approach and the model in other centers/EHRs and to explore augmenting the model with non-EHR data (eg, payer claims). Second, the ADI score relies on an accurate home address in the EHR, which may be inaccurate because of the inherent “messiness” of EHR data,¹⁸  and may not fully reflect individual-level social needs. Individual-level adverse social determinants of health (SDH; eg, poverty, limited transportation) and adverse social conditions associated with poor health (eg, housing instability) are social risk factors³⁷  identified via validated SDH screening instruments³⁸ ; Within EHR systems with universal SDH screening, these could be explored as incremental data model features that might offer a more nuanced assessment of social needs. Third, model predictions focused on risk for future hospitalization has its limitations. For example, our model predictions did not account for preventability of hospitalizations nor did it include health service use external to our center. Although acute care use is an important health service outcome and feasible to measure in EHRs, future adaptation of this data model should incorporate patient-/family-reported health outcomes. Finally, emphasis on diagnoses and encounters as model features updated monthly rather than real-time laboratory values and vital sign trends limited detection of acute health status changes. However, development of multi-morbidity with establishment of co-occurring chronic conditions occurs over years for many children with complex chronic conditions³⁹ ; this incremental natural course is better aligned with our model’s framework.

The data model is now being implemented in a real-world primary care setting to test the feasibility and impact of using it as a tool for prioritizing CC services for CCHN. We are also expanding access to the model’s predictions and visualizations (online dashboard) so that clinicians and population health staff across our health system can use the model as a care management tool. Future studies could explore the impact of dynamic changes in hospitalization risk (eg, at what risk level should care intensity be titrated upward or downward or stopped; should risk scores from multiple past time points be considered when deciding on intervention delivery; and what are the patient-level impacts of these approaches?).

A novel EHR data-based predictive model was developed and validated as a CDS tool to identify CCHN at high risk for future hospitalization. The overall favorable performance of the predictive model as a CDS rule highlighted potential opportunities for an EHR data–based risk modeling approach to help inform future population-level clinical initiatives.

We wish to acknowledge staff at Duke Children’s Primary Care and Duke University Health System Population Health Management Office, including Barbara Donadio, RN, BSN; Mary McKeveny, RN, BSN; Lauren Goslin, MSW, LCSW; Reneé Amos, MSW, LCSW; Regina Virgle, RN, BSN; Jodi Wert, CHW; Marlyn Wells, NBC-HWC; and Atalaysha Churchwell, PhD, for partnership on reviewing care coordination clinical workflows that was essential for developing a predictive model aligned with routine practice.