OBJECTIVES:

To target interventions to prevent readmission, we sought to develop clinical prediction models for 30-day readmission among children with complex chronic conditions (CCCs).

METHODS:

After extracting sociodemographic and clinical characteristics from electronic health records for children with CCCs admitted to an academic medical center, we constructed a multivariable logistic regression model to predict readmission from characteristics obtainable at admission and then a second model adding hospitalization and discharge variables to the first model. We assessed model performance using c-statistic and calibration curves and internal validation using bootstrapping. We then created readmission risk scoring systems from final model β-coefficients.

RESULTS:

Of the 2296 index admissions involving children with CCCs, 188 (8.2%) had unplanned 30-day readmissions. The model with admission characteristics included previous admissions, previous emergency department visits, number of CCC categories, and medical versus surgical admission (c-statistic 0.65). The model with hospitalization and discharge factors added discharge disposition, length of stay, and weekday discharge to the admission variables (c-statistic 0.67). Bootstrap samples had similar c-statistics, and slopes did not suggest significant overfitting for either model. Readmission risk was 3.6% to 4.9% in the lowest risk quartile versus 15.9% to 17.6% in the highest risk quartile (or 3.6–4.5 times higher) for both models.

CONCLUSIONS:

Clinical variables related to the degree of medical complexity and illness severity can stratify children with CCCs into groups with clinically meaningful differences in the risk of readmission. Future research will explore whether these models can be used to target interventions and resources aimed at decreasing readmissions.

Recent advancements in medical care and social services have significantly altered pediatric health, with children now capable of living for extended periods with what may previously have been fatal conditions.1  This has led to an increasing number of children with medical complexity (CMC), who are relatively medically fragile, requiring more intensive care.2  Accordingly, a common means of defining medical complexity is through identification of complex chronic conditions (CCCs) expected to last greater than a year, affect at least 1 organ system severely, and likely to require subspecialty care and hospitalization.3,4  Although CMC represent a small percentage of the pediatric population, studies reveal that their health care use is disproportionately high, with increased rates of hospital use, longer lengths of stay, increased technology dependence, and higher hospital charges.3,59  Thirty-day readmission rates are also disproportionately elevated in this population, reaching as high as 24%.6,10  Not surprisingly, CMC account for approximately one-third of all child health care costs, with the majority attributable to inpatient hospitalization and with readmissions representing the highest proportion of subsequent costs.6,7 

Given that CMC represent a population both at high risk for readmission and accountable for high health care costs, targeting interventions for those with identifiable risk factors associated with readmission is desirable. Although previous association studies have provided some insight into factors influencing the likelihood of readmission among CMC, including use of home nursing care and timing of outpatient follow-up, a prediction model developed for use in clinical practice is absent from the literature.6,1118  Such a prediction model may help to identify CMC at highest risk for readmission in real time, aiding in the targeting of interventions and resources to reduce readmission rates and associated costs while improving patient and family quality of life and experiences.17,19,20  This study therefore aimed to develop clinical prediction models for 30-day readmission among children with CCCs capable of use during admission for targeting of inpatient interventions, care coordination, or discharge planning, or for use at discharge for targeting outpatient services.

We used a retrospective cohort design in this study. The index admission was defined as the first hospitalization occurring between October 1, 2010, and July 31, 2016, during which the patient met previously published standard criteria for medical complexity.4  The primary outcome (dependent variable) was unplanned readmission within 30 days of the index admission discharge date.

Eligible patients were 6 months to 18 years old, admitted to an urban academic medical center between October 1, 2010, and July 31, 2016, with the lower age limit set to 6 months because our study design required a look-back period of 6 months for assessment of previous hospital use. We excluded any pregnancy-related or psychiatric admissions, using pregnancy and childbirth and mental health diagnostic-related groups.12,21,22  As in other studies, we excluded admissions ending in death (not at risk for readmission outcome), discharges against medical advice (inadequate opportunity to implement care plan and discharge instruction), or discharge to hospice (terminally ill patients have different goals of care compared with nonhospice peers). Once a cohort of eligible admissions was established, we excluded children without medical complexity through application of the widely used CCC taxonomy (Feudtner method, version 2).4  This taxonomy uses International Classification of Diseases, Ninth Revision and International Classification of Diseases, 10th Revision codes to identify CMC based on the presence of a CCC within 11 broad medical categories such as neurologic, cardiovascular, or gastrointestinal. For patients with multiple hospitalizations, only the index admission was retained and assessed for readmission. For each patient, we then determined whether there was an unplanned readmission within 30 days of index admission discharge. Using the methodology of Berry et al,21  we excluded planned readmissions for chemotherapy and planned pediatric procedures.22 

All patient data were extracted from a prospectively collected hospital data set maintained by the academic center for determination of quality metrics, augmented by extraction of the number of home medications at admission from the electronic health record (EHR). Independent variables, selected a priori based on clinical judgment and literature review, included sociodemographic factors (age, race and ethnicity, sex, non-English primary language, insurance, neighborhood per capita income), measures of hospital use (any admissions and any emergency department [ED] visits in the 6 months leading up to index admission), clinical measures (CCC category, number of CCC categories, technology assistance, number of home medications at admission, and admission type), and hospitalization and discharge characteristics (any ICU use, discharge disposition, length of stay [LOS], and weekday versus weekend discharge during index admission).

Neighborhood per capita income was determined from zip codes.23  Admission type was dichotomized as either medical or surgical, defined by the admitting service team. CCC categories, number of CCC categories, and technology assistance were determined through application of the CCC taxonomy.4  Discharge disposition was categorized as discharge from the hospital without services, from the hospital with services, or to another facility.

Descriptive statistics were used to determine the frequencies of clinical and demographic characteristics in the study population. Univariate logistic regression was then used to test the association between each independent variable and the primary outcome (unplanned 30-day readmission). For continuous variables, we confirmed the assumption of linearity. Variables reaching a P value <.2 in univariate analysis were considered as candidates for the multivariable models. We first constructed a model using only characteristics obtainable at admission (“model at admission”), using multivariable logistic regression and backward selection with Akaike information criterion.24  To determine if hospitalization and discharge-specific factors improved model performance, a second model was then developed including variables available at discharge. To derive the second model (“model at discharge”), hospitalization and discharge factors passing univariate screening were forced into the final model at admission to assess their incremental predictive ability.

We checked for collinearity among final model variables using the variance inflation factor and assessed for potential influence points.25  Final model performance was then determined by using c-statistic and calibration curves, and internal validation was performed by using a bootstrapping algorithm with 500 samples randomly selected with replacement from the derivation set.2628  When applicable, the automated variable selection procedures used in model derivation were repeated across each bootstrapped sample; c-statistics and slopes were then recalculated by applying the models obtained from the bootstrapping algorithm to the original derivation population and averaged to determine model performance.

To facilitate use in clinical practice, the β-coefficient of each variable in the final model was multiplied by a constant factor and rounded to the nearest integer to create a point system; points obtained for each variable were then added to derive a readmission risk score per patient. We evaluated the risk score distribution in our study population by using descriptive statistics to determine the proportion of total admissions and readmissions within each risk score level. The readmission risk score discriminative performance was then determined by using c-statistic.

All statistical analyses were performed by using R statistical software version 3.4.2 (RStudio version 1.1.383, Vienna, Austria; the stats package was used for model development, and the rms package was used for model evaluation and validation). This study was approved by the Tufts Medical Center Institutional Review Board.

There were 11 151 pediatric admissions during the period under study. After application of exclusion criteria, 2296 index admissions for unique children with CCCs remained, and 188 (8.2%) had unplanned 30-day readmissions (Supplemental Fig 2).

Among our study patients, median age was 8 years (interquartile range 2–14 years), 56% were boys, 90% were English-speaking, and 61% had private insurance (Table 1). Neurologic, cardiovascular, and gastrointestinal CCCs were most predominant, and nearly one-fourth of the patients were technology dependent.

TABLE 1

Sociodemographic and Clinical Characteristics of the Study Cohort

CharacteristicOverall (n = 2296 patients)Readmitted (n = 188)Not Readmitted (n = 2108)
Sociodemographic factors    
 Age, n (%)    
  6 mo–<2 y 437 (19.0) 48 (25.5) 389 (18.5) 
  2–5 y 500 (21.8) 42 (22.3) 458 (21.7) 
  6–12 y 625 (27.2) 47 (25.0) 578 (27.4) 
  13–18 y 734 (32.0) 51 (27.1) 683 (32.4) 
 Race and/or ethnicity, n (%)    
  Non-Hispanic white 1370 (59.7) 105 (55.9) 1265 (60.0) 
  Hispanic 432 (18.8) 36 (19.1) 396 (18.8) 
  Non-Hispanic African American 243 (10.6) 25 (13.3) 218 (10.3) 
  Asian American 191 (8.3) 18 (9.6) 173 (8.2) 
  Other 60 (2.6) 4 (2.1) 56 (2.7) 
 Boys, n (%) 1286 (56.0) 113 (60.1) 1173 (55.6) 
 Non-English primary language, n (%) 237 (10.3) 24 (12.8) 213 (10.1) 
 Insurance type, n (%)    
  Private 1410 (61.4) 107 (56.9) 1303 (61.8) 
  Public 872 (38.0) 81 (43.1) 791 (37.5) 
  Uninsured 14 (0.6) 14 (0.7) 
 Neighborhood per capita income, $a 30 520 (22 714–37 424) 30 074 (23 350–35 959) 30 520 (22 714–37 424) 
 Missing data 
Clinical factors, n (%)    
 Any admissions in previous 6 mo 333 (14.5) 46 (24.5) 287 (13.6) 
 Any ED visits in previous 6 mo 203 (8.8) 33 (17.6) 170 (8.1) 
 No. home medications at admission, n (%)    
  0–3 1770 (78.1) 134 (71.7) 1636 (78.7) 
  4–7 301 (13.3) 26 (13.9) 275 (13.2) 
  ≥8 196 (8.6) 27 (14.1) 169 (8.1) 
  Missing data 29 28 
 CCC category, n (%)    
  Neurologic 550 (24.0) 37 (19.7) 513 (24.3) 
  Cardiovascular 492 (21.4) 37 (19.7) 455 (21.6) 
  Gastrointestinal 419 (18.2) 54 (28.7) 365 (17.3) 
  Other 1452 (63.2) 134 (71.3) 1318 (62.5) 
 No. CCC categories, n (%)    
  1 1621 (70.6) 107 (56.9) 1514 (71.8) 
  2 439 (19.1) 47 (25.0) 392 (18.6) 
  ≥3 236 (10.3) 34 (18.1) 202 (9.6) 
 Technology assistance, n (%) 545 (23.7) 56 (29.8) 489 (23.2) 
 Admission type, n (%)    
  Medical 1440 (62.7) 140 (74.5) 1300 (61.7) 
  Surgical 856 (37.3) 48 (25.5) 808 (38.3) 
Hospitalization and discharge factors, n (%)    
 ICU use 1087 (47.3) 81 (43.1) 1006 (47.7) 
 Discharge disposition from the hospital    
  Home, no services 1706 (74.3) 112 (59.6) 1594 (75.6) 
  Home with services 468 (20.4) 63 (33.5) 405 (19.2) 
  Other facility 122 (5.3) 13 (6.9) 109 (5.2) 
 LOS in d    
  0–1 668 (29.1) 42 (22.3) 626 (29.7) 
  2–5 1172 (51.0) 89 (47.3) 1083 (51.4) 
  ≥6 456 (19.9) 57 (30.3) 399 (18.9) 
 Weekday discharge 1746 (76.0) 153 (81.4) 1593 (75.6) 
CharacteristicOverall (n = 2296 patients)Readmitted (n = 188)Not Readmitted (n = 2108)
Sociodemographic factors    
 Age, n (%)    
  6 mo–<2 y 437 (19.0) 48 (25.5) 389 (18.5) 
  2–5 y 500 (21.8) 42 (22.3) 458 (21.7) 
  6–12 y 625 (27.2) 47 (25.0) 578 (27.4) 
  13–18 y 734 (32.0) 51 (27.1) 683 (32.4) 
 Race and/or ethnicity, n (%)    
  Non-Hispanic white 1370 (59.7) 105 (55.9) 1265 (60.0) 
  Hispanic 432 (18.8) 36 (19.1) 396 (18.8) 
  Non-Hispanic African American 243 (10.6) 25 (13.3) 218 (10.3) 
  Asian American 191 (8.3) 18 (9.6) 173 (8.2) 
  Other 60 (2.6) 4 (2.1) 56 (2.7) 
 Boys, n (%) 1286 (56.0) 113 (60.1) 1173 (55.6) 
 Non-English primary language, n (%) 237 (10.3) 24 (12.8) 213 (10.1) 
 Insurance type, n (%)    
  Private 1410 (61.4) 107 (56.9) 1303 (61.8) 
  Public 872 (38.0) 81 (43.1) 791 (37.5) 
  Uninsured 14 (0.6) 14 (0.7) 
 Neighborhood per capita income, $a 30 520 (22 714–37 424) 30 074 (23 350–35 959) 30 520 (22 714–37 424) 
 Missing data 
Clinical factors, n (%)    
 Any admissions in previous 6 mo 333 (14.5) 46 (24.5) 287 (13.6) 
 Any ED visits in previous 6 mo 203 (8.8) 33 (17.6) 170 (8.1) 
 No. home medications at admission, n (%)    
  0–3 1770 (78.1) 134 (71.7) 1636 (78.7) 
  4–7 301 (13.3) 26 (13.9) 275 (13.2) 
  ≥8 196 (8.6) 27 (14.1) 169 (8.1) 
  Missing data 29 28 
 CCC category, n (%)    
  Neurologic 550 (24.0) 37 (19.7) 513 (24.3) 
  Cardiovascular 492 (21.4) 37 (19.7) 455 (21.6) 
  Gastrointestinal 419 (18.2) 54 (28.7) 365 (17.3) 
  Other 1452 (63.2) 134 (71.3) 1318 (62.5) 
 No. CCC categories, n (%)    
  1 1621 (70.6) 107 (56.9) 1514 (71.8) 
  2 439 (19.1) 47 (25.0) 392 (18.6) 
  ≥3 236 (10.3) 34 (18.1) 202 (9.6) 
 Technology assistance, n (%) 545 (23.7) 56 (29.8) 489 (23.2) 
 Admission type, n (%)    
  Medical 1440 (62.7) 140 (74.5) 1300 (61.7) 
  Surgical 856 (37.3) 48 (25.5) 808 (38.3) 
Hospitalization and discharge factors, n (%)    
 ICU use 1087 (47.3) 81 (43.1) 1006 (47.7) 
 Discharge disposition from the hospital    
  Home, no services 1706 (74.3) 112 (59.6) 1594 (75.6) 
  Home with services 468 (20.4) 63 (33.5) 405 (19.2) 
  Other facility 122 (5.3) 13 (6.9) 109 (5.2) 
 LOS in d    
  0–1 668 (29.1) 42 (22.3) 626 (29.7) 
  2–5 1172 (51.0) 89 (47.3) 1083 (51.4) 
  ≥6 456 (19.9) 57 (30.3) 399 (18.9) 
 Weekday discharge 1746 (76.0) 153 (81.4) 1593 (75.6) 
a

Values reported as median (25th–75th percentile).

A complete case analysis was performed because missing data were minimal (∼1%) and limited to the income and medication variables (Table 1). Of the 2296 children with CCCs in our cohort, 2267 complete cases were used in the regression analyses. Table 2 summarizes the univariate and multivariable associations with 30-day readmission. In unadjusted analyses, age and public insurance were the only sociodemographic factors meeting our a priori inclusion criterion with P value <.2. Many clinical characteristics were found to have a univariate association with unplanned readmission, including any admissions in the previous 6 months, any ED visits in the previous 6 months, number of home medications at admission, number of CCC categories, technology assistance, and medical (versus surgical) admission type (all P values <.05). Hospitalization and discharge characteristics passing the univariate screen included discharge disposition and LOS (both P values <.001) and weekday discharge (P value .07).

TABLE 2

Univariate and Multivariable Associations With 30-Day Readmission for Children With Medical Complexity

Univariate ScreenFinal Multivariable Model at AdmissionFinal Multivariable Model at Discharge
Unadjusted OR (95% CI)PAdjusted OR (95% CI)Adjusted OR (95% CI)
Admission characteristics     
 Age  .10a   
  6 mo–<2 y Reference  — — 
  2–5 y 0.74 (0.48–1.15)  — — 
  6–12 y 0.66 (0.43–1.01)  — — 
  13–18 y 0.61 (0.40–0.92)  — 
 Race  .65   
  White Reference  — — 
  African American 1.38 (0.86–2.15)  — — 
  Asian American 1.25 (0.72–2.07)  — — 
  Hispanic 1.10 (0.73–1.61)  — — 
  Other 0.86 (0.26–2.15)  — — 
 Boys 1.20 (0.89–1.63) .24 — — 
 Non-English primary language 1.30 (0.81–2.01) .25 — — 
 Public insurance 1.23 (0.90–1.65) .19a — — 
 Neighborhood per capita income (per $10 000 increase) 0.94 (0.81–1.08) .38 — — 
 Any admissions in previous 6 mo 2.06 (1.43–2.91) <.001 1.70 (1.15–2.45) 1.70 (1.16–2.46) 
 Any ED visits in previous 6 mo 2.43 (1.59–3.61) <.001 2.04 (1.32–3.10) 2.04 (1.31–3.11) 
 No. home medications at admission  .01a   
  0–3 Reference  — — 
  4–7 1.15 (0.73–1.76)  — — 
  ≥8 1.95 (1.23–2.99)  — — 
 No. CCC categories  <.001   
  1 Reference  Reference Reference 
  2 1.70 (1.17–2.42)  1.73 (1.19–2.49) 1.56 (1.06–2.26) 
  ≥3 2.38 (1.56–3.56)  2.30 (1.50–3.47) 1.72 (1.08–2.69) 
 Technology assistance 1.40 (1.00–1.94) .04a — — 
 Medical admission 1.82 (1.30–2.57) <.001 1.82 (1.30–2.63) 1.75 (1.23–2.49) 
Hospitalization and discharge characteristics  .22   
 ICU use 0.83 (0.61–1.12)  — — 
 Discharge disposition from the hospital  <.001   
  No services Reference  — Reference 
  With services 2.21 (1.59–3.06)  — 1.69 (1.17–2.44) 
  Other facility 1.70 (0.89–3.01)  — 1.15 (0.58–2.13) 
 LOS in d  <.001   
  0–1 Reference  — Reference 
  2–5 1.22 (0.84–1.81)  — 1.15 (0.78–1.72) 
  ≥6 2.13 (1.41–3.25)  — 1.45 (0.90–2.33) 
 Weekday discharge 1.41 (0.98–2.08) .07 — 1.23 (0.84–1.85) 
Univariate ScreenFinal Multivariable Model at AdmissionFinal Multivariable Model at Discharge
Unadjusted OR (95% CI)PAdjusted OR (95% CI)Adjusted OR (95% CI)
Admission characteristics     
 Age  .10a   
  6 mo–<2 y Reference  — — 
  2–5 y 0.74 (0.48–1.15)  — — 
  6–12 y 0.66 (0.43–1.01)  — — 
  13–18 y 0.61 (0.40–0.92)  — 
 Race  .65   
  White Reference  — — 
  African American 1.38 (0.86–2.15)  — — 
  Asian American 1.25 (0.72–2.07)  — — 
  Hispanic 1.10 (0.73–1.61)  — — 
  Other 0.86 (0.26–2.15)  — — 
 Boys 1.20 (0.89–1.63) .24 — — 
 Non-English primary language 1.30 (0.81–2.01) .25 — — 
 Public insurance 1.23 (0.90–1.65) .19a — — 
 Neighborhood per capita income (per $10 000 increase) 0.94 (0.81–1.08) .38 — — 
 Any admissions in previous 6 mo 2.06 (1.43–2.91) <.001 1.70 (1.15–2.45) 1.70 (1.16–2.46) 
 Any ED visits in previous 6 mo 2.43 (1.59–3.61) <.001 2.04 (1.32–3.10) 2.04 (1.31–3.11) 
 No. home medications at admission  .01a   
  0–3 Reference  — — 
  4–7 1.15 (0.73–1.76)  — — 
  ≥8 1.95 (1.23–2.99)  — — 
 No. CCC categories  <.001   
  1 Reference  Reference Reference 
  2 1.70 (1.17–2.42)  1.73 (1.19–2.49) 1.56 (1.06–2.26) 
  ≥3 2.38 (1.56–3.56)  2.30 (1.50–3.47) 1.72 (1.08–2.69) 
 Technology assistance 1.40 (1.00–1.94) .04a — — 
 Medical admission 1.82 (1.30–2.57) <.001 1.82 (1.30–2.63) 1.75 (1.23–2.49) 
Hospitalization and discharge characteristics  .22   
 ICU use 0.83 (0.61–1.12)  — — 
 Discharge disposition from the hospital  <.001   
  No services Reference  — Reference 
  With services 2.21 (1.59–3.06)  — 1.69 (1.17–2.44) 
  Other facility 1.70 (0.89–3.01)  — 1.15 (0.58–2.13) 
 LOS in d  <.001   
  0–1 Reference  — Reference 
  2–5 1.22 (0.84–1.81)  — 1.15 (0.78–1.72) 
  ≥6 2.13 (1.41–3.25)  — 1.45 (0.90–2.33) 
 Weekday discharge 1.41 (0.98–2.08) .07 — 1.23 (0.84–1.85) 

OR, odds ratio; —, not applicable.

a

Passed univariate screen but eliminated from model by backward selection.

After backward selection, the multivariable regression model at admission included any admissions in the preceding 6 months, any ED visits in the preceding 6 months, number of CCC categories, and medical (versus surgical) admission. The c-statistic of this model was 0.65 (95% confidence interval [CI] 0.61–0.69).

When incorporating hospitalization and discharge characteristics, the model at discharge added discharge disposition, LOS, and weekday discharge to the model at admission, and resulted in a c-statistic of 0.67 (95% CI 0.63–0.72).

In both prediction models, patients with any ED visit in the preceding 6 months had approximately twice the odds of readmission compared with patients without (Table 2). Number of CCC categories was also a significant risk factor for readmission because patients with 3 or more CCC categories had 1.7 to 2.3 times the odds of readmission compared with patients with only 1 CCC category. Admissions in the previous 6 months (versus none), medical (versus surgical) admissions, and discharge disposition home with services (compared to without services) imparted similar 1.7 to 1.8 increased odds of readmission.

For the model at admission, Table 3 presents the readmission risk scores, ranging from 0 to 7, depending on individual patient characteristics. Patients with the highest risk scores (≥5) had a readmission rate of 25% but accounted for <5% of admissions and only 13% of readmissions (Table 4). For comparison, risk scores for the model at discharge ranged from 0 to 12 (Supplemental Table 5) but had a similar risk score distribution as that seen for the model at admission. Patients with the highest risk scores (≥8) represented only 5% of admissions and 16% of readmissions but had a high readmission rate of 29% (Supplemental Table 6). For both models, the discriminative performance of the associated risk scores was identical to that of the final model, with c-statistics of 0.65 and 0.67.

TABLE 3

Thirty-Day Readmission Risk Scores for the Model at Admission

Risk Factorβ-CoefficientPointsa
Any admissions in previous 6 mo .53 
Any ED visits in previous 6 mo .71 
No. CCC categories   
 1 Reference 
 2 .55 
 ≥3 .83 
Medical admission .60 
Readmission risk score — Range 0–7 
Risk Factorβ-CoefficientPointsa
Any admissions in previous 6 mo .53 
Any ED visits in previous 6 mo .71 
No. CCC categories   
 1 Reference 
 2 .55 
 ≥3 .83 
Medical admission .60 
Readmission risk score — Range 0–7 

—, not applicable.

a

Points calculated by multiplying β-coefficients by 2.5 and rounding to the nearest integer; this allowed 1 or more variables to have a point value of 1, generating a more continuous distribution of risk scores, and differentiated between levels of the categorical variable. Points added to compute a readmission risk score.

TABLE 4

Thirty-Day Readmission Risk Score Distribution for the Model at Admission

Risk ScoreNo. Patients (Total = 2267)aCumulative Percentage of AdmissionsObserved Percentage ReadmittedCumulative Percentage of Observed Readmissions
5+ 95 25 13 
225 14 14 29 
333 29 13 52 
925 70 82 
223 79 89 
466 100 100 
Risk ScoreNo. Patients (Total = 2267)aCumulative Percentage of AdmissionsObserved Percentage ReadmittedCumulative Percentage of Observed Readmissions
5+ 95 25 13 
225 14 14 29 
333 29 13 52 
925 70 82 
223 79 89 
466 100 100 
a

Patients with missing data for number of home medications at admission were not included in model development nor this analysis.

No collinearity was found among the variables included in the final models, and no influential points were detected. Internal validation produced similar c-statistics, and slopes did not suggest substantial overfitting for both models. Calibration curves (Fig 1) did not indicate any particular pattern of risk over- or underestimation. Patients in the highest quartile of risk had 3.6 to 4.5 times the likelihood of readmission compared with patients in the lowest quartile, with observed readmission rates 3.6% to 4.9% in the lowest quartile vs 15.9% to 17.6% in the highest quartile for the 2 models.

FIGURE 1

Calibration plots by quartiles of 30-day readmission risk for the final models. Bars represent observed readmission %.

FIGURE 1

Calibration plots by quartiles of 30-day readmission risk for the final models. Bars represent observed readmission %.

Close modal

In this study, we aimed to identify independent risk factors for readmission among children with CCCs and to develop clinical prediction models and novel readmission risk scoring systems, providing a practical application for the models in real time, which has not been accomplished in previous pediatric readmission studies. All of the predictive factors identified in our models could be viewed as reflecting either degree of medical complexity or illness severity, although the possibility of unmeasured or nonmedical factors cannot be excluded. Both prediction models were capable of separating patients into quartiles or risk score levels with clinically meaningful differences in the likelihood of readmission, with approximately a fourfold higher readmission rate in the highest compared to the lowest risk quartile.

The modest level of discrimination achieved by our models and risk scores (c-statistic 0.65–0.67) is similar to that seen in previous adult and pediatric readmission studies using retrospective data and likely reflects the limitations of predicting readmission using data from EHRs alone.29,30  It also may reflect relative homogeneity of risk in the patient population because inclusion within our study sample was conditioned on medical complexity. The individual risk factors in our final prediction models support the findings observed in previous readmission studies involving CMC.6,1115,18,31  Our study also demonstrated that weekday versus weekend discharge did not maintain a statistically significant association with unplanned readmission in multivariable models, consistent with findings from a recent study in the general pediatric population by Auger and Davis.32  However, while some previous studies identified technology assistance and number of medications as risk factors for readmission among CMC, these factors did not retain sufficient predictive value within our study population to be included in the final multivariable models after controlling for other measures of medical complexity (eg, number of CCC categories).6,13,15,18  Some previous pediatric studies have also demonstrated that social determinants of health reflected in demographic variables, including race, public insurance, or income were predictive of readmission.29,31,3335  Interestingly, none of these variables had sufficiently significant association with readmission for inclusion in our final models. This may indicate that social determinants of health are less predictive of readmission than measures of medical complexity or illness severity within a population of CMC, particularly given nearly universal insurance coverage in this population. This difference could also be attributed to the location of our study (a state with long-standing support for disadvantaged families and nearly universal health insurance) or may reflect the relative imprecision of measures of social determinants of health obtained through EHR data compared with directly surveying patients or family members (eg, neighborhood per capita income from zip code compared with self-report of individual household income).

As a contribution beyond previous studies, we developed clinical prediction models and readmission risk scores that allow key risk factors to be applied in clinical practice or research to identify higher risk patients for targeted interventions or resources. Although theoretically all CMC could be targeted through larger programmatic interventions such as initiation of a complex care coordination program, this approach requires significant resources that may not be attainable by all institutions caring for these children. Risk scores from our prediction models could provide practitioners and investigators with an evidence-based and efficient approach to target patients at highest risk with focused clinical interventions specifically aimed at reducing readmission, with risk score thresholds tailored to the local population, costs, effectiveness, and/or burden on the specific institution for individual interventions. For example, if strategically targeting an expensive intervention or one with limited dedicated personnel (eg, case managers or patient navigators), providers could use a risk score threshold of ≥3 (Table 4), efficiently focusing efforts on 29% of admissions for CMC, yet 52% of readmissions. Because hospitalization and discharge-specific factors only modestly improved the readmission models, clinicians and investigators could use the model with only admission characteristics to target interventions applicable both during inpatient and postdischarge settings for an early, simplified, and practical approach.

The risk factors we identified by our models could also guide the development of interventions aimed at reducing readmission among CMC. For instance, because discharge disposition from the hospital with services was a marker for being at particularly high risk for readmission, improving or enhancing communication between the hospital service team and home service providers could be considered.

The strengths of our study include the large sample size; the use of sociodemographic and clinical variables routinely captured by health systems, facilitating use of our models by others and replication using larger national data sets; the translation of our models into useable risk scores for practical real-time application; robust methods to identify patients with medical complexity; and the exclusion of planned readmissions, which is not always accomplished in readmission studies.30  By creating a model using only characteristics obtainable at admission, our study predictions could be used to target either inpatient or outpatient interventions and to guide discharge planning.

Our study has several limitations. Given that this study was performed with pediatric patients with medical complexity hospitalized at one urban academic center, our findings may not be generalizable to patients in different geographic locations or different hospital systems or to patients in a broader general pediatric population. Although internal validation did not suggest substantial overfitting, external validation would be necessary to establish generalizability. Our study cohort’s readmission percentage (8.2%) is also lower than some readmission rates reported in other studies of CMC, which could reflect our inability to capture readmissions to other hospitals or could possibly be explained by differences in case mix, complexity, or care practices at our institution compared with others or near-universal health insurance coverage in our state.6  In using International Classification of Diseases, Ninth Revision and International Classification of Diseases, 10th Revision codes to define the presence of CCCs, patients with newly acquired conditions may have been underrepresented. Additionally, variables considered for our models were limited to those available in the EHR, so we were unable to assess more contextually complex medical, social, or potential outpatient drivers of readmission such as follow-up care.

We determined which characteristics independently predict 30-day readmission among children with CCCs discharged from a tertiary care center, and the resultant clinical prediction models and readmission risk scores should aid in identifying patients at highest risk for readmission. Our findings could facilitate strategic targeting of interventions to decrease the risk of these costly readmissions, thereby improving readmission outcomes and patient and family experiences while decreasing health care use and health care costs. Future proposals could involve prospective studies validating the prediction models, qualitative studies to determine more nuanced reasons behind readmissions, and development and evaluation of interventions to improve care transitions and care coordination to reduce readmissions among CMC at especially high risk.

Dr Leary conceptualized and designed the study, supervised data collection, conducted the initial analyses, drafted the initial manuscript, and reviewed and revised the manuscript; Drs Wong, Kent, and Freund designed the study and critically reviewed and revised the manuscript; Ms Scott collected data and reviewed and revised the manuscript; Ms Price designed the study, supervised data collection and analyses, and reviewed and revised the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Dr Leary was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, grant 5TL1TR0001062-03. The project described was supported by the National Center for Advancing Translational Sciences, National Institutes of Health, award UL1TR001064. The content is solely the responsibility of the authors and does not necessarily represent the official views of the NIH. Funded by the National Institutes of Health (NIH).

1
Wise
PH
.
The transformation of child health in the United States
.
Health Aff (Millwood)
.
2004
;
23
(
5
):
9
25
2
Cohen
E
,
Kuo
DZ
,
Agrawal
R
, et al
.
Children with medical complexity: an emerging population for clinical and research initiatives
.
Pediatrics
.
2011
;
127
(
3
):
529
538
3
Simon
TD
,
Berry
J
,
Feudtner
C
, et al
.
Children with complex chronic conditions in inpatient hospital settings in the United States
.
Pediatrics
.
2010
;
126
(
4
):
647
655
4
Feudtner
C
,
Feinstein
JA
,
Zhong
W
,
Hall
M
,
Dai
D
.
Pediatric complex chronic conditions classification system version 2: updated for ICD-10 and complex medical technology dependence and transplantation
.
BMC Pediatr
.
2014
;
14
:
199
5
Newacheck
PW
,
Kim
SE
.
A national profile of health care utilization and expenditures for children with special health care needs
.
Arch Pediatr Adolesc Med
.
2005
;
159
(
1
):
10
17
6
Cohen
E
,
Berry
JG
,
Camacho
X
,
Anderson
G
,
Wodchis
W
,
Guttmann
A
.
Patterns and costs of health care use of children with medical complexity
.
Pediatrics
.
2012
;
130
(
6
).
7
Neff
JM
,
Sharp
VL
,
Muldoon
J
,
Graham
J
,
Myers
K
.
Profile of medical charges for children by health status group and severity level in a Washington State Health Plan
.
Health Serv Res
.
2004
;
39
(
1
):
73
89
8
Mallory
LA
,
Osorio
SN
,
Prato
BS
, et al
;
IMPACT Pilot Study Group
.
Project IMPACT pilot report: feasibility of implementing a hospital-to-home transition bundle
.
Pediatrics
.
2017
;
139
(
3
):
e20154626
9
Berry
JG
,
Hall
M
,
Neff
J
, et al
.
Children with medical complexity and Medicaid: spending and cost savings
.
Health Aff (Millwood)
.
2014
;
33
(
12
):
2199
2206
10
Leyenaar
JK
,
Lagu
T
,
Shieh
M-S
,
Pekow
PS
,
Lindenauer
PK
.
Management and outcomes of pneumonia among children with complex chronic conditions
.
Pediatr Infect Dis J
.
2014
;
33
(
9
):
907
911
11
Coller
RJ
,
Nelson
BB
,
Sklansky
DJ
, et al
.
Preventing hospitalizations in children with medical complexity: a systematic review
.
Pediatrics
.
2014
;
134
(
6
).
12
Brittan
MS
,
Sills
MR
,
Fox
D
, et al
.
Outpatient follow-up visits and readmission in medically complex children enrolled in Medicaid
.
J Pediatr
.
2015
;
166
(
4
):
998
1005.e1
13
Jurgens
V
,
Spaeder
MC
,
Pavuluri
P
,
Waldman
Z
.
Hospital readmission in children with complex chronic conditions discharged from subacute care
.
Hosp Pediatr
.
2014
;
4
(
3
):
153
158
14
Hudson
SM
.
Hospital readmissions and repeat emergency department visits among children with medical complexity: an integrative review
.
J Pediatr Nurs
.
2013
;
28
(
4
):
316
339
15
Berry
JG
,
Agrawal
R
,
Kuo
DZ
, et al
.
Characteristics of hospitalizations for patients who use a structured clinical care program for children with medical complexity
.
J Pediatr
.
2011
;
159
(
2
):
284
290
16
Gay
JC
,
Thurm
CW
,
Hall
M
, et al
.
Home health nursing care and hospital use for medically complex children
.
Pediatrics
.
2016
;
138
(
5
):
e20160530
17
Stephens
JR
,
Kimple
KS
,
Steiner
MJ
,
Berry
JG
.
Discharge interventions and modifiable risk factors for preventing hospital readmissions in children with medical complexity
.
Rev Recent Clin Trials
.
2017
;
12
(
4
):
290
297
18
O’Brien
JE
,
Dumas
HM
,
Fragala-Pinkham
MA
,
Berry
JG
.
Admissions to acute care within 30 and 90 days of discharge home from a pediatric post-acute care hospital
.
Hosp Pediatr
.
2017
;
7
(
11
):
682
685
19
Rodday
AM
,
Graham
RJ
,
Weidner
RA
,
Terrin
N
,
Leslie
LK
,
Parsons
SK
.
Predicting health care utilization for children with respiratory insufficiency using parent-proxy ratings of children’s health-related quality of life
.
J Pediatr Health Care
.
2017
;
31
(
6
):
654
662
20
Coller
RJ
,
Klitzner
TS
,
Lerner
CF
, et al
.
Complex care hospital use and postdischarge coaching: a randomized controlled trial
.
Pediatrics
.
2018
;
142
(
2
):
e20174278
21
Berry
JG
,
Toomey
SL
,
Zaslavsky
AM
, et al
.
Pediatric readmission prevalence and variability across hospitals [published correction appears in JAMA. 2013;309(10):986]
.
JAMA
.
2013
;
309
(
4
):
372
380
22
Khan
A
,
Nakamura
MM
,
Zaslavsky
AM
, et al
.
Same-hospital readmission rates as a measure of pediatric quality of care
.
JAMA Pediatr
.
2015
;
169
(
10
):
905
912
23
United States Census Bureau
.
Available at: https://factfinder.census.gov. Accessed June 1, 2017
24
Burnham
KP
,
Anderson
DR
.
Multimodel inference: understanding AIC and BIC in model selection
.
Sociol Methods Res
.
2004
;
33
(
2
):
261
304
25
Friedman
J
,
Hastie
T
,
Tibshirani
R
.
Sparse inverse covariance estimation with the graphical lasso
.
Biostatistics
.
2008
;
9
(
3
):
432
441
26
Sing
T
,
Sander
O
,
Beerenwinkel
N
,
Lengauer
T
.
ROCR: visualizing classifier performance in R
.
Bioinformatics
.
2005
;
21
(
20
):
3940
3941
27
Hosmer
DW
,
Hosmer
T
,
Le Cessie
S
,
Lemeshow
S
.
A comparison of goodness-of-fit tests for the logistic regression model
.
Stat Med
.
1997
;
16
(
9
):
965
980
28
Steyerberg
EW
,
Harrell
FE
 Jr.
,
Borsboom
GJ
,
Eijkemans
MJ
,
Vergouwe
Y
,
Habbema
JD
.
Internal validation of predictive models: efficiency of some procedures for logistic regression analysis
.
J Clin Epidemiol
.
2001
;
54
(
8
):
774
781
29
Sills
MR
,
Hall
M
,
Cutler
GJ
, et al
.
Adding social determinant data changes children’s hospitals’ readmissions performance
.
J Pediatr
.
2017
;
186
:
150
157.e1
30
Kansagara
D
,
Englander
H
,
Salanitro
A
, et al
.
Risk prediction models for hospital readmission: a systematic review
.
JAMA
.
2011
;
306
(
15
):
1688
1698
31
Feudtner
C
,
Levin
JE
,
Srivastava
R
, et al
.
How well can hospital readmission be predicted in a cohort of hospitalized children? A retrospective, multicenter study
.
Pediatrics
.
2009
;
123
(
1
):
286
293
32
Auger
KA
,
Davis
MM
.
Pediatric weekend admission and increased unplanned readmission rates
.
J Hosp Med
.
2015
;
10
(
11
):
743
745
33
Berry
JG
,
Hall
DE
,
Kuo
DZ
, et al
.
Hospital utilization and characteristics of patients experiencing recurrent readmissions within children’s hospitals
.
JAMA
.
2011
;
305
(
7
):
682
690
34
Ju
M
,
Luna
N
,
Park
KT
.
The effect of limited English proficiency on pediatric hospital readmissions
.
Hosp Pediatr
.
2017
;
7
(
1
):
1
8
35
Parikh
K
,
Berry
J
,
Hall
M
, et al
.
Racial and ethnic differences in pediatric readmissions for common chronic conditions
.
J Pediatr
.
2017
;
186
:
158
164.e1

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data