Assessing the Quality Measure for Follow-up Care After Children’s Psychiatric Hospitalizations

This study used administrative data from Alabama’s stand-alone CHIP, ALL Kids, from 2013 to 2016 to calculate the quality measure for follow-up after psychiatric hospitalizations. Throughout this time period, ALL Kids coverage was available in 12-month enrollment periods to Alabama children age 18 years and younger with family income between 100% and 300% of the federal poverty level (FPL). Annual premiums and copayments for services are determined by family income as a percentage of the FPL. This study was approved by an institutional review board.

We constructed the FUH measure according to the CMS technical specifications manual.¹⁸  This measure is stewarded by the National Committee for Quality Assurance and uses Healthcare Effectiveness Data and Information Set (HEDIS) value set specifications for diagnosis and procedure codes, providers, and revenue center codes (ie, what hospital department performed the procedure). First, to calculate the denominator for this measure, children aged 6 to 18 years with a qualifying hospital discharge for acute psychiatric conditions were identified on the basis of the HEDIS value set. Then, all nonacute inpatient stays (based on HEDIS value sets) were excluded. Next, eligibility was confirmed on the basis of continuous enrollment for at least 30 days from discharge. Finally, exclusions were made for any transfers or readmissions within 30 days if those events were nonacute inpatient stays or nonpsychiatric diagnoses. Per measure specifications, all readmissions and transfers for psychiatric diagnoses within 30 days were combined into single episodes. The numerator was calculated for a follow-up visit within 7 to 30 days. Qualifying follow-up visits required specific procedure codes for care from a mental health provider or outpatient visits to specified types of behavioral health care settings determined from revenue center codes.

Enrollee characteristics were derived from ALL Kids administrative systems. Eligibility categories based on the FPL and race and/or ethnicity were identified from self-reported information provided by enrollee families on the enrollment application. Categories corresponding to the cost-sharing group included “low fee” (between 146% and 156% of the FPL), “fee” (between 157% and 208% of the FPL), and “expansion” (between 209% and 317% of the FPL). A fourth group of children, the “no fee” group, was exempted from cost-sharing by federal criteria, such as those with American Indian heritage. We grouped enrollees into 3 mutually exclusive racial and/or ethnic categories: white, African American, and other. Two geographic measures were considered. First, rural-urban residence was based on the zip code of the enrollment address categorized into rural-urban commuting area codes. Second, to serve as a proxy variable for proximity to care, we constructed a measure of mental health provider density within 25 miles of an enrollee’s residence and each provider’s practice address using the geodetic distance from centroid to centroid.

To explore potential differences in follow-up related to unique characteristics of psychiatric conditions, index hospitalization primary diagnosis codes were combined into broader groupings of depression, mood disorders, behavioral disorders, bipolar disorder, psychosis, and an “other” category, which included schizophrenia, autism, adjustment disorders, and all other diagnoses (Supplemental Table 6). Length of stay for the index hospitalization was categorized as 0 to 3, 4 to 7, or 8 or more days.

To understand specific components of the measure that were challenging to achieve, we calculated the number of enrollees who would have met the measure under 2 different measure specifications when holding the denominator constant. First, we examined the impact of extending the follow-up window to 45 and 60 days postdischarge. Second, we recalculated the 30-day measure with primary care visits and any health care visits within 30 days of discharge. For primary care visits, we investigated whether some type of follow-up care with a child’s primary care provider may have been substituted for measure-defined care (ie, with a qualified mental health provider), which potentially could be a proxy measure of poor access to mental health providers. For the any health care visits, we considered any medical or dental claims (excluding pharmacy claims) paid during this time period to represent active use of health care services among enrollees.

Finally, we identified enrollees with hospitalizations and ED visits in the months after the psychiatric hospitalization. Among enrollees in the measure denominator, we identified a subset with at least 30 days of continuous enrollment before their psychiatric hospitalization and at least 120 days after discharge. Within the 120-day period, we identified psychiatric hospitalizations and ED visits consistent with HEDIS definitions and, separately, hospitalizations and ED visits for any reason. Comparison of this subset to the overall study population did not reveal substantive differences in any characteristics; however, the subset on average was slightly older and of lower FPL (results available on request). Given the magnitude of these differences, and adjustment for these characteristics in the models, we do not have evidence that these differences resulted in biases affecting internal validity or generalizability.

Bivariate comparisons of enrollee characteristics associated with receiving follow-up care were compared by using χ² tests. Multivariable comparisons were estimated with logit regression models controlling for all other characteristics. We report marginal effects at the means, which represent the percentage point change in the likelihood of the outcome for each predictor, holding all other factors constant at the mean value.

A sensitivity analysis was conducted to determine if a multilevel logit model was appropriate given the nesting of hospitalizations within hospitals. We observed 31 different hospitals; however, 95% of discharges took place at 11 facilities. We recoded any hospital with <2.5% of observations into the other category, representing 5% of observations. The multilevel modeling approach allowed us to partition the total variance in follow-up into individual- and hospital-level components. From the intraclass correlation coefficient, hospital-level variation explained just 3% of the total variation in follow-up within 7 days and 1% within 30 days. Coefficients from covariates were consistent with single-level logit models. Thus, we opted to report single-level logit models with hospital fixed effects (ie, dummy variables) and clustered SEs accounting for repeated visits among enrollees.

Separately, we estimated the association of meeting the follow-up measure at 30 days with the likelihood of having psychiatric hospitalizations or ED visits using logit regression with clustered SEs to estimate marginal effects representing the percentage point change in likelihood, holding all other covariates constant at the mean.

Between 2013 and 2016, 1072 psychiatric hospitalizations among 866 enrollees met measure criteria for inclusion. Among those hospitalizations, 356 (33.2%) received follow-up care with a mental health provider within 7 days and 566 (52.8%) received follow-up care within 30 days. The extension of time to 45 or 60 days increased follow-up for 38 hospitalizations (3.5 percentage points) and 56 hospitalizations (5.2 percentage points), respectively. Among hospitalizations that otherwise had no qualifying follow-up care, an additional 8 (0.7 percentage points) had a primary care visit and 172 (23.4 percentage points) used any services.

Characteristics of enrollees receiving follow-up within 7 and 30 days are shown in Table 1. The second column from the left displays characteristics of all enrollees with psychiatric hospitalizations. Hospitalizations most commonly were among those aged 11 to 15 years (57.5%), girls (57.6%), white patients (69.0%), urban residents (71.4%), and those with a depression diagnosis (49.0%). The third column displays row percentages of enrollees who met the measure divided by the number of enrollees with that characteristic, enabling comparison vis-à-vis overall follow-up proportion. Girls (35.7%; P = .05), other multiracial and/or non–African American patients (42.2%; P = .02), small-town or rural residents (43.5%; P = .05), those with a primary diagnosis of depression (39.1%; P < .001), and those with a length of hospitalization of 0 to 3 days (43.1%; P < .001) had greater frequency of follow-up within 7 days. African American patients (28.1%; P = .02), patients with a bipolar diagnosis (20.7%; P = .01), and patients with length of hospitalization of 8 or more days (22.9%; P < .001) had lower frequency of follow-up within 7 days. Finally, the fifth column displays factors associated with follow-up within 30 days. Increased frequency of follow-up included being female (55.9%, P = .02), having a primary diagnosis of depression (58.5%; P ≤ .001), and having a length of hospitalization of 0 to 3 days (63.6%; P < .001).

Individual characteristics associated with the likelihood of receiving follow-up care within 7 and 30 days estimated from logit models and controlling for all other covariates are shown in Table 2. The largest effects of reduced likelihood of follow-up within 7 days of discharge were observed for hospital length of stay, followed by age and diagnosis. Hospital stays of 8 days or more lowered the likelihood of follow-up by 18 percentage points relative to stays of 3 days or fewer (P < .001). Children aged 11 to 15 years were 13 percentage points less likely to receive follow-up compared with those aged 6 to 10 years (P = .03). Children with hospitalizations for mood disorders were 9 percentage points less likely to receive follow-up than children hospitalized for depression (P = .02). The largest effects of increased likelihood of follow-up within 7 days were observed for hospitalizations in 2014 relative to 2013 (9 percentage points; P = .03). This was followed by the effect on children who identified as being of multiracial or non–African American race and/or ethnicity compared with whites, who had an increased likelihood of 9 percentage points (P = .02). Characteristics with statistically significant associations with the likelihood of follow-up within 30 days were fewer than those observed for follow-up within 7 days, with only length of stay falling within conventional thresholds. Children hospitalized for 8 days or more were 26 percentage points less likely to receive follow-up within 30 days than those with hospitalizations of 3 days or fewer (P < .001).

Among 745 index hospitalizations wherein enrollees had a minimum of 30 days continuous enrollment before admission and 120 days postdischarge, the proportion with subsequent ED visits and hospitalizations is shown in Table 3. The majority of hospitalizations were psychiatric. Enrollees with follow-up care had lower frequencies of subsequent hospital and ED use, although many differences were not statistically significant at conventional levels. A statistically significant association was observed for psychiatric hospitalizations within 60 days of discharge; 3.0% of enrollees with follow-up care were rehospitalized compared with 6.0% of those without follow-up care (P = .05).

The estimated likelihoods of subsequent ED visits are shown in Table 4, and hospitalizations are shown in Table 5. After controlling for all covariates, having follow-up care was associated with a reduced likelihood of subsequent psychiatric hospitalizations and ED visits, ranging between 2 and 4 percentage points. However, statistical significance was not observed in the fully adjusted models.

The receipt of timely follow-up care after psychiatric hospitalization is low. On the basis of measure specifications, approximately half of hospitalizations received qualifying follow-up care within 30 days of discharge. Previous studies have consistently observed lower-than-optimal rates of follow-up among individuals, but comparisons are difficult to interpret given differences in populations by age, diagnosis, health insurance coverage (ie, Medicaid, private, etc), and inconsistent follow-up care specifications.^{10–12,14–17,19,20}  National performance on this measure is highly variable, and comparability across states and programs is difficult to ascertain.^21,22  Differences between Medicaid and CHIP notwithstanding, some state CHIPs report only on fee-for-service enrollees, whereas others include managed care or specific administrative subpopulations. Furthermore, although many states use external quality review organizations,²³  no single entity calculates performance across all state programs, and therefore, it is conceivable that differences in performance could be explained by interpretations of the measure, coding or billing practices, administrative subpopulation definitions, or other factors. We did not observe that follow-up care was sensitive to the measure definitions of timing or mental health provider criteria. Notably, we observed a sizeable proportion of enrollees without follow-up care who used services within 30 days of discharge, which is evidence that they were active in the health system. A greater understanding of this subpopulation may elucidate barriers to follow-up care.

Hospitalization length of stay was the only consistent factor associated with receipt of follow-up care within 7 and 30 days after controlling for covariates. Age and hospitalization primary diagnosis groups were associated only with follow-up in 7 days. We did not observe effects by hospital, provider density, geographic setting, cost-sharing strata, or race, all of which were hypothesized on the basis of findings from other studies.^{10–12,14–17,19,20}  However, differences could be attributable to studies focused only on adults,^12,15  Medicaid recipients,^16,19  ED visits (rather than hospitalizations),¹⁵  or disease-specific cohorts.^17,20  We also did not see large differences in performance over time despite the growing national emphasis on quality measurement after the Children’s Health Insurance Program Reauthorization Act of 2009 and release of the initial Child Core Set technical specifications in February 2011, just 2 years before the study period.²⁴  Previous studies have noted differences in follow-up care based on diagnosis. Compared with hospitalizations for depression, we observed that all other diagnoses received follow-up care less often. Depression diagnoses were the most common among all hospitalizations we observed, and the findings may reflect specific challenges of different conditions. Specifically, we observed that depression hospitalizations tended to have shorter lengths of stay (results available on request), whereas other diagnoses, such as bipolar disorder and psychosis, had longer durations. Additionally, because the HEDIS measure definition uses administrative claims data, it is not sensitive to characteristics of conditions, such as self-harm or suicidal behavior. This could influence follow-up on both the patient and provider side, which is consistent with the hypothesis that those with more severe illnesses are more likely to follow-up.²⁰  However, we observed that longer length of stay reduces the likelihood of follow-up, which may be counterintuitive to the severity hypothesis and require future investigation. For example, analyses with greater numbers of hospitalizations than we observed could parse out complex interactions between diagnosed conditions and the hospital length of stay. Likewise, more detailed information on disease characteristics and/or severity may enable a greater understanding of the unique challenges of follow-up care above and beyond diagnosis and length of stay.

Although modest in the absolute effect size, there appears to be a consistent benefit of receiving timely follow-up care in the reduction of subsequent psychiatric hospitalization. We observed subsequent hospitalizations and ED visits for psychiatric conditions to occur infrequently; the observed risk was between 4.4% and 10.6% in our population. Follow-up care was associated with large relative effects; in the case of reducing psychiatric hospitalizations within 60 days, the relative difference was 50% lower for children with follow-up care. Although timeliness is important to ensure that patients are stabilized and have a successful transition home, reducing subsequent hospitalizations may also be an indicator of linkage to stable outpatient treatment, improved disease management, or more efficient use of health services. We observed fewer subsequent hospitalizations for enrollees receiving follow-up care, but adjusting for covariates attenuated this association. Perhaps surprisingly, evidence in the scientific literature showing follow-up care has a demonstrable reduction in readmissions, subsequent hospitalizations, and ED visits is equivocal. Although some studies of Medicaid recipients observed that outpatient follow-up reduced psychiatric readmissions,^11,12  a systematic review by Daniel et al²⁵  did not find consistent evidence of a benefit of follow-up care, and subsequent studies had similar conclusions.^10,16,17  This counterintuitive observation may reflect variations in the quality of follow-up care or unmeasured disease severity.²⁶  Repeated use of high-acuity services, such as hospitalizations and ED visits, for psychiatric care is a complex phenomenon with many contributing demographic and clinical factors.²⁷  A strong and consistent predictor of repeated hospitalizations and ED visits is having a previous psychiatric hospitalization or ED visit.^27–29  Thus, follow-up care in the outpatient setting may reduce the reliance of the ED in the continuum of care for psychiatric care.²⁸  Despite the modest associations between follow-up care and subsequent hospitalizations and ED visits, quality-improvement efforts could aim to enhance the effectiveness of outpatient follow-up care. Furthermore, challenges and gaps in measuring quality of care remain, particularly for psychiatric care.^30–33  Our analyses highlight potential long-term effects (ie, reduced psychiatric hospitalizations and ED visits) that routine measurement may miss. Thus, mental health quality measures should continue to be reevaluated and continuously improved.

Our findings must be interpreted along with some limitations. Notably, the HEDIS measure we used is designed for administrative claims data, which are limited in ability to capture all sociodemographic factors, including parent and/or caregiver decisions and/or other barriers to care. We attempted to address this limitation by controlling for multiple covariates, but unmeasured confounding is possible. Furthermore, administrative data cannot capture all potential benefits of follow-up care, such as qualitative assessment of “smooth transitions” or quality of life. Our study period coincided with the start of guidance from the CMS to voluntarily track and report quality measures that are updated annually, and likewise, coding and/or billing practices may have changed, resulting in missing information about follow-up care. Finally, the generalizability beyond hospitalizations within Alabama’s CHIP may be limited, particularly in states with more or less availability of community-based mental health services and/or providers or among children with different family income ranges. It is possible that factors associated with use may differ in other states than what we observed in Alabama as well as the outcomes associated with follow-up care.

Opportunities to increase follow-up care after psychiatric hospitalizations exist within Alabama and nationally because many states report low percentages. We did not observe follow-up rates that were sensitive to measurement definitions. We identified a limited influence by modifiable characteristics. Although psychiatric conditions are complex, and individual patients may present unique challenges, we observed that conditions other than depression as well as longer length of stay were associated with reduced follow-up, suggesting that these patients’ conditions are especially challenging. We observed a modest benefit of follow-up care in reducing subsequent psychiatric care use.

We acknowledge the contributions of the late Dr Meredith Kilgore, whose methodical expertise and claims data experience were valuable in the development and completion of this study.