A Community Development Program and Reduction in High-Cost Health Care Use

Healthy Neighborhoods Healthy Families (HNHF) is a multiagency community development initiative situated on the south side of Columbus, Ohio, in a neighborhood traditionally referred to as Southern Orchards. According to the 2010 census, this 3–zip code area, located adjacent to Nationwide Children’s Hospital (NCH), was characterized by one of the region’s highest vacancy rates. Vacancy rate, defined as the proportion of housing units in a defined area that are vacant or abandoned over a defined time period, can reflect neighborhood blight and is associated with multiple negative health outcomes.¹²  The neighborhood also has a history of high crime rates and poor school performance. One-third of households had income <100% of the federal poverty line, and only 2 in 5 households were owner occupied (37.8%). The HNHF program’s central goal is to enhance the health and well-being of families residing in this neighborhood.

The HNHF’s multifaceted approach to community improvement targets housing, health, education, resident engagement, employment, and safety. Components include a housing program, school- and community-based mentoring, job training and placement programs, school-based health clinics, neighborhood cleanups, and more. The cornerstone of the HNHF effort is its housing program, coordinated through Healthy Homes, a nonprofit housing organization created through a partnership between NCH and Community Development for All People, an area faith-based organization. The program’s primary initiatives are focused on stabilizing homeownership, subsidizing owner-occupied home repair, and expanding access to affordable rental units. Healthy Homes started modestly in 2008, impacting a total of 23 homes (3 gut rehabilitations and 20 home repairs) in its first 2 years and ramping up to an average of 34 homes per year. Since its inception, Healthy Homes has impacted >389 residential properties and generated >$40 million in direct and indirect investment in the surrounding neighborhoods. HNHF program evaluation documented measurable community improvement in reducing vacancy rates and increasing high school graduation.¹³ 

Although the primary goals of HNHF have always been neighborhood stabilization and improvement, NCH, as a health care provider, has often questioned whether this work is also enhancing the health of children and families. As such, a team of researchers and program implementers came together to frame some challenging questions, including the following:

• Is it reasonable to expect health changes at the neighborhood level as a result of this work, particularly within a relatively short implementation period?

• If so, what are the changes we should be measuring?

• How can we measure these changes when we did not have an integrated research and/or evaluation strategy at the start of the project?

It was ultimately decided that improvement in neighborhood-level health and well-being indicators was a reasonable, if lofty, expectation, and that innovative methodologic approaches leveraging multiple data sources could allow us to, at a minimum, detect an effect “signal.” On the basis of this decision, the research team, including experts in epidemiology, geography, health services research, and qualitative methods, joined with the Healthy Homes’ administrative, operational, and physician leadership to implement an ambitious measurement plan, starting with a study of health care use change. In this analysis of health care use, we treat HNHF as a cohesive “intervention,” but we note that over the past decade, HNHF has added and removed initiatives on the basis of program successes and failures and the expressed needs of residents. No single component of the program can be studied independently.

In his quasi-experimental study, we compared health care use in the HNHF program neighborhood (intervention) to use in local comparison neighborhoods. To reduce threats to internal validity resulting from nonrandom assignment to residence in the HNHF area, we employed a novel method to select the comparison neighborhoods. A propensity score model (PSM) was developed, which predicted the probability intervention and nonintervention census tracts were similar at baseline, in a manner consistent with previous observational research.^14,15  Propensity score modeling is a statistical method that may be used when researchers cannot randomly assign individuals to the intervention state, such as in this project in which people could not be randomly assigned to a neighborhood of residence. The method statistically balances characteristics of the intervention and control groups at the start of the study in a way that attempts to mimic random assignment. We specifically incorporated measures related to “neighborhood effects” and neighborhood housing structure. These neighborhood-level measures were obtained at baseline (∼2010) and included demographics (percentage that was black, neighborhood deprivation index, and population density), crime (total crime rate, violent crime rate, and property crime rate), housing instability (foreclosure rate, eviction rate, and percentage of properties vacant), neighborhood housing variables (percentage for rent, median year built, median home value, percentage of residential properties below average condition, and residential properties median sale price), racial segregation (mortgage lending racial bias index, percentage redlined), and pollution (traffic-related air pollution index). Redlining refers to the historical practice of mortgage lenders drawing red lines around areas in which they did not wish to make loans, often on the basis of racial bias.¹⁶  The process was made unlawful under the Fair Housing Act in 1968, but the impact on neighborhoods has persisted.

Because HNHF investment was not uniform across tracts of intervention, and to test the robustness of our PSM to changes in treatment designation, we examined differences in PSM results using 3 potential intervention neighborhood definitions: all tracts where HNHF invested in properties, majority rental investment tracts, and majority homeowner-based investment tracts. However, resulting high-propensity tracts, meaning those that best matched the investment tracts, were similar across all PSMs, which warranted the same comparison neighborhoods for our analysis regardless of investment type. Clustering high-propensity tracts were then aggregated to construct 2 comparison “neighborhoods,” near north and near west (Fig 1). Notably, these neighborhoods were consistent with ones that we had initially hypothesized on the basis of the team’s knowledge of the characteristics of neighborhoods in the city.

This method was used to effectively control for unobserved confounders. Near north and near west closely mirrored the intervention neighborhood, with similar values for housing structure, racial segregation, and socioeconomic conditions at baseline, indicated by similar mean values for PSM model variables (Table 1). The only significant difference at baseline between the intervention and comparison neighborhoods was the significantly smaller black population in near west.

Member-level claims and Medicaid eligibility data were obtained from the Medicaid accountable care organization serving all Medicaid-enrolled children in the HNHF and comparison neighborhoods under a data use agreement. Only members whose address fell into 1 of our 3 neighborhoods of interest during our 2-year pre- (August 2008 to July 2010) or post- (August 2015 to July 2017) study period were included. Residential addresses from the monthly eligibility files were geocoded by using an iterative process. First, we geocoded addresses using the US Census Bureau’s 2018 Topologically Integrated Geographic Encoding and Referencing/Line Shapefile Address Range feature for Franklin County.¹⁷  Addresses that received a match score <85 were then geocoded by using Google’s application programming interface geocoder in the R software package “ggmap.”¹⁸  We excluded 944 records because of incorrect, missing, or unmatchable address information. The total match rate was 98.9%. Member-level data for 61 727 Medicaid-eligible children ages 0 to 17 who resided in either the intervention or comparison neighborhoods at time of eligibility were included in our final analytic file. We excluded children who did not have complete age or sex data or resided in more than one neighborhood of interest during each period.

Our primary measures of use were number of emergency department (ED) visits, probability of hospital admission (inpatient admission), and length of stay (LOS) for those admitted. Covariates of interest included member-level demographic factors from the administrative database: sex (male or female), age, and number of months eligible for Medicaid while residing in neighborhoods of interest.

This research was approved by the NCH Institutional Review Board.

For each neighborhood and time period, ED use rates were calculated as the number of ED visits per 100 Medicaid-eligible children. Inpatient admission probability was calculated as the number who were admitted per 100 Medicaid-eligible children. Average LOS was calculated as the sum of all admitted days per total admitted Medicaid-eligible children. We restricted LOS analyses to members in the 95th percentile for LOS to reduce the influence of high users.

We estimated a multivariate regression specification of a difference-in-differences (DID) model to estimate the relative change in each outcome between the pre- and postperiods for intervention and comparison neighborhoods. In DID, it is assumed that in the absence of the HNHF intervention, health indicators in these neighborhoods would follow a similar pattern of change over time. Our method for comparison selection was used to ensure that our intervention and comparison neighborhoods were exchangeable, excluding the HNHF exposure. However, although propensity score matching was done to ensure that neighborhoods were similar at baseline, we also included control variables to correct for potential imbalances in patient characteristics between intervention and comparison areas. All models controlled for child’s sex and age and the number of months of Medicaid eligibility. Logistic regression models were used to estimate DID coefficients for the inpatient admission outcome, and negative binomial regression models were used to estimate DID coefficients for analysis of the number of ED visits and LOS.

In our primary analysis, we compared use measures in the intervention area to the near-north and near-west comparison areas combined (eg, HNHF versus comparison). We also ran secondary analyses to compare the intervention neighborhood with each comparison area independently (eg, HNHF versus near north and HNHF versus near west). Predictive margins were calculated for each DID model to estimate the magnitude of intervention effect. We conducted 5 sensitivity analyses using the same models with different aggregations of high-propensity tracts to test the robustness of results on the basis of neighborhood definition (Supplemental Fig 3). Stata (Stata Corp, College Station, TX) and R were used to conduct all analyses.

Descriptive analysis of the change in ED visits, inpatient admissions, and LOS indicate that the rate of ED visits in the intervention neighborhood decreased 20.8% compared to 16.1% in the comparison area (the combined near-north and near-west neighborhoods), whereas inpatient admissions decreased 12.7% in the intervention neighborhood compared to 12.2% in the comparison areas (Table 2). The average LOS increased 28.9% in the intervention neighborhood but increased by 36.6% in the comparison areas. This indicates slightly greater decreases in ED and inpatient use in the intervention area relative to the comparison areas and slightly shorter increases in LOS (Supplemental Table 6).

DID model results for the primary analysis indicate no statistically significant difference between intervention and comparison areas over time for any of the outcomes (Table 3). Models indicated a small but not statistically significant decrease in the ED visit count (adjusted DID estimate = −0.05; 95% confidence interval [CI]: −0.11 to 0.00) and probability of an inpatient admission (DID = −0.05; CI: −0.17 to 0.07) as well as a smaller increase in LOS for members in the intervention neighborhood relative to the comparison neighborhoods (DID = −0.06; CI: −0.14 to 0.03). The main effect for time in the models indicates that both intervention and comparison neighborhoods experienced a significant decrease in ED visits (β = −.19; CI: = −0.22 to −0.16) and increase in LOS (β =.32; CI: 0.27 to 0.37) over time, whereas, in the main effect for the intervention, a difference between intervention and comparison neighborhoods is indicated only for ED visits (β = −.06; CI: −0.10 to −0.02). There were no significant main effects for treatment or time on the probability of an inpatient admission.

Predicted probabilities and counts are used to help visualize the main effects of time and neighborhood for each use measure over the study period (Fig 2). The predicted count of ED visits per member was greater in the aggregated comparison area (1.04 visits; CI: 1.019 to 1.07) relative to the intervention (0.98 visits; CI: 0.95 to 1.016) before the HNHF investment. Over the study period, both areas saw a significant decrease in ED visits per eligible member (intervention ED visit ∆ = −0.21 visits [CI: −0.25 to −0.17]; primary ED visit ∆ = −0.18 visits [CI: −0.21 to −0.15]). Before the intervention, the average LOS was not significantly different between members in the intervention and comparison areas (intervention = 2.91 days [CI: 2.76 to 3.06]; primary = 2.88 days [CI: 2.77 to 2.99]). However, there was a significant increase in LOS over the study period in both areas (intervention LOS ∆ = 0.89 days [CI: 0.65 to 1.14]; comparison LOS ∆ = 1.1 days [CI: 0.93 to 1.28]).

DID models comparing the HNHF treatment area to the near-north neighborhood indicate similar results to the primary analysis (Table 4). There were no differences between treatment and control neighborhoods over time for any of the outcomes. Significant main effects for time across all 3 outcomes indicate a decrease in inpatient admissions (β = −.19; CI: −0.29 to −0.08), decrease in ED visits (β = −.23; CI: −0.28 to −0.18), and increase in LOS (β =.25; CI: 0.17 to 0.32) over time, whereas the main effect for the intervention indicates a difference between intervention and control neighborhoods at baseline only for ED visits (β =.09; CI: 0.04 to 0.14).

DID models comparing the HNHF treatment area to the near-west neighborhood indicate significant differences between these areas over time (Table 5). HNHF investment was used to positively impact the number of ED visits (DID = −0.07; CI: −0.14 to −0.01), probability of an inpatient admission (DID = −0.15; CI: −0.29 to −0.02), and expected LOS for those admitted (DID = −0.11; CI: −0.21 to −0.01). Predicted counts reveal the intervention neighborhood would have expected 0.03 more ED visits per member if the intervention had not taken place. Likewise, although members in the HNHF intervention had an increased LOS, they were expected to spend 0.4 more days admitted. They also would have expected a 0.01 greater probability of inpatient admission.

Sensitivity analyses provided mixed results. In sensitivity aggregations 1 to 3, there were no significant DID estimates for ED visits and LOS use measures. This was consistent across the aggregated and disaggregated models. In sensitivity aggregations 4 and 5, findings from the original tract aggregations were confirmed. LOS was significantly decreased in the intervention neighborhood relative to the near-west neighborhood by using these aggregations. The number of ED visits also significantly decreased in the intervention neighborhood relative to the near-west neighborhood as well as the primary neighborhood. The sensitivity aggregations used to examine changes in inpatient probability were mixed. Again, the probability was decreased for the intervention neighborhood relative to the near-west neighborhood in aggregations 4 and 5. However, in sensitivity aggregations 2 to 4, the probability of an inpatient admission decreased more in the near-north neighborhood relative to the intervention neighborhood, and these differences were significant. These results imply that findings are highly dependent on neighborhood definition.

Our findings suggest that a community development initiative may influence health care use for children at the neighborhood level. However, the effect was modest, limited to ED visits, and significant in only one of our propensity score–matched communities.

Beyond revealing the potential for impact, this work reveals that midimplementation evaluative research based on observational data is feasible, even when this was not envisioned at the start of the program. Lessons learned in this approach include the following:

1. Expecting large-scale changes in neighborhood health outcomes that were not the specific target of the intervention may be unrealistic. We hypothesized that our broad-based improvements in the residential environment would have an indirect effect on health care use, but it is highly likely that more proximate person-level health factors overwhelmed that effect. The cumulative effects of childhood toxic stress, epigenetic changes, and adverse childhood experiences are known to become evident later in adulthood.¹⁹  Our 10-year program period, which includes the early years of ramp-up, may not be sufficient to measure longer-term effects. Rather than being disheartened by the modest changes, we must instead recognize that although these findings are early and small, they serve to open the discussion of return on investment to the health care system.

2. Measurement is hard. Identifying appropriate comparison communities required the use of geographic and statistical approaches leveraging multiple data sources, and, ultimately, our matches were still not perfect. The fact that the magnitude of the intervention effect differed when different comparator communities were used highlights the fact that the unique features that make neighborhoods “neighborhoods” are unlikely to ever be fully controlled for in a research model. As more comparative studies of place-based interventions are conducted, best practices for research and evaluation design will further crystalize.

3. Even using strong statistical methods, this in an ecologic approach, and we cannot claim to have “caused” the observed changes, nor can we pinpoint which components of the multifaceted intervention are driving them. This is acceptable (even if it frustrates researchers accustomed to testing discrete treatments). True community-engaged, iteratively designed initiatives will never generate the strength of evidence associated with randomized trials, but, on the other hand, randomized trials do not reflect the real world. There is a place for both.

4. Results must be considered as part of the big picture. Such work must be done with attentiveness to displacement of the poorest residents, which may make use data look better but not represent true improvements for the long-time residents of the neighborhood. This was the first question raised by stakeholders who previewed these results, and the concern regarding gentrification is real. To date, Medicaid births in the affected census tracts in our study have not declined, suggesting no major displacement of low-income families, but vigilance to this concern will continue in future phases of research.

A community development intervention emanating from a hospital-community partnership was associated with some decreases in health care use for all children in the target neighborhood compared to matched comparison neighborhoods in the first years after launching. This outcomes evaluation case study demonstrates that neighborhood-level outcomes can and should be measured, but such measurement requires multidisciplinary teams, scientific rigor, and patience.

We thank Gretchen West and Angela Mingo from the HNHF team for their thoughtful advising on this article. We also thank the Partners For Kids data team for their assistance in creating the analytic data set.

CI

confidence interval

DID

difference-in-differences

ED

emergency department

HNHF

Healthy Neighborhoods Healthy Families

LOS

length of stay

NCH

Nationwide Children’s Hospital

PSM

propensity score model