Video Abstract

Video Abstract

Close modal

Adverse housing and neighborhood conditions influence child health. The Healthy Neighborhoods Healthy Families community development initiative was established in 2008 to address housing, education, employment, and other neighborhood-level, child health–influencing factors on the south side of Columbus, Ohio, with the goal of improving child health and well-being. In this article, we discuss the path from advocacy to outcomes analysis in this initiative and assess changes in high-cost health care use by children in the target area over the first decade of implementation. Change in health care use was measured by using a difference-in-differences approach comparing emergency department visits, inpatient stays, and inpatient length of stay in the intervention neighborhood and a propensity score–matched, pooled comparator neighborhood in the same city. The baseline and follow-up periods were August 2008 to July 2010 and August 2015 to July 2017, respectively. Findings from this analysis reveal that compared to 2 pooled comparison neighborhoods, the intervention neighborhood trended, nonsignificantly, toward greater decreases in inpatient stays and emergency department visits and smaller increases in length of stays. These results suggest that our community development activities may be influencing health care use outcomes, but in the early years of the intervention relative changes are modest and are variable based on the definition of the intervention and comparator neighborhoods. Lessons learned in expanding from advocacy to analysis include the importance of building multidisciplinary teams that can apply novel approaches to analysis, moderating expectations, and retaining focus on the broader social context.

Neighborhood and housing factors have been shown to have an impact on a diverse set of child health outcomes, including asthma, injuries, substance use, and cognitive development.17  Community development has the potential to reduce negative health outcomes by changing factors affecting whole communities.8  Yet these programs face challenges in both implementation and evaluation.9  The potential of community development programs to address neighborhood needs in ways that improve health status and reduce unnecessary health care use is beginning to draw interest and investment by leading health systems, payers, and funders10 ; however, research on health-specific outcomes for these initiatives is lacking.11  In this article, we describe our process for adding an outcomes research component to a multipronged, health system–led community development intervention and present an analysis of changes in high-cost health care use for children living in the intervention neighborhood. We also discuss the methodologic challenges associated with evaluating community development using observational study designs.

Healthy Neighborhoods Healthy Families (HNHF) is a multiagency community development initiative situated on the south side of Columbus, Ohio, in a neighborhood traditionally referred to as Southern Orchards. According to the 2010 census, this 3–zip code area, located adjacent to Nationwide Children’s Hospital (NCH), was characterized by one of the region’s highest vacancy rates. Vacancy rate, defined as the proportion of housing units in a defined area that are vacant or abandoned over a defined time period, can reflect neighborhood blight and is associated with multiple negative health outcomes.12  The neighborhood also has a history of high crime rates and poor school performance. One-third of households had income <100% of the federal poverty line, and only 2 in 5 households were owner occupied (37.8%). The HNHF program’s central goal is to enhance the health and well-being of families residing in this neighborhood.

The HNHF’s multifaceted approach to community improvement targets housing, health, education, resident engagement, employment, and safety. Components include a housing program, school- and community-based mentoring, job training and placement programs, school-based health clinics, neighborhood cleanups, and more. The cornerstone of the HNHF effort is its housing program, coordinated through Healthy Homes, a nonprofit housing organization created through a partnership between NCH and Community Development for All People, an area faith-based organization. The program’s primary initiatives are focused on stabilizing homeownership, subsidizing owner-occupied home repair, and expanding access to affordable rental units. Healthy Homes started modestly in 2008, impacting a total of 23 homes (3 gut rehabilitations and 20 home repairs) in its first 2 years and ramping up to an average of 34 homes per year. Since its inception, Healthy Homes has impacted >389 residential properties and generated >$40 million in direct and indirect investment in the surrounding neighborhoods. HNHF program evaluation documented measurable community improvement in reducing vacancy rates and increasing high school graduation.13 

Although the primary goals of HNHF have always been neighborhood stabilization and improvement, NCH, as a health care provider, has often questioned whether this work is also enhancing the health of children and families. As such, a team of researchers and program implementers came together to frame some challenging questions, including the following:

  • Is it reasonable to expect health changes at the neighborhood level as a result of this work, particularly within a relatively short implementation period?

  • If so, what are the changes we should be measuring?

  • How can we measure these changes when we did not have an integrated research and/or evaluation strategy at the start of the project?

It was ultimately decided that improvement in neighborhood-level health and well-being indicators was a reasonable, if lofty, expectation, and that innovative methodologic approaches leveraging multiple data sources could allow us to, at a minimum, detect an effect “signal.” On the basis of this decision, the research team, including experts in epidemiology, geography, health services research, and qualitative methods, joined with the Healthy Homes’ administrative, operational, and physician leadership to implement an ambitious measurement plan, starting with a study of health care use change. In this analysis of health care use, we treat HNHF as a cohesive “intervention,” but we note that over the past decade, HNHF has added and removed initiatives on the basis of program successes and failures and the expressed needs of residents. No single component of the program can be studied independently.

In his quasi-experimental study, we compared health care use in the HNHF program neighborhood (intervention) to use in local comparison neighborhoods. To reduce threats to internal validity resulting from nonrandom assignment to residence in the HNHF area, we employed a novel method to select the comparison neighborhoods. A propensity score model (PSM) was developed, which predicted the probability intervention and nonintervention census tracts were similar at baseline, in a manner consistent with previous observational research.14,15  Propensity score modeling is a statistical method that may be used when researchers cannot randomly assign individuals to the intervention state, such as in this project in which people could not be randomly assigned to a neighborhood of residence. The method statistically balances characteristics of the intervention and control groups at the start of the study in a way that attempts to mimic random assignment. We specifically incorporated measures related to “neighborhood effects” and neighborhood housing structure. These neighborhood-level measures were obtained at baseline (∼2010) and included demographics (percentage that was black, neighborhood deprivation index, and population density), crime (total crime rate, violent crime rate, and property crime rate), housing instability (foreclosure rate, eviction rate, and percentage of properties vacant), neighborhood housing variables (percentage for rent, median year built, median home value, percentage of residential properties below average condition, and residential properties median sale price), racial segregation (mortgage lending racial bias index, percentage redlined), and pollution (traffic-related air pollution index). Redlining refers to the historical practice of mortgage lenders drawing red lines around areas in which they did not wish to make loans, often on the basis of racial bias.16  The process was made unlawful under the Fair Housing Act in 1968, but the impact on neighborhoods has persisted.

Because HNHF investment was not uniform across tracts of intervention, and to test the robustness of our PSM to changes in treatment designation, we examined differences in PSM results using 3 potential intervention neighborhood definitions: all tracts where HNHF invested in properties, majority rental investment tracts, and majority homeowner-based investment tracts. However, resulting high-propensity tracts, meaning those that best matched the investment tracts, were similar across all PSMs, which warranted the same comparison neighborhoods for our analysis regardless of investment type. Clustering high-propensity tracts were then aggregated to construct 2 comparison “neighborhoods,” near north and near west (Fig 1). Notably, these neighborhoods were consistent with ones that we had initially hypothesized on the basis of the team’s knowledge of the characteristics of neighborhoods in the city.

FIGURE 1

Intervention and high-propensity comparator neighborhoods.

FIGURE 1

Intervention and high-propensity comparator neighborhoods.

Close modal

This method was used to effectively control for unobserved confounders. Near north and near west closely mirrored the intervention neighborhood, with similar values for housing structure, racial segregation, and socioeconomic conditions at baseline, indicated by similar mean values for PSM model variables (Table 1). The only significant difference at baseline between the intervention and comparison neighborhoods was the significantly smaller black population in near west.

TABLE 1

Baseline Mean Value Neighborhood-Level Characteristics for Franklin County, HNHF, Near North, and Near West From ∼2010

Neighborhood VariablesFranklin CountyHNHFNear NorthNear WestF Valuea
Demographics      
 Black, % 23.4 64.1 57.2 27.0 5.08b,* 
 Neighborhood deprivation index −0.07 1.43 1.26 1.55 0.26 
 Population density, per square mile 4384 6463 5568 6479 0.41 
Crime      
 Total crime rate, per 100 population 8.10 23.50 19.00 23.10 0.68 
 Violent crime rate, per 100 population 0.50 2.00 1.60 1.90 0.27 
 Property crime rate, per 100 population 2.60 6.40 5.40 6.90 0.90 
Housing instability      
 Mortgages foreclosed, % 6.9 13.2 13.6 11.7 0.71 
 Eviction rate, per 100 renter homes 5.48 10.69 10.65 9.96 0.23 
 Vacant buildings, % 4.3 13.8 15.7 8.8 4.85c,* 
Housing      
 Rent, % 43.7 54.9 50.6 55.2 0.23 
 Median year built 1968 1943 1948 1947 1.27 
 Median home value, $ 158 220 80 110 70 433 70 933 0.74 
 Condition below average: residential properties, % 5.3 19.3 14.8 16.3 0.49 
 Median sale price per sq ft in 2006–2009, $ 93.12 38.10 47.83 42.19 0.93 
Racial segregation      
 Mortgage lending racial bias index 3.93 3.80 1.19 2.46 0.95 
 Redlined, % 3.9 12.1 16.8 17.1 0.18 
Environment, traffic-related air pollution 150 m 5109 3368 1842 4715 2.44 
Neighborhood VariablesFranklin CountyHNHFNear NorthNear WestF Valuea
Demographics      
 Black, % 23.4 64.1 57.2 27.0 5.08b,* 
 Neighborhood deprivation index −0.07 1.43 1.26 1.55 0.26 
 Population density, per square mile 4384 6463 5568 6479 0.41 
Crime      
 Total crime rate, per 100 population 8.10 23.50 19.00 23.10 0.68 
 Violent crime rate, per 100 population 0.50 2.00 1.60 1.90 0.27 
 Property crime rate, per 100 population 2.60 6.40 5.40 6.90 0.90 
Housing instability      
 Mortgages foreclosed, % 6.9 13.2 13.6 11.7 0.71 
 Eviction rate, per 100 renter homes 5.48 10.69 10.65 9.96 0.23 
 Vacant buildings, % 4.3 13.8 15.7 8.8 4.85c,* 
Housing      
 Rent, % 43.7 54.9 50.6 55.2 0.23 
 Median year built 1968 1943 1948 1947 1.27 
 Median home value, $ 158 220 80 110 70 433 70 933 0.74 
 Condition below average: residential properties, % 5.3 19.3 14.8 16.3 0.49 
 Median sale price per sq ft in 2006–2009, $ 93.12 38.10 47.83 42.19 0.93 
Racial segregation      
 Mortgage lending racial bias index 3.93 3.80 1.19 2.46 0.95 
 Redlined, % 3.9 12.1 16.8 17.1 0.18 
Environment, traffic-related air pollution 150 m 5109 3368 1842 4715 2.44 
a

Analysis of variance between the HNHF, near-north and near-west neighborhood-level factors.

b

Significant difference between HNHF and near west.

c

Significant difference between near north and near west.

*

P < .05.

Member-level claims and Medicaid eligibility data were obtained from the Medicaid accountable care organization serving all Medicaid-enrolled children in the HNHF and comparison neighborhoods under a data use agreement. Only members whose address fell into 1 of our 3 neighborhoods of interest during our 2-year pre- (August 2008 to July 2010) or post- (August 2015 to July 2017) study period were included. Residential addresses from the monthly eligibility files were geocoded by using an iterative process. First, we geocoded addresses using the US Census Bureau’s 2018 Topologically Integrated Geographic Encoding and Referencing/Line Shapefile Address Range feature for Franklin County.17  Addresses that received a match score <85 were then geocoded by using Google’s application programming interface geocoder in the R software package “ggmap.”18  We excluded 944 records because of incorrect, missing, or unmatchable address information. The total match rate was 98.9%. Member-level data for 61 727 Medicaid-eligible children ages 0 to 17 who resided in either the intervention or comparison neighborhoods at time of eligibility were included in our final analytic file. We excluded children who did not have complete age or sex data or resided in more than one neighborhood of interest during each period.

Our primary measures of use were number of emergency department (ED) visits, probability of hospital admission (inpatient admission), and length of stay (LOS) for those admitted. Covariates of interest included member-level demographic factors from the administrative database: sex (male or female), age, and number of months eligible for Medicaid while residing in neighborhoods of interest.

This research was approved by the NCH Institutional Review Board.

For each neighborhood and time period, ED use rates were calculated as the number of ED visits per 100 Medicaid-eligible children. Inpatient admission probability was calculated as the number who were admitted per 100 Medicaid-eligible children. Average LOS was calculated as the sum of all admitted days per total admitted Medicaid-eligible children. We restricted LOS analyses to members in the 95th percentile for LOS to reduce the influence of high users.

We estimated a multivariate regression specification of a difference-in-differences (DID) model to estimate the relative change in each outcome between the pre- and postperiods for intervention and comparison neighborhoods. In DID, it is assumed that in the absence of the HNHF intervention, health indicators in these neighborhoods would follow a similar pattern of change over time. Our method for comparison selection was used to ensure that our intervention and comparison neighborhoods were exchangeable, excluding the HNHF exposure. However, although propensity score matching was done to ensure that neighborhoods were similar at baseline, we also included control variables to correct for potential imbalances in patient characteristics between intervention and comparison areas. All models controlled for child’s sex and age and the number of months of Medicaid eligibility. Logistic regression models were used to estimate DID coefficients for the inpatient admission outcome, and negative binomial regression models were used to estimate DID coefficients for analysis of the number of ED visits and LOS.

In our primary analysis, we compared use measures in the intervention area to the near-north and near-west comparison areas combined (eg, HNHF versus comparison). We also ran secondary analyses to compare the intervention neighborhood with each comparison area independently (eg, HNHF versus near north and HNHF versus near west). Predictive margins were calculated for each DID model to estimate the magnitude of intervention effect. We conducted 5 sensitivity analyses using the same models with different aggregations of high-propensity tracts to test the robustness of results on the basis of neighborhood definition (Supplemental Fig 3). Stata (Stata Corp, College Station, TX) and R were used to conduct all analyses.

Descriptive analysis of the change in ED visits, inpatient admissions, and LOS indicate that the rate of ED visits in the intervention neighborhood decreased 20.8% compared to 16.1% in the comparison area (the combined near-north and near-west neighborhoods), whereas inpatient admissions decreased 12.7% in the intervention neighborhood compared to 12.2% in the comparison areas (Table 2). The average LOS increased 28.9% in the intervention neighborhood but increased by 36.6% in the comparison areas. This indicates slightly greater decreases in ED and inpatient use in the intervention area relative to the comparison areas and slightly shorter increases in LOS (Supplemental Table 6).

TABLE 2

Percentage Change Differences in ED Visit Rates, Inpatient Admission Probability, and Average LOS per Member Pre- and Post-HNHF Intervention

NeighborhoodED VisitsaInpatient AdmissionbLOSc
Pre-HNHFPost-HNHFChange, %Pre-HNHFPost-HNHFChange, %Pre-HNHFPost-HNHFChange, %
HNHF 1.00 0.79 −20.8 0.103 0.090 −12.7 2.94 3.79 28.9 
Primary comparisond 1.05 0.88 −16.1 0.099 0.087 −12.2 2.91 3.97 36.6 
Secondary comparison          
 Near north 0.90 0.73 −18.6 0.104 0.083 −20.1 2.97 3.77 26.9 
 Near west 1.17 0.99 −15.2 0.096 0.090 −5.7 2.86 4.11 43.7 
NeighborhoodED VisitsaInpatient AdmissionbLOSc
Pre-HNHFPost-HNHFChange, %Pre-HNHFPost-HNHFChange, %Pre-HNHFPost-HNHFChange, %
HNHF 1.00 0.79 −20.8 0.103 0.090 −12.7 2.94 3.79 28.9 
Primary comparisond 1.05 0.88 −16.1 0.099 0.087 −12.2 2.91 3.97 36.6 
Secondary comparison          
 Near north 0.90 0.73 −18.6 0.104 0.083 −20.1 2.97 3.77 26.9 
 Near west 1.17 0.99 −15.2 0.096 0.090 −5.7 2.86 4.11 43.7 
a

Number of ED visits per 100 Medicaid-eligible children.

b

Number of inpatient admissions per 100 Medicaid-eligible children.

c

Sum of all admitted days per total admitted Medicaid-eligible children.

d

Pooled near-north and near-west neighborhood measures.

DID model results for the primary analysis indicate no statistically significant difference between intervention and comparison areas over time for any of the outcomes (Table 3). Models indicated a small but not statistically significant decrease in the ED visit count (adjusted DID estimate = −0.05; 95% confidence interval [CI]: −0.11 to 0.00) and probability of an inpatient admission (DID = −0.05; CI: −0.17 to 0.07) as well as a smaller increase in LOS for members in the intervention neighborhood relative to the comparison neighborhoods (DID = −0.06; CI: −0.14 to 0.03). The main effect for time in the models indicates that both intervention and comparison neighborhoods experienced a significant decrease in ED visits (β = −.19; CI: = −0.22 to −0.16) and increase in LOS (β =.32; CI: 0.27 to 0.37) over time, whereas, in the main effect for the intervention, a difference between intervention and comparison neighborhoods is indicated only for ED visits (β = −.06; CI: −0.10 to −0.02). There were no significant main effects for treatment or time on the probability of an inpatient admission.

TABLE 3

Adjusted DID Regression Results With Primary Comparison Group

ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF −.06 −0.1 to −0.02 .004 .07 −0.01 to 0.15 .09 .01 −0.05 to 0.07 .72 
Time −.19 −0.22 to −0.16 <.001 −.05 −0.12 to 0.02 .15 .32 0.27 to 0.37 <.001 
HNHF*time −.05 −0.11 to 0.00 .06 −.05 −0.17 to 0.07 .42 −.06 −0.14 to 0.03 .21 
ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF −.06 −0.1 to −0.02 .004 .07 −0.01 to 0.15 .09 .01 −0.05 to 0.07 .72 
Time −.19 −0.22 to −0.16 <.001 −.05 −0.12 to 0.02 .15 .32 0.27 to 0.37 <.001 
HNHF*time −.05 −0.11 to 0.00 .06 −.05 −0.17 to 0.07 .42 −.06 −0.14 to 0.03 .21 

All estimates include controls for child’s age, sex, and number of months they were Medicaid eligible while in the neighborhood of interest.

a

Number of ED visits per 100 Medicaid-eligible children.

b

Number of inpatient admissions per 100 Medicaid-eligible children.

c

Sum of all admitted days per total admitted Medicaid-eligible children.

Predicted probabilities and counts are used to help visualize the main effects of time and neighborhood for each use measure over the study period (Fig 2). The predicted count of ED visits per member was greater in the aggregated comparison area (1.04 visits; CI: 1.019 to 1.07) relative to the intervention (0.98 visits; CI: 0.95 to 1.016) before the HNHF investment. Over the study period, both areas saw a significant decrease in ED visits per eligible member (intervention ED visit ∆ = −0.21 visits [CI: −0.25 to −0.17]; primary ED visit ∆ = −0.18 visits [CI: −0.21 to −0.15]). Before the intervention, the average LOS was not significantly different between members in the intervention and comparison areas (intervention = 2.91 days [CI: 2.76 to 3.06]; primary = 2.88 days [CI: 2.77 to 2.99]). However, there was a significant increase in LOS over the study period in both areas (intervention LOS ∆ = 0.89 days [CI: 0.65 to 1.14]; comparison LOS ∆ = 1.1 days [CI: 0.93 to 1.28]).

FIGURE 2

Difference in difference comparisons. A, count of emergency department visits. B, count of inpatient days. C, probability of inpatient admission.

FIGURE 2

Difference in difference comparisons. A, count of emergency department visits. B, count of inpatient days. C, probability of inpatient admission.

Close modal

Near-North

DID models comparing the HNHF treatment area to the near-north neighborhood indicate similar results to the primary analysis (Table 4). There were no differences between treatment and control neighborhoods over time for any of the outcomes. Significant main effects for time across all 3 outcomes indicate a decrease in inpatient admissions (β = −.19; CI: −0.29 to −0.08), decrease in ED visits (β = −.23; CI: −0.28 to −0.18), and increase in LOS (β =.25; CI: 0.17 to 0.32) over time, whereas the main effect for the intervention indicates a difference between intervention and control neighborhoods at baseline only for ED visits (β =.09; CI: 0.04 to 0.14).

TABLE 4

Adjusted DID Regression Results for Secondary Analyses By Using Individual Comparison Neighborhoods: Near-North Model

ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF .09 0.04 to 0.14 <.001 −.01 −0.11 to 0.08 .77 −.01 −.08 to 0.06 .80 
Time −.23 −0.28 to −0.18 <.001 −.19 −0.29 to −0.08 <.001 .25 0.17 to 0.32 <.001 
HNHF*time −.01 −0.08 to 0.06 .70 .09 −0.05 to 0.23 .22 .02 −0.08 to 0.12 .73 
ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF .09 0.04 to 0.14 <.001 −.01 −0.11 to 0.08 .77 −.01 −.08 to 0.06 .80 
Time −.23 −0.28 to −0.18 <.001 −.19 −0.29 to −0.08 <.001 .25 0.17 to 0.32 <.001 
HNHF*time −.01 −0.08 to 0.06 .70 .09 −0.05 to 0.23 .22 .02 −0.08 to 0.12 .73 

All estimates include controls for child’s age, sex, and number of months they were Medicaid eligible while in the neighborhood of interest.

a

Number of ED visits per 100 Medicaid-eligible children.

b

Number of inpatient admissions per 100 Medicaid-eligible children.

c

Sum of all admitted days per total admitted Medicaid-eligible children.

Near-West

DID models comparing the HNHF treatment area to the near-west neighborhood indicate significant differences between these areas over time (Table 5). HNHF investment was used to positively impact the number of ED visits (DID = −0.07; CI: −0.14 to −0.01), probability of an inpatient admission (DID = −0.15; CI: −0.29 to −0.02), and expected LOS for those admitted (DID = −0.11; CI: −0.21 to −0.01). Predicted counts reveal the intervention neighborhood would have expected 0.03 more ED visits per member if the intervention had not taken place. Likewise, although members in the HNHF intervention had an increased LOS, they were expected to spend 0.4 more days admitted. They also would have expected a 0.01 greater probability of inpatient admission.

TABLE 5

Adjusted DID Regression Results for Secondary Analyses By Using Individual Comparison Neighborhoods: Near-West Model

ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF −.16 −0.2 to −0.11 <.001 .14 0.05 to 0.23 .003 .03 −0.04 to 0.1 .45 
Time −.17 −0.21 to −0.13 <.001 .05 −0.04 to 0.14 .26 .37 0.31 to 0.44 <.001 
HNHF*time −.07 −0.14 to −0.01 .02 −.15 −0.29 to −0.02 .02 −.11 −0.21 to −0.01 .03 
ED VisitaInpatient AdmissionbLOSc
β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)β Estimate95% CIP(>|z|)
HNHF −.16 −0.2 to −0.11 <.001 .14 0.05 to 0.23 .003 .03 −0.04 to 0.1 .45 
Time −.17 −0.21 to −0.13 <.001 .05 −0.04 to 0.14 .26 .37 0.31 to 0.44 <.001 
HNHF*time −.07 −0.14 to −0.01 .02 −.15 −0.29 to −0.02 .02 −.11 −0.21 to −0.01 .03 

All estimates include controls for child’s age, sex, and number of months they were Medicaid eligible while in the neighborhood of interest.

a

Number of ED visits per 100 Medicaid-eligible children.

b

Number of inpatient admissions per 100 Medicaid-eligible children.

c

Sum of all admitted days per total admitted Medicaid-eligible children.

Sensitivity analyses provided mixed results. In sensitivity aggregations 1 to 3, there were no significant DID estimates for ED visits and LOS use measures. This was consistent across the aggregated and disaggregated models. In sensitivity aggregations 4 and 5, findings from the original tract aggregations were confirmed. LOS was significantly decreased in the intervention neighborhood relative to the near-west neighborhood by using these aggregations. The number of ED visits also significantly decreased in the intervention neighborhood relative to the near-west neighborhood as well as the primary neighborhood. The sensitivity aggregations used to examine changes in inpatient probability were mixed. Again, the probability was decreased for the intervention neighborhood relative to the near-west neighborhood in aggregations 4 and 5. However, in sensitivity aggregations 2 to 4, the probability of an inpatient admission decreased more in the near-north neighborhood relative to the intervention neighborhood, and these differences were significant. These results imply that findings are highly dependent on neighborhood definition.

Our findings suggest that a community development initiative may influence health care use for children at the neighborhood level. However, the effect was modest, limited to ED visits, and significant in only one of our propensity score–matched communities.

Beyond revealing the potential for impact, this work reveals that midimplementation evaluative research based on observational data is feasible, even when this was not envisioned at the start of the program. Lessons learned in this approach include the following:

  1. Expecting large-scale changes in neighborhood health outcomes that were not the specific target of the intervention may be unrealistic. We hypothesized that our broad-based improvements in the residential environment would have an indirect effect on health care use, but it is highly likely that more proximate person-level health factors overwhelmed that effect. The cumulative effects of childhood toxic stress, epigenetic changes, and adverse childhood experiences are known to become evident later in adulthood.19  Our 10-year program period, which includes the early years of ramp-up, may not be sufficient to measure longer-term effects. Rather than being disheartened by the modest changes, we must instead recognize that although these findings are early and small, they serve to open the discussion of return on investment to the health care system.

  2. Measurement is hard. Identifying appropriate comparison communities required the use of geographic and statistical approaches leveraging multiple data sources, and, ultimately, our matches were still not perfect. The fact that the magnitude of the intervention effect differed when different comparator communities were used highlights the fact that the unique features that make neighborhoods “neighborhoods” are unlikely to ever be fully controlled for in a research model. As more comparative studies of place-based interventions are conducted, best practices for research and evaluation design will further crystalize.

  3. Even using strong statistical methods, this in an ecologic approach, and we cannot claim to have “caused” the observed changes, nor can we pinpoint which components of the multifaceted intervention are driving them. This is acceptable (even if it frustrates researchers accustomed to testing discrete treatments). True community-engaged, iteratively designed initiatives will never generate the strength of evidence associated with randomized trials, but, on the other hand, randomized trials do not reflect the real world. There is a place for both.

  4. Results must be considered as part of the big picture. Such work must be done with attentiveness to displacement of the poorest residents, which may make use data look better but not represent true improvements for the long-time residents of the neighborhood. This was the first question raised by stakeholders who previewed these results, and the concern regarding gentrification is real. To date, Medicaid births in the affected census tracts in our study have not declined, suggesting no major displacement of low-income families, but vigilance to this concern will continue in future phases of research.

A community development intervention emanating from a hospital-community partnership was associated with some decreases in health care use for all children in the target neighborhood compared to matched comparison neighborhoods in the first years after launching. This outcomes evaluation case study demonstrates that neighborhood-level outcomes can and should be measured, but such measurement requires multidisciplinary teams, scientific rigor, and patience.

We thank Gretchen West and Angela Mingo from the HNHF team for their thoughtful advising on this article. We also thank the Partners For Kids data team for their assistance in creating the analytic data set.

Dr Chisolm conceptualized and designed the study and drafted the initial complete manuscript; Ms Jones conducted the analyses, participated in refining the methodologic approach, and drafted Methods and Process and Outcomes sections; Dr Root advised Ms Jones on the design and conduct of the analysis and codrafted the initial methods and results sections; Dr Dolce participated in literature review and study design; Dr Kelleher co-designed and co-led the development of the intervention, participated in the study design, and contributed to the drafting of the manuscript; and all authors reviewed and revised the manuscript and approved the final manuscript as submitted.

FUNDING: No external funding.

CI

confidence interval

DID

difference-in-differences

ED

emergency department

HNHF

Healthy Neighborhoods Healthy Families

LOS

length of stay

NCH

Nationwide Children’s Hospital

PSM

propensity score model

1
Hughes
HK
,
Matsui
EC
,
Tschudy
MM
,
Pollack
CE
,
Keet
CA
.
Pediatric asthma health disparities: race, hardship, housing, and asthma in a national survey
.
Acad Pediatr
.
2017
;
17
(
2
):
127
134
2
Colton
MD
,
Laurent
JGC
,
MacNaughton
P
, et al
.
Health benefits of green public housing: associations with asthma morbidity and building-related symptoms
.
Am J Public Health
.
2015
;
105
(
12
):
2482
2489
3
Shields
W
,
McDonald
E
,
Frattaroli
S
,
Bishai
D
,
Ma
X
,
Gielen
A
.
Structural housing elements associated with home injuries in children
.
Inj Prev
.
2016
;
22
(
2
):
105
109
4
Smith
T
,
Hawke
L
,
Chaim
G
,
Henderson
J
.
Housing instability and concurrent substance use and mental health concerns: an examination of Canadian youth
.
J Can Acad Child Adolesc Psychiatry
.
2017
;
26
(
3
):
214
223
5
Minh
A
,
Muhajarine
N
,
Janus
M
,
Brownell
M
,
Guhn
M
.
A review of neighborhood effects and early child development: how, where, and for whom, do neighborhoods matter?
Health Place
.
2017
;
46
:
155
174
6
Delgado
CF
,
Ullery
MA
,
Jordan
M
,
Duclos
C
,
Rajagopalan
S
,
Scott
K
.
Lead exposure and developmental disabilities in preschool-aged children
.
J Public Health Manag Pract
.
2018
;
24
(
2
):
e10
e17
7
Beck
AF
,
Huang
B
,
Chundur
R
,
Kahn
RS
.
Housing code violation density associated with emergency department and hospital use by children with asthma
.
Health Aff (Millwood)
.
2014
;
33
(
11
):
1993
2002
8
Pastor
M
,
Morello-Frosch
R
.
Integrating public health and community development to tackle neighborhood distress and promote well-being
.
Health Aff (Millwood)
.
2014
;
33
(
11
):
1890
1896
9
Smedley
B
,
Amaro
H
.
Advancing the science and practice of place-based intervention
.
Am J Public Health
.
2016
;
106
(
2
):
197
10
Schuchter
J
,
Jutte
DP
.
A framework to extend community development measurement to health and well-being
.
Health Aff (Millwood)
.
2014
;
33
(
11
):
1930
1938
11
Jutte
DP
,
Miller
JL
,
Erickson
DJ
.
Neighborhood adversity, child health, and the role for community development
.
Pediatrics
.
2015
;
135
(
suppl 2
):
S48
S57
12
Wang
K
,
Immergluck
D
.
The geography of vacant housing and neighborhood health disparities after the US foreclosure crisis
.
Cityscape
.
2018
;
20
(
2
):
145
170
13
Kelleher
K
,
Reece
J
,
Sandel
M
.
The Healthy Neighborhood, Healthy Families initiative
.
Pediatrics
.
2018
;
142
(
3
):
e20180261
14
Ma
X
,
Fleischer
NL
,
Liu
J
,
Hardin
JW
,
Zhao
G
,
Liese
AD
.
Neighborhood deprivation and preterm birth: an application of propensity score matching
.
Ann Epidemiol
.
2015
;
25
(
2
):
120
125
15
de Vocht
F
,
Campbell
R
,
Brennan
A
,
Mooney
J
,
Angus
C
,
Hickman
M
.
Propensity score matching for selection of local areas as controls for evaluation of effects of alcohol policies in case series and quasi case-control designs
.
Public Health
.
2016
;
132
:
40
49
16
Board of Governors of the Federal Reserve System
.
Federal fair lending regulations and statutes: Fair Housing Act. 2020
. Available at: https://www.federalreserve.gov/boarddocs/supmanual/cch/fair_lend_fhact.pdf. Accessed February 17, 2020
17
US Census Bureau
.
Spatial data collection and products branch GD.
2018
. Available at: https://www.census.gov/geographies/mapping-files/time-series/geo/tiger-line-file.2018.html. Accessed December 12, 2019
18
Kahle
D
,
Wickham
H
.
ggmap: spatial visualization with ggplot2
.
R J
.
2013
;
5
(
1
):
144
161
19
Shonkoff
JP
,
Garner
AS
;
Committee on Psychosocial Aspects of Child and Family Health
;
Committee on Early Childhood, Adoption, and Dependent Care
;
Section on Developmental and Behavioral Pediatrics
.
The lifelong effects of early childhood adversity and toxic stress
.
Pediatrics
.
2012
;
129
(
1
). Available at: www.pediatrics.org/cgi/content/full/113/2/e232

Competing Interests

POTENTIAL CONFLICT OF INTEREST: Drs Chisolm, Dolce, and Kelleher are employed by Nationwide Children’s Hospital, which is the lead funder for the Healthy Neighborhoods Healthy Families initiative studied; Ms Jones and Dr Root have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.

Supplementary data