A Summer Nutrition Benefit Pilot Program and Low-income Children’s Food Security

To provide the most credible and rigorous estimates of the impact of the demonstrations, we used a random assignment design for the evaluation’s impact analysis (registered at ClinicalTrials.gov: NCT02877147). In the spring of each year, participating School Food Authorities at each site constructed lists of households with eligible children (ie, certified for free or reduced-price meals) and obtained household consent. Using those lists, the evaluation team randomly assigned eligible households (using the SAS SURVEYSELECT procedure [SAS Institute, Inc, Cary, NC]). In 2011 and 2013, almost all households randomly assigned were selected for the evaluation sample. In 2012, in most of the participating sites, more households were assigned SEBTC benefits than required for the evaluation. Consequently, a random sample was selected for the evaluation. Sites and households had no involvement in randomization procedures. Blinding was not possible.

In 2011 and 2012, households were randomly assigned to either a $60-per-eligible-child monthly benefit group or a no-benefit group. In 2013, households were randomly assigned to either a $60- or a $30-per-eligible-child monthly benefit group. Benefits were issued for the summer period when school was not in session and prorated for partial months.

The sample size needed for the evaluation subsample in 2012 (n = 27 000) was calculated before the study began, on the basis of the size needed to estimate an impact of 5 percentage points on the study’s confirmatory outcome at a 95% level of confidence with 80% power. Given the 2012 results, the evaluation sample size in 2013 (n = 18 000) was calculated to detect a difference between the 2 benefit amounts, assuming a constant per-dollar impact.

Table 2 shows the overall allocation of participants and analysis sample sizes for the food security samples pooled across years. (The Supplemental Information depicts both the food security and nutrition samples for each year.) Exact analysis sample sizes vary by analytic outcome (see the results tables), but sample sizes are large; for most analyses, sizes are well over 30 000.

There were a few known crossovers in 2012 and 2013 (for example <50 households in 2013); which occurred when a grantee mistakenly issued summer benefits to families in the incorrect group. Grantees received no reports of benefit sharing.

The study design was reviewed and approved by the institutional review board of Abt Associates. The parts of the study taking place in Washington and Michigan were reviewed and approved by the Washington State Institutional Review Board and the Michigan Department of Community Health Institutional Review Board, respectively. The full study protocol is available on the USDA Food and Nutrition Service Web site.^9,–12 

The evaluation’s data collection team surveyed households in the spring (before the school year ended) and again in the summer. The spring and summer surveys were administered by computer-assisted telephone interview. Households not reached by telephone were assigned to field staff for locating; respondents who were located and agreed to answer the survey were connected to the phone center.

Given the importance of food security for children’s health and development, the primary and confirmatory outcome selected by the USDA for the evaluation was the prevalence of very low food security among children (VLFS-C), the most severe form of food insecurity (see Table 3). VLFS-C occurs when the food intake of any child in the household is reduced and normal eating patterns are disrupted because the household lacks money and other resources for food.

To measure this outcome, the USDA’s 18-item US Household Food Security Survey Module with a 30-day reference period was used in the evaluation. The measure reflects the food security status of adults and children in the household. It has been used widely since the 1990s to assess and monitor food security in large-scale population studies and is included as an annual supplement to the Current Population Survey.^13,14 

Among other exploratory items, we also measured the prevalence of food insecurity among children (FI-C). This occurs when any child in the household experiences either VLFS-C or the less severe low food security among children. We also measured food consumption for a focal child developed by the National Cancer Institute for the 2009–2010 NHANES Multifactor Diet Screener^15,16 to assess the intake of specific dietary factors associated with the 2010 Dietary Guidelines for Americans. One SEBTC-eligible child per household was randomly selected to be the focus of the food consumption questions, with the household’s respondent reporting that child’s consumption of specific items in the last 30 days. Scoring procedures developed by the National Cancer Institute were used to convert the reports into estimated amounts of specific foods and nutrients.¹⁷ 

Building on the random assignment design, impact estimates were computed from weighted linear regressions estimated on pooled 2011–2013 data, by using an intention-to-treat analysis. These estimates allow the determination of whether outcomes differed monthly by per-child benefit level ($0 versus $60 and $30 versus $60).

Analysis weights, developed for each site and each year, were adjusted for the sample design and differential survey nonresponse. Across years and sites, the weights were rescaled such that the sum of the weights equaled the number of completed interviews in the specific year in a specific site.

Simple comparisons of treatment and control group means would be unbiased and consistent; regression-based estimates in general are more precise. For both continuous outcomes and binary outcomes, estimates were computed by using weighted least squares. SEs were robust, accounting both for non-normal residuals induced by the analysis of binary outcomes and the complex nested sample structure (children within households within school food authorities within sites). The number of households in each treatment group used in each analysis is shown in the relevant figures and tables. All results reported in the article were considered statistically significant if they met the P < .05 threshold.

Table 4 provides information on household characteristics at baseline (when participants entered the study), pooled across the 3 years. (Supplemental Tables 6 through 11 report equivalent information for individual years.) Generally, participating households reported lower incomes than households nationally with children receiving free or reduced-price meals in 2013.⁹ 

Across 2011, 2012, and 2013, 90% of the households that were issued a $60 monthly per-child SEBTC benefit amount participated (ie, used those benefits at least once) (Supplemental Tables 12 and 13). For all households assigned benefits, the mean percentage of benefits redeemed was 77%; among households that used any benefits, the redemption rate was 86%. Usage rates were slightly lower for the $30 group, compared with the $60 group; participation rates were 1.3% points lower and redemption was 2.0% points lower (P < .001).

In both the $30 and $60 groups, households issued the WIC-model benefit redeemed less than those issued the SNAP-model benefit. For instance, including those who redeemed no benefit, among the $60 group, households in the WIC-model sites redeemed 61% of the summer benefit compared with the 93% redemption rate among households in in SNAP-model sites. This pattern is consistent with the general pattern of lower redemptions of regular WIC benefits compared with SNAP benefits, which may be due to limitation of the foods that can be redeemed under WIC compared with the allowable SNAP foods.^18,–21 

Relative to no benefit, a $60 SEBTC monthly per-child benefit decreased the prevalence of VLFS-C during the summer by one-third (P < .01). Without SEBTC, 9.1% of households in the evaluation experienced VLFS-C; in contrast, with a $60 SEBTC benefit, 6.1% of households experienced VLFS-C (Fig 1).

SEBTC also reduced the prevalence of FI-C by nearly one-fifth (P < .01; Fig 1). Without SEBTC, 43.0% of households reported children with food insecurity; with a $60-per-child monthly benefit, the rate dropped to 34.7% of households.¹¹ 

These estimated impacts are roughly consistent with Mabli and Worthington’s²² nonexperimental estimates for a much larger change in food assistance. When comparing the differences between households that received no SNAP benefits and those that received full SNAP benefits, the authors estimated that SNAP decreased FI-C by 38%; however, the corresponding estimate for VLFS-C was not statistically different from 0. Corresponding analysis of household food security failed to detect an impact.^22,23 

These results are also qualitatively similar to analyses of the American Recovery and Reinvestment–induced changes in SNAP benefits, in which it was concluded that moderate changes in SNAP benefits have impacts on household food security.^24,–26 For example, Nord²⁵ estimated that the American Recovery and Reinvestment–induced 7% decline in SNAP benefits increased very low food security among households by 16.5%. Katare and Kim²⁶ estimated an increase in very low food security among households of 14% and an increase in food insecurity of 7%.

Although there were differences in benefit-redemption rates between households in WIC- and SNAP-model sites, the study did not find statistically significant differences in children’s food security outcomes by program model (P > .05).

Results were mixed as to whether a $60 monthly benefit had larger impacts than a $30 benefit. The $60 benefit reduced VLFS-C by 0.6 percentage points compared with the $30 benefit (P = .076), but the difference did not meet the study’s threshold for statistical significance (P < .05). (See Fig 2.)

On the basis of guardians’ reports of a randomly selected child’s average daily consumption of foods associated with health (fruits and vegetables, whole grains, dairy foods, and foods and beverages with added sugars²⁷), SEBTC improved children’s nutrition.

Compared with no benefit, the $60 monthly per-child SEBTC benefit improved 6 of the 8 nutrition outcomes measured by the study authors (Table 5). For instance, the $60 benefit increased children’s mean daily consumption of fruits and vegetables by one-third of a cup (0.36 cup Eq; a 13% difference). This is equivalent to the impact of the school-based Fresh Fruits and Vegetable program.²⁸ It also increased their mean consumption of whole grains by a 2.2 oz Eq per day (a 30% difference), which is roughly equivalent to half of a slice of whole wheat bread or one-fourth of a cup of cooked brown rice (see Briefel et al²⁹ for further information of SEBTC and children’s nutrition). Although these changes are moving in the right direction, toward a higher consumption of nutritious foods, children need to make significantly greater changes or improvements to meet the dietary guidelines.³⁰ 

Although there were no differences in children’s food security impacts according to the type of model employed by the site (WIC or SNAP), we found differences in child nutrition outcomes by model. For instance, the impacts on children’s average daily fruit and vegetable consumption (excluding fried potatoes) was approximately twice as large in WIC model sites, and daily whole grain consumption was almost 4 times as large. For daily consumption of total added sugars from all foods and beverages, we found no impact in SNAP-model sites.^29,31,32 Although there was no reduction in these added sugars among children in households in SNAP-model sites, the benefit did not increase purchases of sugar-sweetened beverages, even though these types of beverages can be purchased with SNAP.

Low participation in standard federal summer meals programs raises concern about the food security and food consumption of low-income children during the summer, when many lack access to free or reduced-price school meals. There are few other large random assignment studies of the impact of food assistance policy on children, and there are even fewer on the impact of such policies in the summer. In this rigorous evaluation, we found that SEBTC improved school-aged children’s food security and the nutrition of children. A 33% reduction in summer rates of VLFS-C is substantial and significant.

Given that the sites were not randomly selected to be representative, extrapolating from the results in the demonstration sites during the summer to the nation as a whole during the school year involves a set of methodological challenges and assumptions. However, partially in response to the evaluation’s consistent and promising results, the Obama Administration’s budget proposal for 2017 included a $12 billion nationwide expansion of SEBTC over 10 years.³³ In addition, the Food and Nutrition Service has continued and expanded the SEBTC Demonstration during the summers of 2014–2017 and plans to extend the pilot program to the summer of 2018.³⁴ 

As noted, consistent with levels of the main SNAP and WIC programs, the redemption levels and therefore benefit costs for the new benefit were lower in sites using the WIC model than in the SNAP model. Although there were no statistically significant differences in food security impacts, nutrition impacts were several times larger in WIC-model sites than in SNAP-model sites. Difference in impacts between these types of sites should be interpreted with care because the number of sites is relatively small and the type of model implemented was not randomly assigned. See Gordon et al,³¹ Klerman et al,³² and Briefel et al²⁹ for a more complete discussion.

More broadly, these are the first experimental results to reveal that additional food assistance improves children’s food security and nutrition. The authors of other studies have addressed the issue, but they used quasi-experimental approaches.^{22,24,25,35,–40} SEBTC’s sample sizes are large and the evaluation’s estimates are precise. Our findings provide strong evidence for the premise that food assistance can improve food security and nutrition among low-income households with children.

The results reported in this article have excellent internal validity. They are based on a well-executed random assignment design, a large sample, and a household survey with a high response rate. Food security was estimated by using the standard USDA question battery and coding scheme with demonstrated measurement validity and reliability. Similarly, children’s nutrition estimates relied on the standardized National Cancer Institute dietary questionnaire and coding scheme used in NHANES and other large-scale studies to estimate daily consumption of foods and food groups.

However, as is common with this type of research design, the evaluation was conducted in purposively selected sites, during a specific time period (summers of 2011, 2012, and 2013). Grantees proposed demonstration sites on the basis of perceived need and the capacity of local school food authorities to help with evaluation tasks. Indeed, households in the sample were more disadvantaged than average households with children who received free or reduced-price meals nationally. Extrapolation of the SEBTC findings to other locations and times of the year should be done with caution.

Despite the limitations necessitated by the design, the findings are striking. The SEBTC Demonstrations add important new information about the impact of additional food assistance on children’s food security in the summer. With rigorous random assignment-based evidence across 3 summers from nearly 50 000 households in 16 localities, the SEBTC evaluation reveals that additional food assistance ($60 or even $30 per school-aged child per month) would lead to substantial progress in addressing the problem of summer food insecurity among school-aged children and their households.

In future rigorous evaluations, researchers should explore ways to extrapolate these findings to other populations, other times of the year, and other localities. Research focused on providing additional food assistance for longer periods of time would provide additional guidance on national food assistance policy that aims to end childhood hunger.

FI-C

food insecurity among children

SEBTC

Summer Electronic Benefits Transfer for Children

SNAP

Supplemental Nutrition Assistance Program

USDA

US Department of Agriculture

VLFS-C

very low food security among children

WIC

Special Supplemental Nutrition Program for Women, Infants, and Children

This article is a summary of key findings from the Evaluation of the SEBTC. Many people made this evaluation possible, including research and data collection staff from Abt Associates and Mathematica Policy Research, project officers and other staff from the USDA, and state and local grantees who implemented the demonstrations. In particular, we acknowledge the evaluation’s project officers at the USDA, Hoke Wilson and Chanchalat Chanhatasilpa. We also acknowledge Ayesha Enver from Abt Associates for her work on the electronic benefits transfer analysis and Philip Gleason and Chris Hamilton for their technical reviews of the evaluation over the course of the study. The views and opinions expressed by the authors do not necessarily reflect those of the USDA, Abt Associates, Mathematica Policy Research, or those specifically acknowledged.