Primary Substance Use Prevention Programs for Children and Youth: A Systematic Review

Eight academic databases were searched in March 2020, including Medline (1946 to March 2020), PsycINFO (1806 to March 2020), Education Resources Information Center (1965 to March 2020), Academic Search Complete (1965 to March 2020), SocINDEX (1975 to March 2020), ProQuest (1970 to March 2020), Web of Science (1975 to March 2020), and PubMed (1975 to March 2020).

A systematic search process was conducted in keeping with Cochrane Collaboration methods.¹⁸  A research librarian designed a search strategy to identify studies relevant to the objectives. Databases were searched by using terms specific to preventive services, addictive substances, evidence-based practice, and school-aged children and youth. For example, by using the Medline database, keywords related to the topics included school health services, health education, preventive health services, health promotion, substance-related disorders, alcohol-related disorders, amphetamine, cocaine, marijuana, opioid, phencyclidine, alcohol, psychotropic drug, tobacco, evidence-based practice, treatment outcome, program evaluation, program, development, and outcome assessments. Results were limited to studies that included children and youth aged 5 to 18 years old.

Two reviewers independently screened the abstracts of articles published in English according to 3 criteria: (1) the article was focused on evaluation of the effectiveness of a substance use prevention program, (2) the program was delivered to school-aged children and youth (defined as kindergarten to grade 12), and (3) the program was aimed at preventing the use of psychoactive substances. Programs solely aimed at nicotine prevention were excluded because we focused on programs aimed at preventing the use of substances with significant mind-altering effects (ie, alcohol and drugs) in this review. The Cohen κ coefficient for this initial screening stage was 0.86 (P < .05). Discrepancies were resolved through discussion.

Two reviewers independently screened the full text of articles meeting the above 3 criteria according to 4 additional criteria: (1) the program was designed for a general population of children and youth (ie, the program was not specifically designed for particular target groups, such as minority children, low-income children, children of parents with addictions, or clinical populations), (2) the program was delivered to a general population of children and youth, (3) the program only targeted children and youth (ie, did not also include components targeting systems surrounding children, such as parents or teachers), and (4) the study included a control group. Finally, given that we aimed to identify programs with robust evidence of effectiveness, we excluded programs for which only one study had been conducted (κ = 0.92).

All articles were grouped by program and assigned a study number. Using a standard form, trained researchers extracted data from each article regarding study design and participant characteristics, including the number of participants, ethnicity, and grade. Intervention characteristics extracted included the program name, substances that the prevention program targeted, facilitator type (teachers, other professionals, or peers), program content, and intervention dosage. Information was also extracted relevant to the type of results measured as well as the time point at which outcomes were measured. Program characteristics are summarized in Table 1. Study characteristics are organized by program and summarized in Table 2.

The 27-item Downs and Black¹⁰⁸  checklist for measuring the quality of randomized and nonrandomized studies of health care interventions was selected to assess methodologic quality. The checklist is used to examine 5 dimensions of study quality, including (1) reporting index, (2) external validity index, (3) bias index, (4) confounding index, and (5) power index. In a systematic review of instruments for assessing the methodologic quality of nonrandomized studies of interventions, Deeks et al¹⁰⁹  recommended this checklist and its 5 separate indices. The interrater reliability of the checklist is good (r = 0.75; P = .56).¹⁰⁸  Its usefulness has also been acknowledged by the Cochrane Collaboration.¹⁸ 

Scoring for 2 items on the checklist was revised to better distinguish article quality. Item 18 addresses whether the statistical tests used to assess the main outcomes were appropriate. The original measure scored studies as either 1, indicating that the main outcomes were appropriate, or 0, indicating either that the main outcomes were not appropriate or that it was not possible to determine if the main outcomes were appropriate. Our revision of the measure included the scoring of studies as 0, indicating that all statistical analyses were inappropriate; 1, indicating that at least 1 analysis was inappropriate; and 2, indicating that all analyses were appropriate. Item 27 examines whether the study had sufficient power to detect a clinically important effect, in which the P value for a difference being due to chance is <5%. The original measure scored studies on a scale between 0 and 5 according to an available range of study powers. Our revision of the measure included the scoring of studies on this item as either 0, indicating that no power calculation was reported in the article; 1, indicating that power calculations were reported, that ≥1 of the analyses demonstrated sufficient power, and that ≥1 of the analyses did not demonstrate sufficient power; or 2, indicating that power calculations were reported and that all analyses demonstrated sufficient power. The final checklist consisted of 27 items and had a maximum score of 30 points (a higher score indicates higher quality). At least 2 reviewers independently rated each article according to the checklist. Discrepancies were resolved by consensus.

In the initial search, 10 395 articles were identified (Fig 1). After removal of duplicates, 7875 articles were screened by title and abstract. From these, 676 studies were selected for full-text review, which resulted in the inclusion of 163 articles representing 87 unique programs. The final step involved excluding programs for which only one study had been conducted, given that we aimed to identify programs with the most robust evidence for their effectiveness. This resulted in a final set of 90 studies representing 16 unique programs. Although authors of some of the articles reported findings based on data from participants belonging to the same baseline sample (eg, follow-up studies at different time points), ratings on the Downs and Black¹⁰⁸  checklist are derived from the unique details provided in each article. Consequently, index scores on the Downs and Black¹⁰⁸  checklist for these articles varied, and they were treated as separate studies in the current review. Given the heterogeneity of outcomes measured in the included studies, it was not possible to conduct a statistical meta-analysis of program effectiveness. Therefore, a narrative synthesis of findings is provided.

The 90 included studies revealed wide variability in quality, as reflected by index scores on the Downs and Black¹⁰⁸  checklist. Total index scores ranged from 11 to 23 out of a possible 30. The mean score among articles selected for this review was 17.7 (SD = 2.7).

The reporting index includes 10 items and has a maximum possible score of 11. This index is used to evaluate whether information provided in an article is sufficient for making an unbiased assessment of findings. Articles selected for this review scored between 5 and 9 on the reporting index, with an average score of 7.5 (SD = 1.07). A notable area of deficit was reporting of adverse events. This may have been a consequence of the intervention because there were no articles that met this criterion. There were also no articles in which a full description of principal confounders was provided; a partial description was provided in 54 articles. Other areas in which articles scored lower were in reporting actual probability values (only 49 articles met this criterion) and describing the characteristics of participants lost to follow-up (only 64 articles met this criterion).

The external validity index includes 3 items and has a maximum possible score of 3. Scores on this index reflect the extent to which study findings can be generalized to the population from which participants were derived. Articles scored between 0 and 3 on the external validity index, with an average score of 1.2 (SD = 0.61). All but 6 articles scored a 0 (ie, a “no”) in response to the question of whether the subjects who were asked to participate in the study were representative of the entire population from which they were recruited. In only 17 articles, subjects who were prepared to participate were representative of the entire population from which they were recruited.

The bias index includes 7 items and has a maximum possible score of 8. Scores on this index reflect the extent to which studies include biases in the measurement of interventions and outcomes. Articles scored between 4 and 7 on the bias index, with an average score of 5.6 (SD = 0.70). Two items lowered scores on this index. In particular, all articles received a score of 0 (ie, a response of “no”) with respect to whether an attempt was made to blind those measuring the main outcomes of the intervention, and all but 4 articles received a score of 0 with respect to whether an attempt was made to blind study participants to the intervention they received.

The internal validity index includes 6 items and has a maximum possible score of 6. Scores on this index are used to examine bias in the selection of participants. Articles scored between 1 and 6 on the internal validity index, with an average score of 3.31 (SD = 1.35). On this index, all but 4 articles scored a 0 (ie, “no”) in response to the question of whether the randomized intervention assignment was concealed from both participants and staff until recruitment was complete and irrevocable. There was wide variability in scores on the other internal validity index items. Of note, random assignment to intervention groups was described in 63 articles, which, in most cases, involved random assignment at the school or classroom level rather than random assignment at the individual student level. Also of interest, loss of students to follow-up was not taken into account in 25 articles, which impacted scores on the internal validity index.

The power index includes 1 item and has a maximum possible score of 2. Only 6 articles scored a 2 on this index, indicating that, for these 6 articles, power calculations were reported and all analyses demonstrated sufficient power. Finally, 84 articles were allotted a score of 0 on this index, indicating that no power calculation was reported.

With respect to the 16 programs selected for inclusion in this review, program names, substances targeted, and program content are described in Table 1. Most programs were developed in the United States (n = 11) and approximately one-third within the last 20 years (n = 6). In addition, the majority of programs (n = 10) targeted the prevention of all psychoactive substances (ie, alcohol, drugs, and nicotine). All 16 programs included content aimed at providing both information and skills. The 90 studies conducted on these programs are described in Table 2. Sixty-seven percent of studies selected for this review (n = 60) were conducted in the United States. Articles varied widely with respect to outcomes measured. As depicted in Table 2, authors of most studies reported on substance use outcomes, although many also measured outcomes related to substance use attitudes and beliefs. Overall, the studies that met criteria for this systematic review varied widely with respect to intervention dosage, participant characteristics, results, and ratings of study quality (Table 2). In Fig 2, we present the average index scores by program for overall study quality and each of the 5 dimensions of quality.

The programs evaluated with the largest combined sample sizes, totaled across all studies of the programs, were the Drug Abuse Resistance Education (DARE) program (N = 24 200), Project Adolescent Learning Experiences Resistance Training (ALERT) (N = 19 717), the Life Skills Training (LST) Program (N = 18 800), the Adolescent Alcohol Prevention Trial (N = 15 022), and Project Choice (N = 9856). Statistically significant findings for these 5 programs pertaining to (1) the use of psychoactive substances with significant mind-altering effects; (2) attitudes and beliefs about the use of these substances; and (3) other outcomes, including nicotine use, are summarized below.

Across the 14 studies of the DARE program included in this review,^36–49  1 study (7.1%) demonstrated reductions in the use of substances with significant mind-altering effects (specifically, overall substance use and alcohol use) at 6 months post program.³⁶  Effects on attitudes and beliefs were found in 2 studies (14.3%), with effects on recognizing media portrayal of beer drinking as desirable observed at the 1-year follow-up in 1 study⁴⁰  and positive effects in 4 learning outcomes (getting help from others, communication and listening skills, substance abuse knowledge, and making safe and responsible choices) in the other.⁴⁹  DARE studies varied in quality, scoring between 14 and 20 on the Downs and Black¹⁰⁸  checklist, with an average score of 16.9.

Across the 12 studies included in this review in which Project ALERT was evaluated,^75–85,110  2 studies (16.7%) demonstrated reductions in the use of substances with significant mind-altering effects (cannabis initiation and alcohol abuse at 18 months post program⁷⁹  and cannabis use at 2 years post program).⁸²  Two studies (16.7%) revealed positive effects on attitudes and beliefs about substance use at follow-up periods of 15⁷⁶  and 18 months.⁸⁰  Among the 3 studies (25%) that revealed significant effects on nicotine use, results were mixed. Two of these studies reported reductions in nicotine use, with effects observed at 15⁷⁷  and 18 months⁷⁹  post program. However, the third study revealed that the direction of effects at the 15-month follow-up varied depending on students’ baseline levels of nicotine use, with nicotine use increasing for users and decreasing for experimenters.⁷⁵  Project ALERT studies varied in quality, scoring between 13 and 21 out of a possible 30 on the Downs and Black¹⁰⁸  checklist, with an average score of 17.1.

Across the 17 studies conducted on the LST program,^56–72  10 studies (58.8%) reported reductions in the use of substances with significant mind-altering effects for follow-up periods ranging from time of program completion to 14.5 years post program.^{56,58–60,62,65–69}  Reductions in use were demonstrated for a range of substances, including alcohol, cannabis, and other drugs (eg, heroin, hallucinogens, methamphetamines, prescription drugs); in 7 of these 10 studies (70%), significant effects were found for multiple substances.^{59,60,62,65,67–69}  Effects on substance use attitudes and beliefs were evidenced in 9 of the 17 studies (52.9%) of the LST program and included increased antialcohol and antidrug attitudes, increased knowledge, and improvement in normative beliefs at time periods ranging from program completion to 2 years post program.^{56–58,61,62,64,65,70,71}  Finally, 9 studies (52.9%) detected reductions in nicotine use (ranging from program completion to 14.5 years post program),^{56,61–63,65,67–69,72}  and 4 studies (23.5%) revealed that LST contributed to improved interpersonal skills, self-esteem, assertiveness, anxiety management, and reduced anxiety (ranging from program completion to 2 years).^56,61,64,70  LST studies were of varying quality, scoring between 13 and 22 on the Downs and Black¹⁰⁸  checklist, with an average score of 17.9.

All 3 studies of the Adolescent Alcohol Prevention Trial revealed effects on the use of substances with significant mind-altering effects, specifically reduced alcohol use and delayed onset of alcohol use.^19–21  Effects were detected at follow-up 2 and 5 years after program completion. In addition, 2 studies (66.7%) demonstrated reductions in nicotine use at 5 years post program.^20,21  Studies of the Adolescent Alcohol Prevention Trial varied in quality, ranging from scores of 11 to 19 on the Downs and Black¹⁰⁸  checklist, with an average score of 15.3.

The 2 studies evaluating Project Choice revealed reductions in alcohol use, with effects observed at program completion and at 6 to 7 months post program.^86,87  One study also revealed effects on substance use attitudes and beliefs, which was that after program completion, participants reported lower perceptions of friends’ cannabis use.⁸⁶  Study quality was similar, with scores of 17 and 18 on the Downs and Black¹⁰⁸  checklist.

With the present review, we report on the methodologic quality of evidence for substance abuse prevention programs delivered to school-aged children and youth as well as the effectiveness of identified programs. Ninety studies representing 16 unique programs were identified. Given the heterogeneity of outcomes measured in the included studies, it was not possible to conduct a statistical meta-analysis of program effectiveness. Therefore, a narrative synthesis of findings has been provided.

Studies included in this review varied widely with respect to quality. Many studies demonstrated relatively low quality, as measured by the Downs and Black¹⁰⁸  checklist, and only studies on 1 program, Project Choice, demonstrated consistently high total index scores (although only 2 studies of Project Choice were included in this review). Importantly, there were some items on the checklist for which most or all articles scored a 0. The checklist may, therefore, be less applicable to studies of prevention programs. For example, 1 item on the reporting index of the checklist is used to examine whether all adverse events that may have been a consequence of the intervention were reported. There were no studies that reported all possible adverse events, and therefore all articles included in this review lost a point for this item. As another example, a checklist item on the internal validity index is used to examine whether randomized intervention assignment was concealed from both participants and staff until recruitment was complete and irrevocable. All but 4 studies lost points for this item. The reporting of adverse events and concealment of randomized intervention assignment may be more applicable to studies focused on interventions conducted in controlled settings (eg, medication trials) rather than studies of prevention programs in more complex community settings. However, this checklist was deemed most appropriate for this review given its applicability to nonrandomized interventions.

Considering scores on other quality checklist items provides insight into areas for methodologic improvement. In particular, researchers can more intentionally report on potential confounding variables, how analyses were appropriately adjusted to account for confounding, the characteristics of participants lost to follow-up, and actual probability values. In addition, although most studies used statistical tests that were appropriate for assessing the main outcomes, 22 studies used tests that were not appropriate, representing a critical area of importance in data analysis.

An additional consideration regarding study quality is that most prevention programs for school-aged children and youth are delivered in the classroom setting, which precludes random assignment at the individual student level. Therefore, in studies of program effectiveness, random assignment is generally conducted at the classroom or school level. According to St Pierre et al,⁸¹  it is conceivable that within-school random assignment can cause contamination of a control group because treatment effects may spill over from students in the intervention group. In studies in which within-school random assignment is used, methodologic rigor and a critical analysis of study findings are essential.

The results of this review indicate that the most research has been conducted on the LST program. Ten studies on the LST program demonstrated reductions in the use of substances with significant mind-altering effects, including both alcohol and drugs.^{56,58–60,62,65–69}  Of those 10 studies, reductions in use for >1 type of substance were reported in 70%.^{59,60,62,65,67–69}  With the proliferation of substance abuse prevention programs and accompanying research, it is important that isolated program effects do not advance an intervention to “evidence-based” status.¹¹¹  Multiple studies revealed positive outcomes for the LST program across several domains (including use of psychoactive substances with significant mind-altering effects, substance use attitudes and beliefs, nicotine use, and social and emotional outcomes), suggesting that LST has robust evidence to support its effectiveness, particularly in comparison with other programs in which only a small proportion of studies revealed positive effects over a limited range of outcomes.

Importantly, however, as with most other programs included in this review, studies of LST effectiveness varied in quality, tempering the conclusions drawn. Furthermore, some studies reporting positive effects of LST (again, as with other programs reviewed here) date back nearly 30 years, before the advent of the Internet, social media, digital technologies; the emergence of new synthetic drugs, such as methamphetamines, public health crises surrounding opioid abuse; and the decriminalization of cannabis in Canada. Therefore, caution should be exercised in generalizing program effects found in previous cohorts to today’s children and youth.

In addition to identifying programs with evidence of effectiveness, it is important to attend to widely implemented programs that lack evidence of effectiveness. DARE had no statistically significant impact on psychoactive substance use among youth across 92% of the 14 studies included in this review, despite DARE being described as the most widely implemented substance abuse prevention program in the world.¹¹²  McLennan¹¹³  makes the case that schools continue to implement DARE because of sunk costs and emotional investment in the program. The continued implementation of any program without strong evidence is problematic given the potential for ineffective resource use and missed prevention opportunities.

Given the substantial individual and societal costs of youth substance abuse as a public health problem, it is incumbent on pediatricians and other health care providers to understand the levels of evidence behind substance abuse prevention programs. To this end, with the current review, we provide an updated summary of evidence for universal prevention program effectiveness. In this review, we highlighted that the most research has been conducted on the LST program. As with most other programs included in this review, however, studies of LST effectiveness varied in quality. In the current article, we reviewed programs that only targeted children and youth in an effort to (1) restrict the scope of the review and (2) provide information for practitioners seeking to implement programs that target children and youth. However, it is important to acknowledge that multipronged prevention approaches involving entire schools, families, and communities may be ideal. Nonetheless, with this review, we add to the literature by providing an updated summary of evidence for primary prevention program effectiveness.

ALERT

Adolescent Learning Experiences Resistance Training

DARE

Drug Abuse Resistance Education

LST

Life Skills Training