CONTEXT:

Although adolescent mental health interventions are widely implemented, little consensus exists about elements comprising successful models.

OBJECTIVE:

We aimed to identify effective program components of interventions to promote mental health and prevent mental disorders and risk behaviors during adolescence and to match these components across these key health outcomes to inform future multicomponent intervention development.

DATA SOURCES:

A total of 14 600 records were identified, and 158 studies were included.

STUDY SELECTION:

Studies included universally delivered psychosocial interventions administered to adolescents ages 10 to 19. We included studies published between 2000 and 2018, using PubMed, Medline, PsycINFO, Scopus, Embase, and Applied Social Sciences Index Abstracts databases. We included randomized controlled, cluster randomized controlled, factorial, and crossover trials. Outcomes included positive mental health, depressive and anxious symptomatology, violence perpetration and bullying, and alcohol and other substance use.

DATA EXTRACTION:

Data were extracted by 3 researchers who identified core components and relevant outcomes. Interventions were separated by modality; data were analyzed by using a robust variance estimation meta-analysis model, and we estimated a series of single-predictor meta-regression models using random effects.

RESULTS:

Universally delivered interventions can improve adolescent mental health and reduce risk behavior. Of 7 components with consistent signals of effectiveness, 3 had significant effects over multiple outcomes (interpersonal skills, emotional regulation, and alcohol and drug education).

LIMITATIONS:

Most included studies were from high-income settings, limiting the applicability of these findings to low- and middle-income countries. Our sample included only trials.

CONCLUSIONS:

Three program components emerged as consistently effective across different outcomes, providing a basis for developing future multioutcome intervention programs.

Globally, many adolescents live in environments in which poverty, conflict, or abuse is common, placing them at risk for developing mental disorders1 or engaging in co-occurring risky behaviors such as substance use and physical violence.2 These behaviors have implications for adolescent health and development and contribute to the disease burden in this age group.3 Adolescence is also a time when chronic mental disorders may develop,4 which can place adolescents at further risk for unhealthy behaviors, injuries, and diseases and contribute to poor physical and mental health in later years.5 Young people suffering from mental health problems have more difficulty forming interpersonal relationships, performing in school, and contributing productively in work environments.1 

However, adolescence is also a time of rapid physical, social, and psychological development, and as a result, it offers multiple opportunities for health promotion and disease prevention.6 Authors of previous systematic reviews on interventions to promote mental health and prevent mental disorders and risk behaviors during adolescence have concluded that psychosocial interventions can be effective in improving youth mental health.7,8 These interventions can provide foundational skills for the promotion of healthy behaviors and prevention of risk behaviors, such as violence (including bullying), tobacco use, and alcohol and substance abuse, through further generalizing behavior change improvements to other domains.9 Authors of past reviews have tended to focus on single-issue interventions and outcomes only, such as delaying alcohol use or preventing depression.10,12 In real-life settings, single-issue interventions are more likely to be “crowded out” by other new programs when funding or policy priorities shift; this approach also ignores the fact that risk and protective factors for health and development often overlap.9,13 

The process of synthesizing evidence for programming purposes should thus be reframed; rather than devoting time to developing single-issue interventions, more attention should be paid to identifying common features of proven interventions for use across multiple outcome areas. The use of key component profiles has been used in process evaluation and best practices research, including in mental health case management.14 This strategy improves cost-effectiveness, expands an intervention’s reach and sustainability, and may also cull ineffective or harmful components. It is also of particular interest for low-resource settings in which multi-outcome interventions may be more attractive to policymakers because of their potential to have a broad effect for the cost of a single program.15 

“Helping Adolescents Thrive” is a World Health Organization and United Nations Children’s Fund initiative used to develop a package of evidence-based psychological interventions to promote adolescent mental health and prevent mental disorders and risk behaviors among adolescents. As a part of this project, we conducted a systematic review, meta-analysis, and program components analysis of universally delivered interventions that sought these aims. Our purpose of this review was to inform the development of the intervention package. Specifically, we wanted to identify content-related features of programs (known as program or practice components) that consistently predict larger effect sizes in these programs across a range of outcomes.

A protocol for this systematic review was agreed with the World Health Organization as the version of record (see Supplemental Information). We present findings relating to universal interventions only in this article (programs that are targeted at the whole adolescent population and are designed to benefit everyone, not only specific at-risk groups).

We included (1) randomized controlled trials (RCTs) of psychosocial interventions (2) with adolescent participants between the ages of 10 to 19 (3) in which trial interventions had the primary or secondary aims of promoting mental health or preventing mental disorders, reducing risk behaviors, or reducing self-harm and suicide; additionally, (4) the programs were aimed at the whole adolescent population and were designed to benefit everyone regardless of setting or delivery and (5) published between January 2000 and February 2018 in any language. Studies in which authors compared outcomes between groups who received an intervention and those who received usual or no care and/or those who received a different intervention were included. We included studies if the mean age was between 10 and 19 years or >50% of the participants were between 10 and 19 years old. Outcomes included positive mental health (mental well-being, resilience, coping, emotional regulation), depressive and anxious symptomatology, violence perpetration and bullying, and alcohol and other substance use. We included different time points and coded outcomes according to short (<2 months after intervention completion), medium (2–6 months), and long-term (>6 months).

We searched Medline, PsycINFO, Scopus, Embase, and Applied Social Sciences Index Abstracts and followed references of reviews. Duplicate abstracts were removed, and the remaining abstracts were assessed against inclusion criteria by 2 independent reviewers. Any disagreements were resolved by discussion between the 2 reviewers or resolved by the arbitration of a third reviewer. Subsequently, full-text reports were accessed and assessed. Pairs of reviewers working independently completed this screening process. Data were extracted by using a standardized form and included trial characteristics, setting, sampling, population characteristics, intervention details, outcome measures, study quality (assessed by using the Cochrane risk-of-bias tool), and treatment effects. In addition, each intervention was coded according to the presence of specific practice components. Details were gathered directly from the study publications and directly from intervention manuals when available. We relied on authors’ explicit description of components whenever possible; for example, the presence of “stress management” would not be inferred from a coping skills intervention unless the authors discussed stress specifically. In many cases, authors expounded on program elements in tables or figures. Program content components were coded according to a system based on the work of Boustani et al16 in which the PracticeWise Clinical Coding System17 was used to identify common practices across a range of prevention programs. We also added other program components relating to theoretically relevant methods.18 Finally, on the basis of the PracticeWise recommendations and as implemented by Brown et al19 in a similar activity, we recorded other frequently occurring components as free text and ultimately integrated them as new codes into the framework (Table 1). Some of these included decision-making,20 conflict resolution,21 mindfulness,22,23 and alcohol and drug education.24 

TABLE 1

Included Components

ComponentDefinition
Activity monitoring and schedulinga Practical approach to monitor activities and/or scheduling; completion of an activity chart to aid in motivation and organization 
Alcohol and/or drug educationa Specific knowledge and/or education about the use of or effects of drugs and/or alcohol on development, lifestyle, including harm minimization approaches, and beliefs and/or perceptions about drugs/alcohol 
Anger managementb Skills to manage anger and/or angry feelings; control techniques 
Assertivenessb Techniques to increase confidence, standing up for oneself, standing ground, and/or holding a position 
Behavioral activationc Therapy technique of approaching activities that one is avoiding and analyzing how cognitive processes play a part 
Civic and/or social responsibilityb Engagement with community and/or community-based institution such as school, church, or political system; encouragement to involve oneself; responsibility to others (bystander intervention) 
Cognitive restructuringb Identification and replacement of unhelpful thoughts with more helpful thoughts 
Communication skillsb Improvement of ways in which participants use words; nonverbal styles of communication, expression of feelings or beliefs, and engagement with others 
Conflict resolutiona Skills to resolve conflict or negotiation between ≥2 people 
Coping skillsb Methods a person uses to deal with stressful situations; grief management 
Decision makinga Ability to review information and select a choice 
Emotional regulationc Ability to effectively manage and respond to an emotional experience 
Goal settingb Identification of a goal; establishment of measurable ways to accomplish and timeline 
Insight buildingb Development of internal insights, reflection, probing a better understanding of personal motivations (guided); self-awareness 
Interpersonal relationships and/or skillsa Skills to develop or improved close, strong relationships between ≥2 people 
Mental health literacya Knowledge and beliefs about mental disorders; reducing stigma and increasing awareness 
Mindfulnessc Psychological process of bringing one’s attention to experiences occurring in the present moment, which can be developed through the practice of meditation and other training 
Problem solvingb Process of finding, and perhaps acting on, a solution to a challenge or difficult problem 
Relaxationb Techniques for freeing oneself from tension and anxiety; learning how to remove oneself from a state of agitation 
Resisting drug/alcohol-related peer pressurea Specific refusal skills or self-efficacy as it relates directly to drug or alcohol use or pressure to use 
Self-efficacyb One’s own beliefs and capacity to execute behaviors necessary to produce a specific performance; acting on knowledge learned 
Self-monitoringb Observation and regulation of one’s own mood and behavior in a social setting; diary keeping and/or journaling 
Social skillsb Competence in communicating, interacting, and engaging with others 
Stress managementa A large range of techniques to control levels of stress, especially chronic stress that impedes everyday functioning 
Support networkingb Identification of a group of people who can provide emotional and practical help to manage difficult situations 
ComponentDefinition
Activity monitoring and schedulinga Practical approach to monitor activities and/or scheduling; completion of an activity chart to aid in motivation and organization 
Alcohol and/or drug educationa Specific knowledge and/or education about the use of or effects of drugs and/or alcohol on development, lifestyle, including harm minimization approaches, and beliefs and/or perceptions about drugs/alcohol 
Anger managementb Skills to manage anger and/or angry feelings; control techniques 
Assertivenessb Techniques to increase confidence, standing up for oneself, standing ground, and/or holding a position 
Behavioral activationc Therapy technique of approaching activities that one is avoiding and analyzing how cognitive processes play a part 
Civic and/or social responsibilityb Engagement with community and/or community-based institution such as school, church, or political system; encouragement to involve oneself; responsibility to others (bystander intervention) 
Cognitive restructuringb Identification and replacement of unhelpful thoughts with more helpful thoughts 
Communication skillsb Improvement of ways in which participants use words; nonverbal styles of communication, expression of feelings or beliefs, and engagement with others 
Conflict resolutiona Skills to resolve conflict or negotiation between ≥2 people 
Coping skillsb Methods a person uses to deal with stressful situations; grief management 
Decision makinga Ability to review information and select a choice 
Emotional regulationc Ability to effectively manage and respond to an emotional experience 
Goal settingb Identification of a goal; establishment of measurable ways to accomplish and timeline 
Insight buildingb Development of internal insights, reflection, probing a better understanding of personal motivations (guided); self-awareness 
Interpersonal relationships and/or skillsa Skills to develop or improved close, strong relationships between ≥2 people 
Mental health literacya Knowledge and beliefs about mental disorders; reducing stigma and increasing awareness 
Mindfulnessc Psychological process of bringing one’s attention to experiences occurring in the present moment, which can be developed through the practice of meditation and other training 
Problem solvingb Process of finding, and perhaps acting on, a solution to a challenge or difficult problem 
Relaxationb Techniques for freeing oneself from tension and anxiety; learning how to remove oneself from a state of agitation 
Resisting drug/alcohol-related peer pressurea Specific refusal skills or self-efficacy as it relates directly to drug or alcohol use or pressure to use 
Self-efficacyb One’s own beliefs and capacity to execute behaviors necessary to produce a specific performance; acting on knowledge learned 
Self-monitoringb Observation and regulation of one’s own mood and behavior in a social setting; diary keeping and/or journaling 
Social skillsb Competence in communicating, interacting, and engaging with others 
Stress managementa A large range of techniques to control levels of stress, especially chronic stress that impedes everyday functioning 
Support networkingb Identification of a group of people who can provide emotional and practical help to manage difficult situations 
a

Iteratively added by coding team.

b

From Boustani et al.16 

c

From Singla et al.25 

For reporting and analysis, we categorized all universally delivered programs into face-to-face, digital, or combined modality interventions. Face-to-face interventions consisted of all interventions delivered in schools, communities, or health centers; digital and combined modality interventions consisted of interventions that were solely digitally delivered content or digital content in combination with other modes of delivery.

Effect estimates from included studies were converted to standardized mean differences by using available published formulas.26 A common problem in meta-analyses of complex interventions is that study authors report multiple effect estimates from the same domain (ie, conceptually exchangeable and thus equally valid) in respect to an outcome and often report outcomes from multiple time points. To address this, we used a robust variance estimation meta-analysis model27 to include all relevant information from included studies. We estimated all models using random effects, given high anticipated levels of statistical heterogeneity and an intercorrelation parameter of 0.8, which is standard, to estimate how closely effect estimates within a study are related. Given the number and diversity of components we sought to analyze, we estimated a series of single-predictor meta-regression models. Predictors were entered into models as the study-level mean of a component. In standard 2-arm trials and trials in which components were binary, this variable took on the value of 1 or 0. In multiarm trials in which the ≥2 active arms differed as to the presence of a component, the variable took on the value of the proportion of effect estimates with a specific component. We estimated all models first with effect estimates corresponding to ≤2 months of follow-up and then with effect estimates over all follow-up times. We noted when models could not provide usable evidence because of model instability. We did not formally test publication bias given that these tests are not understood in the context of robust variance estimation meta-analysis. In assessing differences in effect sizes, we used standard thresholds of 0.2 for small effect size, 0.5 for medium effect size, and 0.8 for large effect size.28 

We identified 14 600 records through database searches and hand searching, of which 158 were suitable and reported data suitable for components analysis (Fig 1).

FIGURE 1

Flowchart. Articles in which authors reported on the same study and sample were combined for analysis (thus, n = 492 decreased to 473). a Exclusion reasons were wrong age (n = 59), wrong study design (n = 23), wrong outcome (n = 10), wrong intervention (n = 21), wrong population (n = 18), pre-2000 study (n = 1), could not locate (n = 6), wrong publication (ie, conference proceeding; n = 70).

FIGURE 1

Flowchart. Articles in which authors reported on the same study and sample were combined for analysis (thus, n = 492 decreased to 473). a Exclusion reasons were wrong age (n = 59), wrong study design (n = 23), wrong outcome (n = 10), wrong intervention (n = 21), wrong population (n = 18), pre-2000 study (n = 1), could not locate (n = 6), wrong publication (ie, conference proceeding; n = 70).

The characteristics of studies that met the inclusion criteria and contributed data to the components analysis are summarized in Table 2, and all included studies are listed in Supplemental Information 2. The average intervention duration was 13.88 hours for face-to-face interventions and 6.05 hours for digital interventions (see Supplemental Information 4 and 5). The average number of components per intervention was 5.4 for face-to-face interventions and 5.9 for digital interventions (further details in Supplemental Information 4 and 5).

TABLE 2

Descriptors of the Included Universal Interventions

DescriptorFace-to-face Prevalence (n = 129 Studies)Digital and Combined Prevalence (n = 29 Studies)
Design, n (%)   
 RCT 47 (36.4) 14 (46.7) 
 Cluster RCT 80 (62.0) 15 (53.3) 
 Crossover RCT 2 (1.6) 0 (0.0) 
High-income setting, n (%) 115 (89.1) 28 (96.7) 
 United States 72 (55.8) 13 (46.7) 
 Australia 16 (12.4) 9 (30.0) 
LMIC, n (%) 14 (10.9) 1 (3.3) 
Age in categories, n (%)   
 10–14 y 90 (69.8) 17 (56.7) 
 15–19 y 24 (18.6) 9 (33.3) 
 Across both categories 12 (9.3) 3 (10.0) 
 Missing data 3 (2.3) 0 (0.0) 
Setting, n (%)   
 School 111 (86.0) N/A 
 Community 8 (6.2) N/A 
 Multisetting 5 (3.9) N/A 
 Health center 4 (3.1) N/A 
 University 1 (0.8) N/A 
 Digital only N/A 20 (70.0) 
 Combined digital N/A 9 (30.0) 
Sex   
n 113 28 
 Girls, % 51.8 55.9 
 Boys, % 48.4 44.1 
Sample size, average (SD) 1415 (2341.86) 1650 (2111.94) 
DescriptorFace-to-face Prevalence (n = 129 Studies)Digital and Combined Prevalence (n = 29 Studies)
Design, n (%)   
 RCT 47 (36.4) 14 (46.7) 
 Cluster RCT 80 (62.0) 15 (53.3) 
 Crossover RCT 2 (1.6) 0 (0.0) 
High-income setting, n (%) 115 (89.1) 28 (96.7) 
 United States 72 (55.8) 13 (46.7) 
 Australia 16 (12.4) 9 (30.0) 
LMIC, n (%) 14 (10.9) 1 (3.3) 
Age in categories, n (%)   
 10–14 y 90 (69.8) 17 (56.7) 
 15–19 y 24 (18.6) 9 (33.3) 
 Across both categories 12 (9.3) 3 (10.0) 
 Missing data 3 (2.3) 0 (0.0) 
Setting, n (%)   
 School 111 (86.0) N/A 
 Community 8 (6.2) N/A 
 Multisetting 5 (3.9) N/A 
 Health center 4 (3.1) N/A 
 University 1 (0.8) N/A 
 Digital only N/A 20 (70.0) 
 Combined digital N/A 9 (30.0) 
Sex   
n 113 28 
 Girls, % 51.8 55.9 
 Boys, % 48.4 44.1 
Sample size, average (SD) 1415 (2341.86) 1650 (2111.94) 

N/A, not applicable.

In general, risk of bias was low across most categories, with the exception of allocation concealment and random sequence generation (see Fig 2 and Supplemental Information 3 for full details). In the majority of studies, it was unclear who had been responsible for randomization as well as how the randomization sequence was generated. It was also unclear if this sequence was protected sufficiently to prevent the research team from predicting the next treatment allocation during the process. Furthermore, in many studies, blinding of participants and outcome assessment was not possible because of the study design, particularly in school-based settings in which whole schools or specific classes were allocated to the intervention status. Outcome data assessment largely presented a low risk of bias, but approximately one-third of studies had unclear risk of attrition or other biases. Almost 90% of studies had a low risk of bias for selective reporting.

FIGURE 2

Risk of bias across all studies.

FIGURE 2

Risk of bias across all studies.

The overall effect sizes of universally delivered interventions on each of the study outcomes are reported in Table 3. Self-harm and suicide were not included in the analyses reported here because there were only 2 universally delivered interventions located in which these outcomes were reported. For face-to-face interventions in the short-term, there were significant differences between intervention and control groups for positive mental health and depression and anxiety symptoms. Across all time points, there were significant differences between intervention and control groups for positive mental health, depression and anxiety symptoms, and violence, aggression, and bullying. For digital or combined modality interventions, only depression and anxiety outcomes improved in the short-term, but this was not evident across all time points. For all time points, there were significant differences between intervention and control for positive mental health and substance use. All overall significant effect sizes were small to moderate and indicated beneficial effects of interventions. There were no differences for face-to-face interventions for substance use at any time point or for violence, aggression, and bullying in the short-term. There were no differences for digital and combined modality interventions for short-term positive mental health and substance use outcomes, depression and anxiety beyond the short-term, or for aggression, violence, and bullying across any time point.

TABLE 3

Overall Effect Sizes per Outcome

<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Positive mental health   
 Face-to-face 0.247 (0.100 to 0.395) 0.257 (0.097 to 0.416) 
 Digital and combined 0.175 (−0.034 to 0.383) 0.197 (0.016 to 0.379) 
Depression and anxiety symptoms   
 Face-to-face −0.104 (−0.197 to −0.01) −0.088 (−0.151 to −0.025) 
 Digital and combined −0.094 (−0.183 to −0.004) −0.054 (−0.181 to 0.074) 
Violence, aggression, and bullying   
 Face-to-face −0.138 (−0.235 to 0.049) −0.294 (−0.564 to −0.024) 
 Digital and combined −0.073 (−0.242 to 0.095) −0.075 (−0.249 to 0.099) 
Substance use   
 Face-to-face 0.017 (−0.085 to 0.119) −0.04 (−0.117 to 0.037) 
 Digital and combined −0.048 (−0.16 to 0.064) −0.114 (−0.199 to −0.029) 
<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Positive mental health   
 Face-to-face 0.247 (0.100 to 0.395) 0.257 (0.097 to 0.416) 
 Digital and combined 0.175 (−0.034 to 0.383) 0.197 (0.016 to 0.379) 
Depression and anxiety symptoms   
 Face-to-face −0.104 (−0.197 to −0.01) −0.088 (−0.151 to −0.025) 
 Digital and combined −0.094 (−0.183 to −0.004) −0.054 (−0.181 to 0.074) 
Violence, aggression, and bullying   
 Face-to-face −0.138 (−0.235 to 0.049) −0.294 (−0.564 to −0.024) 
 Digital and combined −0.073 (−0.242 to 0.095) −0.075 (−0.249 to 0.099) 
Substance use   
 Face-to-face 0.017 (−0.085 to 0.119) −0.04 (−0.117 to 0.037) 
 Digital and combined −0.048 (−0.16 to 0.064) −0.114 (−0.199 to −0.029) 

For positive mental health, a positive effect size denotes a beneficial effect. For all other outcomes, a negative effect size denotes a beneficial effect. CI, confidence interval; ES, effect size.

Seven intervention components predicted only positive effects, that is, their presence was associated with more successful programs (see Tables 4 and 5). These were interpersonal skills, emotional regulation, alcohol and drug education, mindfulness, problem solving, assertiveness training, and stress management. The presence of interpersonal skills was most consistently associated with larger effect sizes, yielding improved effects for positive mental health, depression and anxiety prevention, and prevention of substance use. Emotional regulation was associated with greater effectiveness in improving positive mental health and greater reductions in depressive and anxious symptomatology. Alcohol and drug education predicted positive outcomes for non–alcohol- and non–drug-related outcomes, namely, positive mental health in face-to-face interventions and aggression in digital interventions. The remaining components were associated with larger effect sizes in 1 outcome category only. Mindfulness was associated with a decrease in anxiety and depression symptoms in face-to-face interventions. Problem solving was associated with a decrease in depression and anxiety symptoms in digital and combined interventions. Assertiveness and stress management predicted larger effect sizes for the prevention of substance use in digital interventions. See Supplemental Information 2 for full details of the presence of program components in interventions.

TABLE 4

Face-to-face Program Component Effect Sizes per Outcome Area

Program ComponentsPromotion of Positive Mental HealthPrevention of Anxious and Depressive SymptomologyPrevention of Violence, Aggression, and BullyingPrevention of Substance Use
<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Activity monitoring and scheduling −0.264 (−1.061 to 0.534)a −0.237 (−1.172 to 0.644)a 0.162 (0.061 to 0.263) 0.123 (0.057 to 0.19) — — — — 
Alcohol and/or drug education 0.139 (−0.015 to 0.293) −0.200 (−0.751 to 0.351)a — — 0.087 (−0.11 to 0.284) 0.357 (−0.263 to 0.977)a −0.032 (−0.234 to 0.169) −0.025 (−0.181 to 0.131) 
Anger management 0.155 (−0.015 to 0.293) −0.038 (−0.400 to 0.324) 0.062 (−0.317 to 0.44)a 0.063 (−0.098 to 0.224) 0.216 (−0.075 to 0.508) 0.409 (0.003 to 0.789) 0.077 (−0.032 to 0.185) −0.093 (−0.735 to 0.548)a 
Assertiveness 0.100 (−0.356 to 0.555) 0.071 (−0.345 to 0.486) 0.125 (−0.083 to 0.333) −0.009 (−0.167 to 0.149) −0.263 (−1.05 to 0.525) −0.371 (−1.242 to 0.5) 0.068 (−0.203 to 0.339)a −0.078 (−0.221 to 0.066) 
Behavioral activation −0.199 (1.064 to 0.667)a −0.203 (−0.62 to 0.214)a — — 0.114 (−0.497 to 0.725)a 0.296 (−0.225 to 0.313)a — — 
Civic responsibility 0.093 (−0.515 to 0.701) −0.005 (−0.439 to 0.428) 0.335 (−1.793 to 2.463)a 0.284 (−1.99 to 2.558)a −0.189 (−1.197 to 0.32)a 0.076 (−0.424 to 0.576) — −0.069 (−0.552 to 0.415)a 
Cognitive restructuring 0.174 (−0.193 to 0.542) 0.037 (−0.287 to 0.361) −0.033 (−0.227 to 0.161) −0.056 (−0.191 to 0.08) −0.184 (−0.871 to 0.503) −0.068 (−0.802 to 0665) 0.166 (−0.704 to 1.036)a 0.168 (−0.346 to 0.682) 
Communication skills 0.026 (−0.367 to 0.418) −0.041 (−0.409 to 0.326) 0.190 (−0.006 to 0.387) 0.111 (−0.015 to 0.238) −0.012 (−0.562 to 0.538) −0.161 (−0.856 to 0.534) 0.133 (−0.097 to 0.363) 0.006 (−0.125 to 0.136) 
Conflict resolution −0.286 (−0.871 to 0.300) −0.257 (−0.707 to 0.194) 0.179 (−0.056 to 0.413) 0.147 (0.015 to 0.279) 0.211 (−0.104 to 0.525) 0.08 (−0.578 to 0.737) −0.127 (−0.234 to −0.021) −0.066 (−0.384 to 0.253) 
Coping skills 0.001 (−0.326 to 0.328) −0.115 (−0.428 to 0.198) 0.134 (−0.047 to 0.315) 0.049 (−0.081 to 0.179) 0.083 (−0.300 to 0.466) −0.163 (−0.800 to 0.473) 0.166 (−0.704 to 1.036) 0.073 (−0.150 to 0.295) 
Decision-making −0.274 (−0.557 to 0.009) −0.207 (−0.465 to 0.051) 0.263 (0.093 to 0.433) 0.116 (−0.031 to 0.263) 0.058 (−0.361 to 0.478) 0.306 (−0.090 to 0.703) −0.068 (−0.283 to 0.148) −0.064 (−0.201 to 0.072) 
Emotional regulation 0.33 (−0.035 to 0.694) 0.123 (−0.201 to 0.447) −0.087 (−0.293 to 0.124) −0.062 (−0.199 to 0.075) −0.250 (−0.800 to 0.300) −0.305 (−0.887 to 0.277) 0.174 (−0.306 to 0.655) 0.051 (−0.421 to 0.524) 
Goal setting 0.123 (−0.365 to 0.61) 0.027 (−0.376 to 0.431) 0.194 (−0.032 to 0.421) 0.175 (0.025 to 0.326) −0.015 (−0.316 to 0.335)a 0.274 (−0.271 to 0.818) 0.185 (−0.248 to 0.617) 0.099 (−0.105 to 0.304) 
Insight building 0.142 (−0.269 to 0.553) 0.067 (−0.265 to 0.399) 0.157 (−0.04 to 0.353) 0.070 (−0.074 to 0.215) −0.205 (−1.131 to 0.72) −0.074 (−0.791 to 0.646) −0.023 (−0.238 to 0.192)a 0.040 (−0.111 to 0.191) 
Interpersonal relationships and/or skills 0.048 (−0.354 to 0.45) 0.143 (−0.256 to 0.542) 0.070 (−0.169 to 0.309) 0.066 (−0.074 to 0.206) 0.044 (−0.377 to 0.465) −0.175 (−0.716 to 0.366) −0.160 (−0.344 to 0.024) −0.190 (−0.313 to −0.067) 
Mental health literacy −0.048 (−0.650 to 0.555) −0.071 (−0.659 to 0.517) −0.029 (−0.521 to 0.463) 0.005 (−0.228 to 0.237) 0.106 (−1.247 to 1.459) 0.343 (−0.331 to 1.517) — −0.036 (−1.056 to 1.434) 
Mindfulness 0.157 (−0.236 to 0.550) 0.149 (−0.266 to 0.564) −0.272 (−0.481 to −0.063) −0.219 (−0.399 to −0.040) −0.529 (−1.599 to 0.542) −0.428 (−1.692 to 0.836) −0.041 (−0.149 to 0.068) — 
Problem solving 0.059 (−0.263 to 0.381) −0.055 (−0.390 to 0.280) 0.115 (−0.076 to 0.305) 0.061 (−0.074 to 0.195) 0.209 (−0.154 to 0.572) 0.136 (−0.422 to 0.694) 0.349 (−0.134 to 0.832) −0.019 (−0.242 to 0.205) 
Relaxation 0.226 (−0.161 to 0.614) 0.147 (−0.219 to 0.513) −0.081 (−0.278 to 0.117) −0.101 (−0.245 to 0.043) −0.818 (−3.047 to 1.412) −0.758 (−2.387 to 0.871) −0.041 (−0.149 to 0.068) — 
Resisting peer pressure −0.201 (−0.881 to 0.479) −0.293 (0.610 to 0.025) — 0.193 (−0.358 to 0.744) — 0.459 (−0.022 to 0.939) 0.010 (−0.212 to 0.232) −0.059 (−0.190 to 0.073) 
Self-efficacy 0.037 (−0.402 to 0.477) 0.434 (−0.387 to 1.255) 0.251 (0.089 to 0.412) 0.141 (0.003 to 0.278) 0.150 (−1.057 to 1.356) −0.622 (−2.803 to 1.559) 0.137 (−0.206 to 0.480) 0.155 (−0.044 to 0.354) 
Self-monitoring −0.126 (−0.473 to 0.222) −0.160 (−0.479 to 0.159) −0.040 (−0.224 to 0.144) −0.046 (−0.018 to 0.088) −0.298 (−1.000 to 0.404) −0.286 (−1.180 to 0.609) −0.092 (−0.773 to 0.590) 0.230 (−0.488 to 0·948) 
Social skills −0.201 (−0.582 to 0.181) 0.132 (−0.451 to 0.714) 0.138 (−0.078 to 0.354) 0.079 (−0.079 to 0.237) −0.157 (−0.788 to 0.473) −0.352 (−1.269 to 0.556) −0.138 (−0.413 to 0.141) −0.015 (−0.243 to 0.214) 
Stress management −0.039 (0.368 to 0.289) −0.071 (−0.375 to 0.234) −0.090 (−0.290 to 0.111) −0.084 (−0.250 to 0.082) 0.198 (−0.159 to 0.555) 0.344 (−0.080 to 0.767) −0.041 (−0.149 to 0.068) 0.065 (−0.524 to 0.654) 
Support networking 0.015 (−0.340 to 0.369) −0.113 (0.437 to 0.210) −0.027 (−0.258 to 0.203) 0.010 (−0.148 to 0.168) 0.092 (−0.271 to 0.456) 0.307 (−0.144 to 0.757) −0.012 (−0.883 to 0.86) 0.019 (−0.149 to 0.187) 
Program ComponentsPromotion of Positive Mental HealthPrevention of Anxious and Depressive SymptomologyPrevention of Violence, Aggression, and BullyingPrevention of Substance Use
<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Activity monitoring and scheduling −0.264 (−1.061 to 0.534)a −0.237 (−1.172 to 0.644)a 0.162 (0.061 to 0.263) 0.123 (0.057 to 0.19) — — — — 
Alcohol and/or drug education 0.139 (−0.015 to 0.293) −0.200 (−0.751 to 0.351)a — — 0.087 (−0.11 to 0.284) 0.357 (−0.263 to 0.977)a −0.032 (−0.234 to 0.169) −0.025 (−0.181 to 0.131) 
Anger management 0.155 (−0.015 to 0.293) −0.038 (−0.400 to 0.324) 0.062 (−0.317 to 0.44)a 0.063 (−0.098 to 0.224) 0.216 (−0.075 to 0.508) 0.409 (0.003 to 0.789) 0.077 (−0.032 to 0.185) −0.093 (−0.735 to 0.548)a 
Assertiveness 0.100 (−0.356 to 0.555) 0.071 (−0.345 to 0.486) 0.125 (−0.083 to 0.333) −0.009 (−0.167 to 0.149) −0.263 (−1.05 to 0.525) −0.371 (−1.242 to 0.5) 0.068 (−0.203 to 0.339)a −0.078 (−0.221 to 0.066) 
Behavioral activation −0.199 (1.064 to 0.667)a −0.203 (−0.62 to 0.214)a — — 0.114 (−0.497 to 0.725)a 0.296 (−0.225 to 0.313)a — — 
Civic responsibility 0.093 (−0.515 to 0.701) −0.005 (−0.439 to 0.428) 0.335 (−1.793 to 2.463)a 0.284 (−1.99 to 2.558)a −0.189 (−1.197 to 0.32)a 0.076 (−0.424 to 0.576) — −0.069 (−0.552 to 0.415)a 
Cognitive restructuring 0.174 (−0.193 to 0.542) 0.037 (−0.287 to 0.361) −0.033 (−0.227 to 0.161) −0.056 (−0.191 to 0.08) −0.184 (−0.871 to 0.503) −0.068 (−0.802 to 0665) 0.166 (−0.704 to 1.036)a 0.168 (−0.346 to 0.682) 
Communication skills 0.026 (−0.367 to 0.418) −0.041 (−0.409 to 0.326) 0.190 (−0.006 to 0.387) 0.111 (−0.015 to 0.238) −0.012 (−0.562 to 0.538) −0.161 (−0.856 to 0.534) 0.133 (−0.097 to 0.363) 0.006 (−0.125 to 0.136) 
Conflict resolution −0.286 (−0.871 to 0.300) −0.257 (−0.707 to 0.194) 0.179 (−0.056 to 0.413) 0.147 (0.015 to 0.279) 0.211 (−0.104 to 0.525) 0.08 (−0.578 to 0.737) −0.127 (−0.234 to −0.021) −0.066 (−0.384 to 0.253) 
Coping skills 0.001 (−0.326 to 0.328) −0.115 (−0.428 to 0.198) 0.134 (−0.047 to 0.315) 0.049 (−0.081 to 0.179) 0.083 (−0.300 to 0.466) −0.163 (−0.800 to 0.473) 0.166 (−0.704 to 1.036) 0.073 (−0.150 to 0.295) 
Decision-making −0.274 (−0.557 to 0.009) −0.207 (−0.465 to 0.051) 0.263 (0.093 to 0.433) 0.116 (−0.031 to 0.263) 0.058 (−0.361 to 0.478) 0.306 (−0.090 to 0.703) −0.068 (−0.283 to 0.148) −0.064 (−0.201 to 0.072) 
Emotional regulation 0.33 (−0.035 to 0.694) 0.123 (−0.201 to 0.447) −0.087 (−0.293 to 0.124) −0.062 (−0.199 to 0.075) −0.250 (−0.800 to 0.300) −0.305 (−0.887 to 0.277) 0.174 (−0.306 to 0.655) 0.051 (−0.421 to 0.524) 
Goal setting 0.123 (−0.365 to 0.61) 0.027 (−0.376 to 0.431) 0.194 (−0.032 to 0.421) 0.175 (0.025 to 0.326) −0.015 (−0.316 to 0.335)a 0.274 (−0.271 to 0.818) 0.185 (−0.248 to 0.617) 0.099 (−0.105 to 0.304) 
Insight building 0.142 (−0.269 to 0.553) 0.067 (−0.265 to 0.399) 0.157 (−0.04 to 0.353) 0.070 (−0.074 to 0.215) −0.205 (−1.131 to 0.72) −0.074 (−0.791 to 0.646) −0.023 (−0.238 to 0.192)a 0.040 (−0.111 to 0.191) 
Interpersonal relationships and/or skills 0.048 (−0.354 to 0.45) 0.143 (−0.256 to 0.542) 0.070 (−0.169 to 0.309) 0.066 (−0.074 to 0.206) 0.044 (−0.377 to 0.465) −0.175 (−0.716 to 0.366) −0.160 (−0.344 to 0.024) −0.190 (−0.313 to −0.067) 
Mental health literacy −0.048 (−0.650 to 0.555) −0.071 (−0.659 to 0.517) −0.029 (−0.521 to 0.463) 0.005 (−0.228 to 0.237) 0.106 (−1.247 to 1.459) 0.343 (−0.331 to 1.517) — −0.036 (−1.056 to 1.434) 
Mindfulness 0.157 (−0.236 to 0.550) 0.149 (−0.266 to 0.564) −0.272 (−0.481 to −0.063) −0.219 (−0.399 to −0.040) −0.529 (−1.599 to 0.542) −0.428 (−1.692 to 0.836) −0.041 (−0.149 to 0.068) — 
Problem solving 0.059 (−0.263 to 0.381) −0.055 (−0.390 to 0.280) 0.115 (−0.076 to 0.305) 0.061 (−0.074 to 0.195) 0.209 (−0.154 to 0.572) 0.136 (−0.422 to 0.694) 0.349 (−0.134 to 0.832) −0.019 (−0.242 to 0.205) 
Relaxation 0.226 (−0.161 to 0.614) 0.147 (−0.219 to 0.513) −0.081 (−0.278 to 0.117) −0.101 (−0.245 to 0.043) −0.818 (−3.047 to 1.412) −0.758 (−2.387 to 0.871) −0.041 (−0.149 to 0.068) — 
Resisting peer pressure −0.201 (−0.881 to 0.479) −0.293 (0.610 to 0.025) — 0.193 (−0.358 to 0.744) — 0.459 (−0.022 to 0.939) 0.010 (−0.212 to 0.232) −0.059 (−0.190 to 0.073) 
Self-efficacy 0.037 (−0.402 to 0.477) 0.434 (−0.387 to 1.255) 0.251 (0.089 to 0.412) 0.141 (0.003 to 0.278) 0.150 (−1.057 to 1.356) −0.622 (−2.803 to 1.559) 0.137 (−0.206 to 0.480) 0.155 (−0.044 to 0.354) 
Self-monitoring −0.126 (−0.473 to 0.222) −0.160 (−0.479 to 0.159) −0.040 (−0.224 to 0.144) −0.046 (−0.018 to 0.088) −0.298 (−1.000 to 0.404) −0.286 (−1.180 to 0.609) −0.092 (−0.773 to 0.590) 0.230 (−0.488 to 0·948) 
Social skills −0.201 (−0.582 to 0.181) 0.132 (−0.451 to 0.714) 0.138 (−0.078 to 0.354) 0.079 (−0.079 to 0.237) −0.157 (−0.788 to 0.473) −0.352 (−1.269 to 0.556) −0.138 (−0.413 to 0.141) −0.015 (−0.243 to 0.214) 
Stress management −0.039 (0.368 to 0.289) −0.071 (−0.375 to 0.234) −0.090 (−0.290 to 0.111) −0.084 (−0.250 to 0.082) 0.198 (−0.159 to 0.555) 0.344 (−0.080 to 0.767) −0.041 (−0.149 to 0.068) 0.065 (−0.524 to 0.654) 
Support networking 0.015 (−0.340 to 0.369) −0.113 (0.437 to 0.210) −0.027 (−0.258 to 0.203) 0.010 (−0.148 to 0.168) 0.092 (−0.271 to 0.456) 0.307 (−0.144 to 0.757) −0.012 (−0.883 to 0.86) 0.019 (−0.149 to 0.187) 

For positive mental health, a positive effect size denotes a beneficial effect. For all other outcomes, a negative effect size denotes a beneficial effect. CI, confidence interval; ES, effect size; —, models that did not run because of limited variation.

a

Models are indicative only given the statistical estimation procedures used.

TABLE 5

Digital and Combined Program Component Effect Sizes per Outcome Area

Program ComponentsPromotion of Positive Mental HealthPrevention of Anxious and Depressive SymptomologyPrevention of Violence, Aggression, and BullyingPrevention of Substance Use
<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Activity monitoring and scheduling — — — — — — — — 
Alcohol and/or drug education −0.119 (−0.512 to 0.275) −0.042 (−0.385 to 0.301) — 0.081 (−1.190 to 1.352) −0.402(−0.543 to −0.261) −0.401 (0.543 to −0.254) — — 
Anger management — — — — — — — — 
Assertiveness 0.019 (−0.430 to 0.468) 0.108 (−0.278 to 0.493) −0.112 (−0.375 to 0.151) 0.012 (−0.497 to 0.520) — — −0.055 (−0.482 to 0.371) −0.151 (−0.303 to 0.001) 
Behavioral activation — — — — — — — — 
Civic responsibility — 0.095 (−0.112 to 0.301) — 0.348 (0.246 to 0.451) — 0.104 (−0.153 to 0.362) — — 
Cognitive restructuring −0.028 (−0.637 to 0.582) −0.023 (−0.656 to 0.610) −0.018 (−0.229 to 0.193) −0.095 (−0.346 to 0.155) — — — — 
Communication skills 0.080 (−0.492 to 0.652) 0.118 (−0.257 to 0.494) — 0.081 (−1.19 to 1.352) −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.088 (−0.334 to 0.158) −0.095 (−0.258 to 0.068 
Conflict resolution −0.192 (−0.427 to 0.652) −0.012 (−0.749 to 0.724) — 0.081 (−1.19 to 1.352) −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.136 (−0.425 to 0.152) −0.063 (−0.218 to 0.093) 
Coping skills −0.301 (−0.651 to 0.043) −0.181 (0.555 to 0.193) −0.101 (−0.291 to 0.090) −0.038 (−0.264 to 0.187) — — −0.153 (−0.460 to 0.154) −0.194 (−0.340 to 0.049) 
Decision-making 0.241 (−1.156 to 1.637) 0.208 (−1.257 to 1.673) — — −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.056 (−0.285 to 0.173) −0.110 (−0.272 to 0.052) 
Emotional regulation 0.114 (−1.515 to 1.742) 0.150 (−0.319 to 0.619) −0.121 (−0.192 to −0.049) −0.022 (−0.319 to 0.275) — — — −0.062 (−0.373 to 0.25) 
Goal setting 0.080 (−0.492 to 0.652) 0.118 (−0.257 to 0.494) — 0.081 (−1.190 to 1.352) — — — −0.125 (−0.274 to 0.024) 
Insight building −0.192 (−0.427 to 0.042) −0.012 (−0.749 to 0.724) — 0.081 (−1.130 to 1.352) — — — −0.062 (−0.373 to 0.250) 
Interpersonal relationship and/or skills 0.220 (−0.171 to 0.61) 0.300 (−0.004 to 0.636) −0.096 (−0.224 to 0.032) −0.033 (−0.290 to 0.224) −0.100 (−0.445 to 0.245) −0.098 (0.467 to 0.272) −0.316 (−0.425 to 0.152) −0.127 (−0.289 to 0.034) 
Mental health literacy −0.259 (−0.627 to 0.109) −0.175 (−0.530 to 0.181) 0.018 (−0.300 to 0.335) 0.105 (−0.278 to 0.488) — — −0.055 (−0.482 to 0.371) −0.042 (−0.219 to 0.135) 
Mindfulness 0.273 (−1.301 to 1.848) 0.220 (−1.495 to 1.936) 0.127 (−0.323 to 0.577) 0.074 (−0.423 to 0.571) — — — — 
Problem solving 0.019 (−0.430 to 0.468) 0.108 (−0.278 to 0.493) −0.128 (−0.191 to 0.064) −0.095 (−0.317 to 0.127) −0.037 (−0.232 to 0.158) −0.135 (−0.331 to 0.061) — −0.134 (−0.351 to 0.083) 
Relaxation −0.301 (−0.651 to 0.048) −0.181 (−0.555 to 0.193) −0.101 (−0.291 to 0.09) −0.038 (−0.264 to 0.187) — — −0.233 (−1.159 to 0.693) −0.185 (−0.382 to 0.011) 
Resisting peer pressure −0.132 (−0.34 to 0.577) −0.027 (−0.394 to 0.339) — 0.081 (−1.190 to 1.352) −0.402 (−0.543 to 0.261) −0.401 (−0.548 to −0.254) −0.099 (−0.406 to 0.209) −0.142 (−0.552 to 0.269) 
Self-efficacy −0.132 (−0.840 to 0.577) −0.027 (−0.394 to 0.339) −0.113 (−0.229 to 0.003) 0.002 (−0.42 to 0.423) −0.035 (−0.672 to 0.602) −0.030 (1.083 to 1.023) 0.038 (−0.289 to 0.365) −0.034 (−0.313 to 0.245) 
Self-monitoring 0.128 (−0.248 to 0.503) 0.132 (−0.233 to 0.498) 0.083 (−0.061 to 0.227) 0.147 (−0.073 to 0.366) — 0.104 (−0.153 to 0.362) −0.019 (−0.268 to 0.231 0.013 (−0.153 to 0.179) 
Social skills −0.132 (−0.840 to 0.577) −0.027 (−0.394 to 0.339) — 0.081 (−1.19 to 1.352) 0.177 (−0.326 to 0.681) 0.182 (−0.361 to 0.725) — −0.097 (−0.294 to 0.101) 
Stress management −0.265 (−0.628 to 0.098) −0.250 (−0.608 to 0.108) −0.038 (−0.206 to 0.13) −0.126 (−0.407 to 0.155) — — −0.233 (1.159 to 0.693) −0.248 (−0.446 to 0.049) 
Support networking 0.114 (−1.515 to 1.742) 0.150 (−0.319 to 0.619) — 0.081 (−1.19 to 1.352) 0.177 (−0.326 to 0.681) 0.182 (−0.361 to 0.725) −0.056 (0.482 to 0.371) −0.103 (−0.273 to 0.068) 
Program ComponentsPromotion of Positive Mental HealthPrevention of Anxious and Depressive SymptomologyPrevention of Violence, Aggression, and BullyingPrevention of Substance Use
<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)<2 mo, ES (95% CI)All Time Points, ES (95% CI)
Activity monitoring and scheduling — — — — — — — — 
Alcohol and/or drug education −0.119 (−0.512 to 0.275) −0.042 (−0.385 to 0.301) — 0.081 (−1.190 to 1.352) −0.402(−0.543 to −0.261) −0.401 (0.543 to −0.254) — — 
Anger management — — — — — — — — 
Assertiveness 0.019 (−0.430 to 0.468) 0.108 (−0.278 to 0.493) −0.112 (−0.375 to 0.151) 0.012 (−0.497 to 0.520) — — −0.055 (−0.482 to 0.371) −0.151 (−0.303 to 0.001) 
Behavioral activation — — — — — — — — 
Civic responsibility — 0.095 (−0.112 to 0.301) — 0.348 (0.246 to 0.451) — 0.104 (−0.153 to 0.362) — — 
Cognitive restructuring −0.028 (−0.637 to 0.582) −0.023 (−0.656 to 0.610) −0.018 (−0.229 to 0.193) −0.095 (−0.346 to 0.155) — — — — 
Communication skills 0.080 (−0.492 to 0.652) 0.118 (−0.257 to 0.494) — 0.081 (−1.19 to 1.352) −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.088 (−0.334 to 0.158) −0.095 (−0.258 to 0.068 
Conflict resolution −0.192 (−0.427 to 0.652) −0.012 (−0.749 to 0.724) — 0.081 (−1.19 to 1.352) −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.136 (−0.425 to 0.152) −0.063 (−0.218 to 0.093) 
Coping skills −0.301 (−0.651 to 0.043) −0.181 (0.555 to 0.193) −0.101 (−0.291 to 0.090) −0.038 (−0.264 to 0.187) — — −0.153 (−0.460 to 0.154) −0.194 (−0.340 to 0.049) 
Decision-making 0.241 (−1.156 to 1.637) 0.208 (−1.257 to 1.673) — — −0.106 (−0.987 to 0.775) −0.105 (−0.992 to 0.781) −0.056 (−0.285 to 0.173) −0.110 (−0.272 to 0.052) 
Emotional regulation 0.114 (−1.515 to 1.742) 0.150 (−0.319 to 0.619) −0.121 (−0.192 to −0.049) −0.022 (−0.319 to 0.275) — — — −0.062 (−0.373 to 0.25) 
Goal setting 0.080 (−0.492 to 0.652) 0.118 (−0.257 to 0.494) — 0.081 (−1.190 to 1.352) — — — −0.125 (−0.274 to 0.024) 
Insight building −0.192 (−0.427 to 0.042) −0.012 (−0.749 to 0.724) — 0.081 (−1.130 to 1.352) — — — −0.062 (−0.373 to 0.250) 
Interpersonal relationship and/or skills 0.220 (−0.171 to 0.61) 0.300 (−0.004 to 0.636) −0.096 (−0.224 to 0.032) −0.033 (−0.290 to 0.224) −0.100 (−0.445 to 0.245) −0.098 (0.467 to 0.272) −0.316 (−0.425 to 0.152) −0.127 (−0.289 to 0.034) 
Mental health literacy −0.259 (−0.627 to 0.109) −0.175 (−0.530 to 0.181) 0.018 (−0.300 to 0.335) 0.105 (−0.278 to 0.488) — — −0.055 (−0.482 to 0.371) −0.042 (−0.219 to 0.135) 
Mindfulness 0.273 (−1.301 to 1.848) 0.220 (−1.495 to 1.936) 0.127 (−0.323 to 0.577) 0.074 (−0.423 to 0.571) — — — — 
Problem solving 0.019 (−0.430 to 0.468) 0.108 (−0.278 to 0.493) −0.128 (−0.191 to 0.064) −0.095 (−0.317 to 0.127) −0.037 (−0.232 to 0.158) −0.135 (−0.331 to 0.061) — −0.134 (−0.351 to 0.083) 
Relaxation −0.301 (−0.651 to 0.048) −0.181 (−0.555 to 0.193) −0.101 (−0.291 to 0.09) −0.038 (−0.264 to 0.187) — — −0.233 (−1.159 to 0.693) −0.185 (−0.382 to 0.011) 
Resisting peer pressure −0.132 (−0.34 to 0.577) −0.027 (−0.394 to 0.339) — 0.081 (−1.190 to 1.352) −0.402 (−0.543 to 0.261) −0.401 (−0.548 to −0.254) −0.099 (−0.406 to 0.209) −0.142 (−0.552 to 0.269) 
Self-efficacy −0.132 (−0.840 to 0.577) −0.027 (−0.394 to 0.339) −0.113 (−0.229 to 0.003) 0.002 (−0.42 to 0.423) −0.035 (−0.672 to 0.602) −0.030 (1.083 to 1.023) 0.038 (−0.289 to 0.365) −0.034 (−0.313 to 0.245) 
Self-monitoring 0.128 (−0.248 to 0.503) 0.132 (−0.233 to 0.498) 0.083 (−0.061 to 0.227) 0.147 (−0.073 to 0.366) — 0.104 (−0.153 to 0.362) −0.019 (−0.268 to 0.231 0.013 (−0.153 to 0.179) 
Social skills −0.132 (−0.840 to 0.577) −0.027 (−0.394 to 0.339) — 0.081 (−1.19 to 1.352) 0.177 (−0.326 to 0.681) 0.182 (−0.361 to 0.725) — −0.097 (−0.294 to 0.101) 
Stress management −0.265 (−0.628 to 0.098) −0.250 (−0.608 to 0.108) −0.038 (−0.206 to 0.13) −0.126 (−0.407 to 0.155) — — −0.233 (1.159 to 0.693) −0.248 (−0.446 to 0.049) 
Support networking 0.114 (−1.515 to 1.742) 0.150 (−0.319 to 0.619) — 0.081 (−1.19 to 1.352) 0.177 (−0.326 to 0.681) 0.182 (−0.361 to 0.725) −0.056 (0.482 to 0.371) −0.103 (−0.273 to 0.068) 

For positive mental health, a positive effect size denotes a beneficial effect. For all other outcomes, a negative effect size denotes a beneficial effect. CI, confidence interval; ES, effect size; —, models that did not run because of limited variation.

Six practice components revealed mixed results across the different outcomes; these components were conflict resolution, coping skills, goal setting, relaxation, skills to resist peer pressure, and self-efficacy training. In face-to-face interventions, conflict resolution predicted larger effects for substance use but smaller effects for depression and anxiety symptoms. Coping skills content did not predict any outcomes for face-to-face interventions but predicted diminished effectiveness for positive mental health and stronger effectiveness for substance use for digital interventions. Goal setting was predictive of smaller program effects for depression and anxiety in face-to-face interventions but larger effects for digital substance use interventions. Relaxation was associated with smaller effect sizes for digital positive mental health outcomes but stronger effectiveness for substance use. Skills to resist peer pressure predicted larger effects for violence outcome in digital interventions but smaller effects for aggression and positive mental health outcomes in face-to-face interventions. For depression and anxiety symptoms, self-efficacy predicted smaller effect sizes for face-to-face interventions but larger effects for digital interventions.

Across all meta-regressions, 6 components were associated with either attenuated effectiveness or minimal difference in effectiveness, depending on the outcome; these components were activity monitoring and scheduling, anger management, civic responsibility, communication skills, decision-making, and insight building. Communication skills and activity monitoring and scheduling were associated with smaller effect sizes for depression and anxiety outcomes in face-to-face interventions. Digital and combined interventions that included civic responsibility were less effective at reducing depressive and anxious symptomatology. In face-to-face interventions, the inclusion of decision-making activities was associated with smaller effect sizes on positive mental health and depressive and anxious symptomatology. Insight building predicted a smaller effect size for positive mental health when included in digital and combined interventions.

A final set of practice components that did not have a clear relationship to effectiveness in either direction included cognitive restructuring, mental health literacy, self-monitoring, social skills, support networking, and behavioral activation.

This is the first global review of active components present in interventions that are aimed to improve adolescent health across a range of interrelated mental health outcomes. With the results of this review, we indicate, first, that universally delivered interventions can improve adolescent mental health and reduce risk behavior and, second, that there are several content-related program components that are associated with larger or smaller effect sizes. Of these components, however, only 3 predicted positive effects across multiple outcomes: interpersonal skills training, emotional regulation, and alcohol and drug education. This finding reflects those in a review by Singla et al25 which also found that interpersonal and emotional elements had the strongest associations with overall effectiveness across mental health interventions delivered by lay health workers in low- and middle-income countries (LMICs).

Developing skills to improve interpersonal relationships is highly relevant for improving adolescent mental health outcomes, and our findings indicate that including these skills in multioutcome interventions designed to promote mental health and prevent mental disorders and risk behaviors is a valuable strategy. Previous research has revealed that poor-quality relationships consistently predict poor mental health outcomes for adolescents,29,30 whereas positive relationships are associated with better mental health outcomes.31 In this review, intervention content commonly included verbal and nonverbal communication skills16 and was often combined with broader social skills training focusing on how an individual engages in a social setting or larger group.32,34 

Activities used to develop emotional regulation skills were common in interventions that were aimed to reduce depression and promote positive mental health, as well as those that were aimed to reduce aggression. Intervention programs that included emotional regulation encompassed whole-class interventions, cognitive behavioral interventions,35 antibullying interventions,36 and guided expressive writing interventions,37 as well as more broadly focused, integrated interventions.38 Many yoga and mindfulness-based interventions also employed an emotional regulation component, as practitioners guided adolescents through meditative sessions in which observing as well as engaging with emotions was encouraged.23,39 

Alcohol and drug education predicted larger effect sizes for mental health promotion and interventions addressing violence. This term covered a broad range of topics, including facts about alcohol, cannabis, and other illicit drugs, discussion about the risks of using illegal substances, social influences associated with alcohol use,32,40,42 media influences and pressures to use substances,43,45 and parent education about engaging their children in conversations about alcohol.24,46,47 Certain interventions also took a harm minimization approach, teaching adolescents about less harmful ways to use alcohol or ways to reduce risk for themselves or others.48,50 Delivery methods also differed. For example, in 1 digital intervention, participants are walked through the consequences of a virtual night of binge drinking.51 However, the reasons for the effect of alcohol and drug use on outcomes beyond substance use are unknown. It may be because of shared risk and protective factors between these outcomes and shared pathways to effective prevention between different types of outcomes that have been “triggered” by teaching adolescents drug use prevention content.

For “Helping Adolescents Thrive,” it is evident that intervention content that is strongly centered on interpersonal and emotional skills is most likely to be effective across multiple outcome domains. It is also possible that the active components identified in this review, particularly interpersonal skills and emotional regulation, may have effects that extend beyond our defined scope of mental health outcomes to broader mental health domains. For example, in other research, improvements in emotional regulation have been shown to reduce risky sexual behavior during adolescence52,54 because these skills may help adolescents develop stronger and more equitable relationships.52 

Given that the evidence base is almost entirely from high-income countries (HICs), it will be essential to track implementation efforts if and when these interventions are adapted for use in LMICs to ensure that they are implemented in a culturally and contextually valid and appropriate manner.7 Specifically, developing an intervention package on the basis of these findings will require active engagement with adolescents, particularly in low-resource settings, to translate relevant evidence-based principles into feasible and acceptable intervention programs that appeal to and effectively engage adolescents. Pursuing a user-centered design approach by employing multiple stages of engagement and prototyping with adolescents, their parents, their teachers, and other community stakeholders to coproduce the intervention package55 will significantly strengthen the development of the program and its adaptability to different settings.

Again, although this was a global review, the publications eligible for inclusion were overwhelmingly based in HICs. In studies from LMICs, adapted versions of evidence-based interventions from HICs are often used, which may affect the validity and reliability of their results. In addition, studies with randomized designs are more likely to be used to evaluate research program interventions, whereas quasi-experimental and other designs are often used for real-life interventions, meaning that restricting our screening to RCTs only may have limited the applicability of these findings to nonresearch settings.56 

The program components approach depends on the quality of reporting in publications. Brown et al19 note that essential details required to understand content and implementation are often missing from these publications. In the review by Singla et al,25 the authors further noted the lack of reporting about dosage for each component present. It was not always possible to determine program fidelity or mean dosage across participants, limiting the strength of their analyses.57 In this review, few study authors reported intervention components in enough detail to allow for replication; even fewer provided any form of guidance as to how interventions could be scaled up. Finally, a further limitation to be considered is the risk of bias in the included studies. Although considered to be low across most categories, allocation concealment and random sequence generation were high or unclear for the majority of studies, whereas in some cases, the nature of feasible study designs for universally delivered interventions (such as in schools) precluded blinding of participants and outcome assessment. Furthermore, the quality of the body of evidence was not assessed by using the Grading of Recommendations, Assessment, Development and Evaluation tool.

These are novel results that will be used to design a universally delivered intervention as a part of the “Helping Adolescents Thrive” initiative. Further work should be undertaken to develop and test interventions that use these core components, especially in underresourced settings in which multiple risk factors for poor adolescent health are present.

Dr Skeen designed the research plan, oversaw the full review process, and wrote the final study report; Drs Ross, Servili, Dua, and Tomlinson designed the research plan; Ms Laurenzi, Ms Gordon, and Ms du Toit completed all qualitative and quantitative data extraction and contributed toward drafting, reviewing, and revising the report; Ms Carvajal-Aguirre and Drs Eriksson de Carvalho, van der Westhuizen, Fleischmann, Kohl and Lund provided thorough input and feedback on the report at various stages, as well as reviewed the manuscript; Dr Brand conducted all risk-of-bias assessments on the included studies, generated the related figure, and reviewed the manuscript; Mr Dowdall contributed to the search strategy design and reviewed the manuscript; Dr Melendez-Torres conducted the meta-regression analyses and contributed toward designing, drafting, reviewing, and revising the manuscript; and all authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work.

FUNDING: Funded by the World Health Organization.

     
  • HIC

    high-income country

  •  
  • LMIC

    low- and middle-income country

  •  
  • RCT

    randomized controlled trial

1
Patel
V
,
Flisher
AJ
,
Nikapota
A
,
Malhotra
S
.
Promoting child and adolescent mental health in low and middle income countries.
J Child Psychol Psychiatry
.
2008
;
49
(
3
):
313
334
[PubMed]
2
Dick
B
,
Ferguson
BJ
.
Health for the world’s adolescents: a second chance in the second decade.
J Adolesc Health
.
2015
;
56
(
1
):
3
6
[PubMed]
3
Patton
GC
,
Sawyer
SM
,
Santelli
JS
, et al
.
Our future: a Lancet commission on adolescent health and wellbeing.
Lancet
.
2016
;
387
(
10036
):
2423
2478
[PubMed]
4
Kessler
RC
,
Aguilar-Gaxiola
S
,
Alonso
J
,
Chatterji
S
,
Lee
S
,
Ustün
TB
.
The WHO World Mental Health (WMH) surveys.
Psychiatrie (Stuttg)
.
2009
;
6
(
1
):
5
9
[PubMed]
5
Prince
M
,
Patel
V
,
Saxena
S
, et al
.
No health without mental health.
Lancet
.
2007
;
370
(
9590
):
859
877
[PubMed]
6
World Health Organization
.
Global Accelerated Action for the Health of Adolescents (AA-HA!): Guidance to Support Country Implementation
.
Geneva, Switzerland
:
World Health Organization
;
2017
7
Barry
MM
,
Clarke
AM
,
Jenkins
R
,
Patel
V
.
A systematic review of the effectiveness of mental health promotion interventions for young people in low and middle income countries.
BMC Public Health
.
2013
;
13
(
1
):
835
[PubMed]
8
O’Mara
L
,
Lind
C
.
What do we know about school mental health promotion programmes for children and youth?
Adv Sch Ment Health Promot
.
2013
;
6
(
3
):
203
224
9
Rotheram-Borus
MJ
,
Swendeman
D
,
Chorpita
BF
.
Disruptive innovations for designing and diffusing evidence-based interventions.
Am Psychol
.
2012
;
67
(
6
):
463
476
[PubMed]
10
Calear
AL
,
Christensen
H
.
Systematic review of school-based prevention and early intervention programs for depression.
J Adolesc
.
2010
;
33
(
3
):
429
438
[PubMed]
11
Faggiano
F
,
Minozzi
S
,
Versino
E
,
Buscemi
D
.
Universal school-based prevention for illicit drug use.
Cochrane Database Syst Rev
.
2014
;(
12
):
CD003020
12
Foxcroft
DR
,
Tsertsvadze
A
.
Universal school-based prevention programs for alcohol misuse in young people.
Cochrane Database Syst Rev
.
2011
;(
5
):
CD009113
13
Rotheram-Borus
MJ
,
Ingram
BL
,
Swendeman
D
,
Flannery
D
.
Common principles embedded in effective adolescent HIV prevention programs.
AIDS Behav
.
2009
;
13
(
3
):
387
398
[PubMed]
14
Cousins
JB
,
Aubry
TD
,
Fowler
HS
,
Smith
M
.
Using key component profiles for the evaluation of program implementation in intensive mental health case management.
Eval Program Plann
.
2004
;
27
(
1
):
1
23
15
Murray
LK
,
Dorsey
S
,
Haroz
E
, et al
.
A common elements treatment approach for adult mental health problems in low- and middle-income countries.
Cognit Behav Pract
.
2014
;
21
(
2
):
111
123
[PubMed]
16
Boustani
MM
,
Frazier
SL
,
Becker
KD
, et al
.
Common elements of adolescent prevention programs: minimizing burden while maximizing reach.
Adm Policy Ment Health
.
2015
;
42
(
2
):
209
219
[PubMed]
17
PracticeWise
.
Psychosocial and Combined Treatments Coding Manual
.
Satellite Beach, FL
:
PracticeWise
;
2009
18
Graves
KD
.
Social cognitive theory and cancer patients’ quality of life: a meta-analysis of psychosocial intervention components.
Health Psychol
.
2003
;
22
(
2
):
210
219
[PubMed]
19
Brown
FL
,
de Graaff
AM
,
Annan
J
,
Betancourt
TS
.
Annual Research Review: breaking cycles of violence - a systematic review and common practice elements analysis of psychosocial interventions for children and youth affected by armed conflict.
J Child Psychol Psychiatry
.
2017
;
58
(
4
):
507
524
[PubMed]
20
Caria
MP
,
Faggiano
F
,
Bellocco
R
,
Galanti
MR
.
The influence of socioeconomic environment on the effectiveness of alcohol prevention among European students: a cluster randomized controlled trial.
BMC Public Health
.
2011
;
11
(
1
):
312
[PubMed]
21
Roberts
C
,
Williams
R
,
Kane
R
, et al
.
Impact of a mental health promotion program on substance use in young adolescents.
Adv Ment Health
.
2011
;
10
(
1
):
72
82
22
Sibinga
EM
,
Webb
L
,
Ghazarian
SR
,
Ellen
JM
.
School-based mindfulness instruction: an RCT.
Pediatrics
.
2016
;
137
(
1
):
e20152532
[PubMed]
23
Bluth
K
,
Gaylord
SA
,
Campo
RA
,
Mullarkey
MC
,
Hobbs
L
.
Making friends with yourself: a mixed methods pilot study of a mindful self-compassion program for adolescents.
Mindfulness (N Y)
.
2016
;
7
(
2
):
479
492
[PubMed]
24
Spoth
R
,
Trudeau
L
,
Shin
C
,
Redmond
C
.
Long-term effects of universal preventive interventions on prescription drug misuse.
Addiction
.
2008
;
103
(
7
):
1160
1168
[PubMed]
25
Singla
DR
,
Kohrt
BA
,
Murray
LK
,
Anand
A
,
Chorpita
BF
,
Patel
V
.
Psychological treatments for the world: lessons from low- and middle-income countries.
Annu Rev Clin Psychol
.
2017
;
13
:
149
181
[PubMed]
26
Wilson
DB
. Practical meta-analysis effect size calculator [online calculator]. Available at: https:/www.campbellcollaboration.org/research-resources/research-for-resources/effect-size-calculator.html. Accessed March 1, 2019
27
Tanner-Smith
EE
,
Tipton
E
.
Robust variance estimation with dependent effect sizes: practical considerations including a software tutorial in Stata and spss.
Res Synth Methods
.
2014
;
5
(
1
):
13
30
[PubMed]
28
Cohen
J
.
Statistical Power Analysis for the Behavioral Sciences
.
Abingdon, United Kingdom
:
Routledge
;
2013
29
Branje
SJ
,
Hale
WW
 III
,
Frijns
T
,
Meeus
WH
.
Longitudinal associations between perceived parent-child relationship quality and depressive symptoms in adolescence.
J Abnorm Child Psychol
.
2010
;
38
(
6
):
751
763
[PubMed]
30
Hair
EC
,
Moore
KA
,
Garrett
SB
,
Ling
T
,
Cleveland
K
.
The continued importance of quality parent-adolescent relationships during late adolescence.
J Res Adolesc
.
2008
;
18
(
1
):
187
200
31
Majd Ara
E
,
Talepasand
S
,
Rezaei
AM
.
A structural model of depression based on interpersonal relationships: the mediating role of coping strategies and loneliness.
Noro Psikiyatri Arsivi
.
2017
;
54
(
2
):
125
130
[PubMed]
32
Caria
MP
,
Faggiano
F
,
Bellocco
R
,
Galanti
MR
;
EU-Dap Study Group
.
Effects of a school-based prevention program on European adolescents’ patterns of alcohol use.
J Adolesc Health
.
2011
;
48
(
2
):
182
188
[PubMed]
33
Kärnä
A
,
Voeten
M
,
Little
TD
,
Poskiparta
E
,
Kaljonen
A
,
Salmivalli
C
.
A large-scale evaluation of the KiVa antibullying program: grades 4-6.
Child Dev
.
2011
;
82
(
1
):
311
330
[PubMed]
34
Rose
K
,
Hawes
DJ
,
Hunt
CJ
.
Randomized controlled trial of a friendship skills intervention on adolescent depressive symptoms.
J Consult Clin Psychol
.
2014
;
82
(
3
):
510
520
[PubMed]
35
Stallard
P
,
Sayal
K
,
Phillips
R
, et al
.
Classroom based cognitive behavioural therapy in reducing symptoms of depression in high risk adolescents: pragmatic cluster randomised controlled trial.
BMJ
.
2012
;
345
:
e6058
[PubMed]
36
Espelage
DL
,
Low
S
,
Van Ryzin
MJ
,
Polanin
JR
.
Clinical trial of second step middle school program: impact on bullying, cyberbullying, homophobic teasing, and sexual harassment perpetration.
School Psych Rev
.
2015
;
44
(
4
):
464
479
37
Horn
AB
,
Pössel
P
,
Hautzinger
M
.
Promoting adaptive emotion regulation and coping in adolescence: a school-based programme.
J Health Psychol
.
2011
;
16
(
2
):
258
273
[PubMed]
38
Kehoe
CE
,
Havighurst
SS
,
Harley
AE
.
Tuning in to teens: improving parent emotion socialization to reduce youth internalizing difficulties.
Soc Dev
.
2014
;
23
(
2
):
413
431
39
Noggle
JJ
,
Steiner
NJ
,
Minami
T
,
Khalsa
SB
.
Benefits of yoga for psychosocial well-being in a US high school curriculum: a preliminary randomized controlled trial.
J Dev Behav Pediatr
.
2012
;
33
(
3
):
193
201
[PubMed]
40
Gabrhelik
R
,
Duncan
A
,
Miovsky
M
,
Furr-Holden
CD
,
Stastna
L
,
Jurystova
L
.
“Unplugged”: a school-based randomized control trial to prevent and reduce adolescent substance use in the Czech Republic.
Drug Alcohol Depend
.
2012
;
124
(
1–2
):
79
87
41
Miovský
M
,
Vonkova
H
,
Čablová
L
,
Gabrhelík
R
.
Cannabis use in children with individualized risk profiles: predicting the effect of universal prevention intervention.
Addict Behav
.
2015
;
50
:
110
116
[PubMed]
42
Spoth
R
,
Redmond
C
,
Shin
C
,
Greenberg
M
,
Feinberg
M
,
Schainker
L
.
PROSPER community-university partnership delivery system effects on substance misuse through 6 1/2 years past baseline from a cluster randomized controlled intervention trial.
Prev Med
.
2013
;
56
(
3–4
):
190
196
[PubMed]
43
Kupersmidt
JB
,
Scull
TM
,
Benson
JW
.
Improving media message interpretation processing skills to promote healthy decision making about substance use: the effects of the middle school media ready curriculum.
J Health Commun
.
2012
;
17
(
5
):
546
563
[PubMed]
44
Longshore
D
,
Ghosh-Dastidar
B
,
Ellickson
PL
.
National Youth Anti-Drug Media Campaign and school-based drug prevention: evidence for a synergistic effect in ALERT Plus.
Addict Behav
.
2006
;
31
(
3
):
496
508
[PubMed]
45
Perry
CL
,
Komro
KA
,
Veblen-Mortenson
S
, et al
.
A randomized controlled trial of the middle and junior high school D.A.R.E. and D.A.R.E. Plus programs.
Arch Pediatr Adolesc Med
.
2003
;
157
(
2
):
178
184
[PubMed]
46
Ichiyama
MA
,
Fairlie
AM
,
Wood
MD
, et al
.
A randomized trial of a parent-based intervention on drinking behavior among incoming college freshmen.
J Stud Alcohol Drugs Suppl
.
2009
;(
16
):
67
76
47
LaBrie
JW
,
Earle
AM
,
Boyle
SC
, et al
.
A parent-based intervention reduces heavy episodic drinking among first-year college students.
Psychol Addict Behav
.
2016
;
30
(
5
):
523
535
[PubMed]
48
Midford
R
,
Mitchell
J
,
Lester
L
, et al
.
Preventing alcohol harm: early results from a cluster randomised, controlled trial in Victoria, Australia of comprehensive harm minimisation school drug education.
Int J Drug Policy
.
2014
;
25
(
1
):
142
150
[PubMed]
49
Newton
NC
,
Andrews
G
,
Champion
KE
,
Teesson
M
.
Universal Internet-based prevention for alcohol and cannabis use reduces truancy, psychological distress and moral disengagement: a cluster randomised controlled trial.
Prev Med
.
2014
;
65
:
109
115
[PubMed]
50
Vogl
L
,
Teesson
M
,
Andrews
G
,
Bird
K
,
Steadman
B
,
Dillon
P
.
A computerized harm minimization prevention program for alcohol misuse and related harms: randomized controlled trial.
Addiction
.
2009
;
104
(
4
):
564
575
[PubMed]
51
Jander
A
,
Crutzen
R
,
Mercken
L
,
Candel
M
,
de Vries
H
.
Effects of a Web-based computer-tailored game to reduce binge drinking among Dutch adolescents: a cluster randomized controlled trial.
J Med Internet Res
.
2016
;
18
(
2
):
e29
[PubMed]
52
Jewkes
R
,
Nduna
M
,
Levin
J
, et al
.
Impact of stepping stones on incidence of HIV and HSV-2 and sexual behaviour in rural South Africa: cluster randomised controlled trial.
BMJ
.
2008
;
337
:
a506
[PubMed]
53
Messman-Moore
TL
,
Walsh
KL
,
DiLillo
D
.
Emotion dysregulation and risky sexual behavior in revictimization.
Child Abuse Negl
.
2010
;
34
(
12
):
967
976
[PubMed]
54
Pillai
VK
,
Barton
TR
.
Sexual activity among Zambian female teenagers: the role of interpersonal skills.
Adolescence
.
1999
;
34
(
134
):
381
387
[PubMed]
55
Hawkins
J
,
Madden
K
,
Fletcher
A
, et al
.
Development of a framework for the co-production and prototyping of public health interventions.
BMC Public Health
.
2017
;
17
(
1
):
689
[PubMed]
56
Kaminski
JW
,
Valle
LA
,
Filene
JH
,
Boyle
CL
.
A meta-analytic review of components associated with parent training program effectiveness.
J Abnorm Child Psychol
.
2008
;
36
(
4
):
567
589
[PubMed]
57
de Vries
SL
,
Hoeve
M
,
Assink
M
,
Stams
GJ
,
Asscher
JJ
.
Practitioner review: effective ingredients of prevention programs for youth at risk of persistent juvenile delinquency–recommendations for clinical practice.
J Child Psychol Psychiatry
.
2015
;
56
(
2
):
108
121
[PubMed]

Competing Interests

POTENTIAL CONFLICT OF INTEREST: The authors have indicated they have no potential conflicts of interest to disclose.

FINANCIAL DISCLOSURE: The authors have indicated they have no financial relationships relevant to this article to disclose.