Predicting Recovery After Concussion in Pediatric Patients: A Meta-Analysis

The recommendations of the Cochrane Prognosis Methods Group were followed.^13,18  This manuscript adhered to the Transparent Reporting of a Multivariable Prediction Model for Individual Prognosis or Diagnosis for Systematic Reviews and Meta-Analyses guideline.¹⁷  This protocol was registered with the International Prospective Register of Systematic Reviews (CRD42022357503).¹⁹  No protocol deviations occurred. One amendment was made; the earliest publication date was not initially limited. The publication date filter was set to January 1, 1990 to November 21, 2022. This start date was chosen given the lack of a consensus-based definition of mTBI before 1993.²⁰  No automation tools, outside of software identifying duplicate references (EndNote, Covidence), were used.

We operationalized our key question through the PICOTS framework, adapted for PPMs: Population, Index model, Comparator model, Outcome(s), Timing, and Setting.¹³  Population was defined as follows: individuals with a history of mTBI before 18 years of age.^12,13  Potential mTBI definitions included, but were not limited to, those provided by the World Health Organization, the Centers for Disease Control and Prevention, the American Congress of Rehabilitation Medicine, and the Concussion in Sport Group (CiSG).^3,21–23  No specific models (index and comparator) were required. The PPM had to determine a patient’s risk of delayed recovery from mTBI (outcome). This delayed recovery was defined as the presence of at least 1 of the 5 most commonly noted symptoms, including headaches, memory deficits, concentration difficulties, imbalance, and dizziness.²⁴  The model’s prediction horizon (timing) was restricted to at least 28 days (consistent with most major definitions of PPCS), but no more than 1 year postinjury (chosen based on major guidelines).^3,22  The moment of prediction (timing) had to be before the 28th day postinjury. The setting was unrestricted.

Databases were searched using controlled subject heading vocabulary and keywords, informed by the pertinent literature, for 3 concepts combined, as follows: (1) mTBI; (2) prognostic model; and (3) persistent postconcussive symptoms (Supplemental Information).^24–26  Searches were refined through repeated testing with discussions between a systematic review expert (ASH) and a subject matter expert (JMW). Literature searches of Ovid Medline, Embase, Ovid PsycInfo, Web of Science Core Collection, Cumulative Index to Nursing and Allied Health Literature, Cochrane Library, and Google Scholar were run on November 21, 2022. Complete references were imported into EndNote 21.²⁷  Duplicates were automatically removed.

References were uploaded to Covidence, an online systematic review management system, and screened.²⁸  Two reviewers independently screened titles and abstracts, with a third independent reviewer resolving disagreements (ACL, ARK, ASH, JMW, KASY, LAB). Additional duplicates were automatically and manually removed within Covidence.

Full-text articles were independently reviewed for eligibility by 2 reviewers, with a third independent reviewer resolving disagreements (ACL, ARK, ASH, JMW, KASY, LAB). Studies had to meet all PICOTS criteria for inclusion. Additionally, study authors had to examine PPMs with at least 2 predictors (ie, a multivariable model), studies had to have at least 30 participants, and authors had to report model performance (eg, discrimination).¹³  Only peer-reviewed original research was included. Per guidelines, only cohort, nested case-control, and case cohort studies were included.²⁹  Only full-text records in English were included, given this study team’s language limitations. Reference mining did not identify additional studies.

A publicly available template that combines the Critical Appraisal and Data Extraction for Systematic Reviews of Prediction Modeling Studies and Prediction Model Risk of Bias Assessment Tool (PROBAST) checklists was completed (Supplemental Information).^29–31  One reviewer (JMW) independently extracted the data from the included studies. A second independent reviewer (ASH) checked the data extraction for completion and accuracy. Disagreements were resolved through the consensus of a third reviewer (LAB), with discussions as needed. Two reviewers independently assessed included articles for RoB (ASH, JMW, KASY). RoB was determined for each PROBAST domain (participants, predictors, outcome, analysis), which collectively determined overall RoB. A domain was “low RoB” if all domain question answers were “Yes” or “Probably Yes.” A domain was “high RoB” if at least 1 domain question answer was “No” or “Probably No.” Otherwise, a domain was “Unclear.” Per PROBAST, the overall RoB was “low” if all domains had low RoB, “high” if at least 1 domain had high RoB, and “Unclear” otherwise. Disagreements were resolved through a consensus discussion. For this review, sufficient data were available in published records; thus, no study investigators were contacted. Some data were restored through basic arithmetic (eg, percentage female of population).

Although the latest guideline advises meta-analyses if at least 5 external validation studies are available, a previous guideline informed this review’s design, which did not include this recommendation.^13,16  Meta-analyses were performed when the authors of at least 3 studies examined the same PPM.¹³  For each study, the area under the receiving operating characteristic curve (AUC) was extracted. AUC is an effect size that quantifies the ability of a measure to distinguish between 2 separate groups.³²  The logit of each AUC was calculated, as was the variance of each transformed value.¹⁶  An aggregate AUC, a PPM’s pooled ability to accurately predict delayed recovery from mTBI, was estimated using random-effects meta-analysis with effect sizes weighted by precision. The analysis was performed in R 4.2.1 using the Metafor 4.2-0 package.^33,34  Between-study heterogeneity was assessed using I², the percentage of variation attributable to differences across studies rather than chance, with values interpreted according to guidelines.³⁵  The potential presence of publication bias was assessed using a random-effects Egger’s test and through funnel plot visualization.³⁶  All the literature was descriptively synthesized by comparing study characteristics (eg, design), model characteristics (eg, modeling method), predictors, predicted outcomes, risk scores, and performance.³⁷ 

A modification of the Grading of Recommendations Assessment, Development, and Evaluation (GRADE) approach for prognostic factor studies was used to assess the quality of the evidence.^13,18,38  The following GRADE factors were assessed: study limitations; inconsistency; indirectness; imprecision; publication bias; moderate or large effect size; and dose effect. The data guiding GRADE recommendations included the following: number of events (PPCS); events per (candidate) variable (EPV) or events per (candidate) predictor; number of studies; number of cohorts; and validation(s) performed. For this work, model validation replaced the phase of investigation GRADE factor. The overall RoB directly informed the study’ limitations. The definitions of inconsistency (heterogeneity in results across studies), indirectness (equivalent to the applicability of PROBAST), and publication bias were unchanged.³⁸  Imprecision was a combination of precision of results or meta-analyses, number of studies, and power, which was determined to be adequate if the EPV was at least 20 in development studies or at least 100 events were present in validation studies. An AUC of at least 0.7 was the definition of a moderate or large effect size.^13,39  A dose effect was present if a previous mTBI increased the risk of delayed recovery.

Of the 17 433 references identified, 78 full-text articles were reviewed (Fig 1). Most were excluded (Supplemental Information) due to not being original research (n = 29; 37%) or peer-reviewed (n = 23; 30%). Data were extracted from the 6 included studies (Supplemental Information) and synthesized (Tables 1–6, Supplemental Tables 7–9).^40–45 

Studies were published in 2014 through 2022, with participant enrollment between October 2010 and May 2019 (Table 1). Study funding and conflicts of interest were collated in Supplemental Table 7. Most studies were prospective cohorts (n = 4), in EDs (n = 4), and within the United States (n = 4). The average ages of the populations were 11.1 to 14.9 years, with populations ranging from 27% to 51% female. Half of the studies’ authors reported the average time since injury of the analyzed populations, ranging from 2.8 hours to 5.6 days. The most common mechanism of injury in most studies was sport (n = 5).

Thirteen PPMs were examined, as follows: the Predicting and Preventing Postconcussive Problems in Pediatrics (5P) clinical risk score, the Buffalo Concussion Physical Examination risk for delayed recovery (BCPE RDR) score, an unnamed model by Grubenhoff et al, and 10 unnamed models by Bressan et al.^40–45  Half of the studies’ authors examined the 5P.^43–45  Outside of Bressan et al, no study authors examined multiple models.

Most studies used a CiSG consensus statement for the definition of concussion (n = 4; 67%; Table 2). Logistic regression was the most common modeling method (n = 5; 83%). The authors of one study compared multiple methods, including the Cox Proportional Hazard, Accelerated Failure Time, and Binomial Generalized Linear Modeling (bGLM).⁴²  Sample sizes ranged from 125 to 2006, with number of cases ranging from 22 to 510 or 17.6% to 57.8%. Two of the 3 validation studies included >100 events.^43,45  The number of candidate predictors ranged from 4 to 46, with the number of final predictors ranging from 4 to 14.^40,41,45  EPV ranged from 2.4 to 18.5.^40,44,45  In the prospective studies, the percentage of participants with missing data was 4.9% to 42.4%.^{40–42,44,45}  All studies performed complete-case analysis.

The selected predictors (Table 3) can be grouped into 3 domains, including clinical history (eg, patient age), clinical examination (eg, tandem gait), and symptoms (eg, headache). Only the 5P incorporated all 3.⁴⁵  BCPE RDR excluded symptoms.⁴²  Bressan et al was unique in that instead of combining all predictors into a single model, selected components from each version of the third iteration of the Sport Concussion Assessment Tool (Sport Concussion Assessment Tool, Child Version of Sport Concussion Assessment Tool) were independently modeled.⁴⁰  None of these models included history.⁴⁰  The Grubenhoff et al model included only symptoms.⁴¹  Examination and symptom predictors were the most commonly used domains. The 5P and 4 of the Bressan et al models included the Modified Balance Error Scoring System and a cognitive assessment score.^46,47  Of the models with symptom predictors, the 5P, 6 of the Bressan et al models, and the Grubenhoff et al model included at least headache, sensitivity to noise, and fatigue (Supplemental Table 8), which are the most common predictors. Clinical history predictors were included in the 5P and BCPE RDR models, although neither model contained the same ones. The BCPE RDR was the only model that included a composite predictor.⁴² 

All models included a binary outcome (yes/no) for PPCS or an equivalent; however, the operational definitions varied (Table 4). Half of the studies included additional outcomes, including delayed symptom recovery and time to symptom resolution in days.^41–43  Most studies used a version of the International Classification of Diseases, 10^th Revision (ICD-10) definition of PPCS as their framework.^40,41,44,45  Root et al added additional criteria.^44,48,49  Howell et al cited the CiSG definition plus additional criteria.^43,50–53  Haider et al cited a systematic review of criteria used to define recovery from sport-related concussion.⁵⁴ 

The measures used to support outcomes varied. Half of the studies used the Post-Concussion Symptom Inventory, collected within 48 hours to potentially 46 days after injury.^40,44,45  The remaining studies used a modified Concussion Symptom Inventory, clinician judgment (described more below), and the Post-Concussion Symptom Scale.^41–43  Most studies (n = 4; 67%) collected these data at or near enrollment and at follow-up.^40,41,43,44 

All studies required postinjury symptoms to be worse than preinjury symptoms to be classified as PPCS, but the burden varied. The least stringent definition was Howell et al, requiring a Post-Concussion Symptom Scale score of at least 1, while only rating “…those symptoms that started at the time of injury and that they had experienced within the prior 24 hours of the clinical examination.”⁴³  The number of required symptoms varied from any 1 symptom to at least 3 symptoms. The required symptoms varied from any in the aforementioned measures to a specific list.⁴¹  The required minimum difference in symptom measurement scores from preinjury to follow-up ranged from 1 to 6.⁴⁴  The authors of one study required additional criteria to determine if an individual had recovered, including a normal physical examination, the ability to tolerate exercise without exacerbation of symptoms, a successful return to school, and, if an athlete, the demonstration of good exercise tolerance.⁴²  Criteria were the same across all participants in each study, except for Haider et al (athlete criterion) and Root et al (cutoff scores varied based on age).^42,44 

The prediction horizon was roughly 1 month for all studies. The moment of prediction ranged from within 6 hours of injury to within 14 days of injury. Most studies (n = 4; 67%) developed models from patients presenting within 48 hours or less after injury, all of which were ED studies. The 2 outpatient studies required patients to present within 10 to 14 days.^42,43 

The 5P and the BCPE RDR models reported scores with corresponding risk categories (Supplemental Table 9).^42,45  Each 5P risk score also corresponded to a risk percentage with a 95% confidence interval (CI). Only the low-risk categories overlapped between these 2 models. Sensitivity analyses were reported for 1 model, with cut points ranging from 8 to 14 (total score 14), and found that a cut point of 10 was statistically significant in identifying PCS.⁴¹  The authors of one study noted that none of their models were predictive and described how to use them clinically.⁴⁰ 

Most PPMs were validated (n = 5; 83.3%).^40,42–44  The PPMs by Bressan et al and the BCPE RDR were only internally validated.^40,42  One study did not report any validation.⁴¹  The most robust study was Zemek et al, in which the 5P clinical risk score was developed, internally validated, and externally (temporally) validated.⁴⁵  Bootstrapping was the most commonly used internal validation method (n = 2; 67%).^40,45  Two independent external validations (the most common external validation method; n = 2; 67%) of the 5P were conducted.^43,44  One was conducted in a novel setting (concussion clinic) with a broader moment of prediction (within 14 days of injury).⁴³  The 5P clinical risk score was the most extensively validated model.^43–45 

All studies revealed a high RoB, predominantly due to analysis concerns in the PROBAST (Table 5).³¹  The most commonly raised concerns were determining outcomes with knowledge of predictor information, an insufficient number of participants with the outcome, transforming continuous and categorical predictors, complete-case analyses, and a lack of additional considerations for data complexities.³¹  All studies revealed a low RoB in the predictors domain. Almost all studies (n = 5; 83%) revealed a low RoB in the participants domain.^{40–42,44,45}  The one study revealing a high RoB in this domain was due to its retrospective design.⁴³  No concerns regarding applicability were raised.

The authors of all studies reported discrimination via c-statistics. Most studies reported AUCs and graphs (n = 5; 83%). The authors of only 2 studies reported calibration via plots, slope, and Hosmer–Lemeshow tests.^42,45  Of all the studies, Zemek et al reported the most model performance measures (all of the aforementioned).⁴⁵  Most studies (n = 5; 83%) revealed statistically significant model performance.^41–45  The authors of 2 studies compared the 5P with physician prediction; the authors of one reported no statistically significant increase in AUC, whereas the authors of the other demonstrated a statistically significant improvement.^44,45  The study comparing multiple modeling methods revealed bGLM to be superior.⁴² 

The aggregate AUC of the 3 external validations of 5P was 0.69 (95% CI 0.65–0.72), a small to medium effect size (Fig 2).^39,43–45  Although no heterogeneity existed among these 3 effect sizes (Fig 3; I² = 0.00%, 95% CI 0.00% to 97.4%; Egger’s test z = 0.51, P = .609), this result should be considered preliminary. This small number precluded additional analyses.

The overall GRADE quality of evidence was strongest for 5P (Table 6). It was the only externally validated model, and a dose effect was present. Its meta-analysis revealed low heterogeneity (inconsistency); however, only 3 underpowered (imprecision), high-RoB (study limitations) studies were present. The aggregate AUC did not reach 0.7, but its CI included it (moderate or large effect size, imprecision). Publication bias was assumed to be present with the few studies and lack of negative findings. Overall, the 5P GRADE quality was low (“limited confidence”).^43–45,55 

The remaining models were of very low GRADE quality (“very little confidence”). The BCPE RDR model lacked external validation and revealed study limitations, unclear inconsistency, and limitations in imprecision and publication bias, which were ameliorated by a moderate or large effect size and dose effect.⁴⁰  The Bressan et al and Grubenhoff et al models had limited (if any) validation, study limitations, imprecision, unclear inconsistency, assumed publication bias, low AUCs, and an unclear dose effect.^40,41  No model revealed serious limitations in indirectness.

The following PPMs estimate the risk for delayed recovery after mTBI in pediatric patients: the 5P clinical risk score, the BCPE RDR, Grubenhoff et al’s model, and Bressan et al’s models.^40–45  Only the former 3 models revealed better than chance predictions, none above low GRADE quality.

All studies revealed high RoB. The inclusion of symptoms both as predictors and as the outcome raises the potential for RoB. One way to summarize a theme in PROBAST is that low RoB can be achieved when predictors and the outcome are independently and objectively determined (eg, biomarkers predicting death). Collecting symptoms with the same measure at multiple timepoints, including obtaining predictors (symptoms at presentation) and a component of the outcome (preinjury symptoms) at the same timepoint (at enrollment), potentially undermines the validity of determining the outcome without knowledge of predictor information (PROBAST Signaling Question 3.5).⁵⁵  The PROBAST analysis issues may be avoided with guidance from the Cochrane Prognosis Methods Group.^13,18  Some of these analytical concerns are broad, such as advising against transforming variables, analyzing all enrolled patients, and handling missing data appropriately.^55–58  PPM study design considerations include achieving a minimum EPV (20) or number of events (100) and, potentially, additional data handling considerations (eg, participant censoring).⁵⁵  Although none of these development studies reached this EPV, this guideline was published after half of the included studies. Half of the development studies met the previous minimum EPV (10).^40,45,55 

Although no PROBAST applicability concerns were raised, the settings and definitions dictate appropriate use cases and model interpretations. Of pediatric patients with mTBI seeking care, most seek care in outpatient settings; however, most of these studies were performed in EDs.¹  The limited demographic data (age and sex) available for these study populations align with those of existing studies and epidemiologic data (predominantly male participants and at least in late childhood), enhancing the generalizability of these models; however, these studies may overlook populations that suffer from health care inequalities, especially with most of these studies being conducted in the United States.^59–62  The authors of future studies should focus on these health disparities.⁶³ 

Regarding definitions, the frequent use of the CiSG definition of concussion and the ICD-10 definition of PPCS enhanced generalizability; however, the variable operationalization of PPCS undermined this quality. Although the PPCS rates varied, they remained aligned with the existing literature.^5–9  One PPCS definition shares similarities with the latest American Congress of Rehabilitation Medicine mTBI diagnostic criteria (eg, physical examination elements), potentially enhancing the future generalizability of the BCPE RDR.^21,42  The use of a binary outcome limits these PPMs to solely estimating the presence or absence of PPCS after a certain duration. The authors of 2 studies enhanced the predictive capabilities of their examined models by also predicting the number of days until recovery.^42,43 

The lack of external validation studies precluded meta-analyses for most models. This preliminary meta-analysis of the 5P supports its use. Although no study directly compared distinct models, it still may be worth noting that the Bressan et al and Grubenhoff et al models performed the poorest and solely relied on a single domain of predictors. Perhaps this correlation indicates the need for incorporating multiple domains to accurately predict PPCS. Regardless of performance, the reporting of such could be improved, particularly for calibration, overall performance, and clinical utility.^13,18  Model output and risk interpretation were variable, suggesting a potential gap in guidelines, particularly if risk estimates diverge.

Of the models that were predictive, the highest GRADE quality was low. This model (5P) had the most evidence and potentially improved estimations over physicians’ predictions. However, several factors reduced the quality. The evidence supporting its use in an outpatient setting is limited.⁴³  The independent external validation in an ED did not have at least 100 events.⁴⁴  External validation by the group that developed a model is considered less strong than fully independent external validations with different investigators.^45,58  Regardless of these shortcomings, with patients 5 to 18 years of age meeting CiSG criteria for concussion and presenting to an ED within 48 hours of injury or to a concussion clinic within 10 days of injury, clinicians may consider using the 5P model to estimate the risk of PPCS.

This work was limited by the language proficiency of this group; however, 17 433 initial references were still identified. Reviewing this body of literature was inefficient, even with publication year restrictions. Despite following existing search strategy examples, only 6 studies (0.03% of the initial references) were included.^24–26  With the ongoing maturation of PPM science and, potentially, a consensus-based definition of PPCS, these search strategies may be refined, decreasing potential search inaccuracies.^3,17 

All studies revealed high RoB, limiting confidence in the findings. The studies were largely limited to the CiSG definition of mTBI, and the authors did not examine any other consensus-based definitions. The lack of a consensus-based, operational definition for PPCS limited the ability to indirectly compare models. These studies primarily reflect male patients in late childhood or early adolescence presenting to an ED. Data on the PPCS subpopulations were lacking, which could alter confidence in findings, clarify appropriate use cases, or identify variable model performance in subpopulations.^4,6  Despite evidence suggesting that the modeling method influences PPM performance, the authors of most studies examined only 1 modeling method.⁴²  Regardless, the performance of these PPMs outside of their moment of prediction and prediction horizon is unknown. No study authors directly compared PPMs nor evaluated clinical utility (eg, decision curve analysis). Lastly, no formal GRADE process for PPMs exists.

PPMs enhance a clinician’s ability to identify pediatric patients at risk for persistent symptoms soon after a concussion. For patients 5 to 18 years of age meeting the CiSG criteria for a concussion and presenting to an ED within 48 hours of injury or to a concussion clinic within 10 days of injury, clinicians may consider using the 5P model to estimate this risk and guide treatment. However, external validation studies, particularly in non-ED settings, as well as implementation strategies, are warranted. Moreover, issues related to the varied definitions of PPCS may limit further progress in terms of PPM development. Work to achieve consensus regarding operationalized PPCS criteria is recommended.

All data are available.

This material is the result of work supported by resources from the Rocky Mountain Mental Illness Research Education and Clinical Center and the University of Colorado Anschutz Medical Campus Strauss Health Sciences Library. This work does not necessarily represent the views of the US Department of Veterans Affairs or the US Government.

5P

Predicting and Preventing Postconcussive Problems in Pediatrics

AUC

area under the receiving operating characteristic curve

BCPE RDR

Buffalo Concussion Physical Examination risk for delayed recovery

bGLM

binomial generalized linear modeling

CI

confidence interval

CiSG

Concussion in Sport Group

ED

emergency department

EPV

events per variable

GRADE

Grading of Recommendations, Assessment, Development, and Evaluations

ICD-10

International Classification of Diseases, 10^th Revision

mTBI

mild traumatic brain injury

PCS

postconcussive/postconcussion syndrome

PPCS

persistent/prolonged postconcussive/postconcussion symptoms

PPM

prognostic prediction model

PROBAST

Prediction Model Risk of Bias Assessment Tool

RoB

risk of bias