Hierarchy of Evidence Within the Medical Literature

Before delving more deeply into study designs involving human subjects, we would like to note the importance of basic and translational research studies. These studies many times serve as the foundation for clinical research and can help drive the formulation of hypotheses and ascertain the safety of certain drugs and devices before further study in humans. A detailed review of these studies is out of the scope of this article but can be found in a previous issue of Hospital Pediatrics.¹ 

As an overview for clinical research studies, we have included an organizational chart for clinical research study designs involving human subjects (Supplemental Fig 1). In the subsequent sections of this article, we will describe different study designs in the order of how they ascend the evidence hierarchy (Fig 1).

In observational studies, current practices are evaluated, and the researcher neither intervenes in nor changes the care being provided. Case series and case reports have no comparison group, whereas analytic studies, such as cross-sectional, case-control, and cohort studies, include a comparison group.

Case reports and case series describe a clinical event such as a diagnosis or treatment. Typically, case reports describe 4 or fewer patients, whereas case series describe >4 patients.²  In the midst of the coronavirus disease 2019 pandemic, Webb and Osburn published a case series on hospitalized patients with severe acute respiratory syndrome coronavirus 2 infection.³  These types of studies allow clinicians to share their experiences in caring for patients with rare diagnoses or unusual presentations. However, drawing conclusions from these study types to answer clinical questions is not advised because of potential biases resulting from the small number of patients described and the lack of comparison groups. These studies can help generate hypotheses for subsequent studies.

Cross-sectional studies are those in which population data are collected at a specific point in time, with simultaneous measurement of the exposure and outcome. This study design is often used for survey-based studies such as the one by Rassbach and Fiore in 2021 in which fellows were surveyed on research and career outcomes after graduation.⁴  This study found that 77% of fellowship graduates continued to conduct research after graduation, and that those who graduated with a master’s degree were significantly more likely to continue research after fellowship (91% vs 64%, P = .0001).

In case-control studies, 2 existing groups with or without a disease are compared to identify disease risk factors. Moss et al used this method to identify risk factors that contribute to hospital-acquired venous thromboembolism in adults who are admitted to pediatric hospitals.⁵  This type of study is typically quick and economical because there is no need to wait for a disease to develop; this also makes it useful for studying rare diseases as in this example of venous thromboembolism.

Retrospective and prospective cohort studies identify a specific patient population in which a subset of individuals has experienced a particular exposure, and compare the rates of disease development in exposed to unexposed individuals over time. In 2019, Desai et al retrospectively evaluated a cohort of infants with bacteremic urinary tract infections to determine the association between duration of intravenous antibiotics and rates of urinary tract infection recurrence resulting in presentation to the emergency department or hospitalization within 30 days of hospital discharge.⁶ 

RCTs are prospective studies that evaluate the effectiveness of interventions through comparison with placebo or the existing standard of care. Eligible participants are enrolled and randomly allocated to a treatment arm or comparison arm. Multicenter studies may use cluster randomization of entire groups of participants from hospitals or study sites to 1 arm. In some RCTs, participants and/or study personnel may be blinded to the assigned group. One recent investigator-blinded RCT evaluated the effect of interactive versus didactic asthma education on emergency department visits and hospitalizations.⁷  An accompanying commentary highlighted the need for more RCTs in the pediatric hospital medicine literature given their relative paucity.⁸ 

The RCT is considered the best study design to evaluate comparative interventions, and the only study type able to establish causation as opposed to association. Although observational studies are limited by the investigators’ abilities to adjust for unmeasured participant characteristics, a well-conducted RCT provides a way to reduce selection and confounding biases by leveraging the random allocation of participants to balance both observed and unobserved participant characteristics. Blinding is an additional strategy that has long been believed to improve the validity of a trial and 1 that is only possible in a prospective study. Additionally, in RCTs, the research team has greater control over the intervention under evaluation with regard to variables such as exposure amount and timing. The summative result of these study design strengths is a study with higher internal validity than can often be achieved with observational study methods.

SRs and MAs are considered to be the highest-quality study designs when conducted in a rigorous manner. The goal of SRs is to produce an answer to a clinical question on the basis of the synthesis of available evidence evaluated from multiple distinct studies. SRs can focus on any type of research question, whether related to diagnosis, treatment, prognosis, education, or quality improvement. It is important to note that SRs differ from narrative reviews, which seek to describe previously published works on a specific topic but usually without systematic methods to determine study eligibility, perform a comprehensive search strategy, and appraise studies for bias. This can potentially make narrative reviews susceptible to selection bias through omission of key studies or bias from emphasis of results produced by flawed methods.

MAs are 1 component of the synthesis phase of SRs. This quantitative statistical process combines numerical outcome data from multiple studies as if they were the result of a single larger study. The methods of SRs/MAs involve formulation of a clinical question, determination of study inclusion and exclusion criteria, a comprehensive literature search of published and unpublished studies, appraisal of bias in studies, determination of the quality of evidence for each outcome studied, and qualitative and/or quantitative synthesis of data. The quality of evidence for each outcome across studies is often determined using the Grading of Recommendations Assessment, Development and Evaluation criteria, which evaluates the initial quality according to the study design, and downgrades or upgrades the evidence level on the basis of several factors. Studies can be downgraded for risk of bias, inconsistency, indirectness, imprecision, or publication bias, and upgraded for large-effect or dose response.⁹ 

SRs/MAs potentially offer the highest-quality evidence because of the rigorous methods and broad utilization of study results, regardless of study size or publication status. Unfortunately, when the quality of included studies is low, further studies are often needed to address the clinical question. Furthermore, heterogeneity, either clinical or statistical, may plague SRs/MAs if not identified and addressed by researchers. Outcomes for studies with clinical heterogeneity, meaning differing patient populations, interventions, or outcomes, should not be quantitatively synthesized. Statistical heterogeneity, symbolized by I², measures whether the variability between study results is because of chance. Excessive heterogeneity should alert researchers to potential differences in design, population, or attributes of intervention between studies that is not because of chance. For example, a reanalysis of 2 previous MAs regarding the association between hypertonic saline and hospital length of stay in bronchiolitis patients was able to discern significant heterogeneity that, when accounting for those factors causing heterogeneity, came to a different conclusion than the original studies.¹⁰  These limitations and sources of error are important to keep in mind when creating or reviewing any SRs/MAs. Further, although out of the scope of this article, hospitalists may come across newer methods of evidence synthesis in the literature, such as individual patient data MAs and network MAs, and one may refer to these referenced resources to understand their unique advantages and limitations.^11,12 

There are important nuances of diagnostic testing articles that require consideration. Diagnostic testing research can take the form of various study designs depending on the stage of the research. Initial studies may be descriptive, where diagnostic test results are compared in patients with and without the disease to show that there is a difference in the result. Subsequent studies of diagnostic accuracy are commonly cross-sectional, where the investigational test and a reference standard are employed at 1 point in time to determine sensitivity, specificity, predictive values, and, ultimately, how well the test can distinguish between those with and without disease. Once tests are determined to be accurate, RCTs can be conducted to determine patient outcomes in those receiving the test versus a different test or no test at all. SRs/MAs can also provide a high level of evidence to answer diagnostic questions on the basis of a synthesis of available literature.

The evidence hierarchy should be considered while performing literature searches and to prioritize articles for critical appraisal during a busy clinical service. The ultimate quality of the evidence, however, ultimately depends on evaluation of the strengths and limitations of a particular study during the critical appraisal process. Various free tools exist on the Internet to aid clinicians in critical appraisal sorted by study design as offered through Critical Appraisal Skills Program and Centre for Evidence-Based Medicine.^13,14  Knowledge of the existing evidence base, the evidence hierarchy, and strengths and limitations of study designs should also be considered by researchers seeking to design studies to build upon current evidence for particular questions.

• The evidence hierarchy of study designs can be used to guide efficient literature searches and article appraisals for clinical questions.

• The availability of only lower-level observational study designs for clinical questions may prompt researchers to consider designing higher-level experimental designs if feasible and ethical to do so.

• Each study design inherently has strengths and weaknesses that should be kept in mind during critical appraisal of research articles and design of novel studies.