Essential Concepts for Reducing Bias in Observational Studies Free

Below and in Table 1 we describe techniques, including study design and analytic methods, that can be used to account for differences in populations in observational studies.

Randomization within RCTs balances many important clinical and sociodemographic variables across groups, thus simplifying the comparison of treatments on outcomes. Careful selection and incorporation of important clinical and sociodemographic variables are essential steps in ensuring internal validity within any study but are of utmost importance for observational studies, which are not randomized and often rely on retrospectively collected data collected for nonresearch purposes (eg, billing). In this case, important variables may not be available and surrogate (ie, an indirect or alternative measure that represents the preferred measure) or composite (ie, multiple discrete measures that combine to form a singular measure) measures may need to be used. The use of a composite measure is highlighted in the work of Krager et al, which leveraged the Child Opportunity Index (COI) 2.0, a composite measure of 29 individual indicators of neighborhood opportunity across 3 domains, to examine the influence of a child’s neighborhood environment on hospitalization rates for ambulatory care sensitive conditions.

One common and straightforward strategy to address population equivalence is based on the concept of restriction. Restriction involves the application of strict inclusion and exclusion criteria to develop “equivalent” populations for comparison. One example from the literature can be found in the work by Gill et al in which the authors used multiple exclusion criteria (eg, excluding children with immunodeficiency, oncologic processes, and competing diagnoses with high likelihood of corticosteroid administration such as asthma) to examine the use of corticosteroids in a cohort of children hospitalized with orbital cellulitis.⁴  Another example can be found in the work of Stephens et al that used the concept of restriction using All Patient Refined Diagnosis-Related Group (APR-DRG) to focus on patients within the lowest severity of illness (SOI) level.⁵  In this study, Stephens et al restricted the population to those in APR-DRG SOI level 1 to reduce the influence of SOI driving variation in laboratory testing practices across hospitals and conditions.

Stratification involves dividing a population into subgroups based on a characteristic, such as risk. In observational studies, stratification is generally applied in the analysis phase and consequently is a flexible and reversible approach to addressing equivalence. One important consideration to the use of stratification is whether there are sufficient patients per strata to appropriately power analyses. Stratification can be found in investigations such as that by Jeffries et al that explores racial and ethnic differences in severe pediatric unintentional injuries and that of Congdon et al that explores the impact of race and ethnicity on the management of pediatric gastroenteritis within a quality improvement framework.^6,7  In both of these works, stratifying the population followed by regression modeling (discussed next) allowed for an examination of differences in hospitalization rates based on a demographic characteristic.

To address the complexity of research performed in real-world settings, robust multivariable regression models adjusting for (ie, accounting for) important covariates are frequently used to improve the interpretation of observational study outcomes. Multivariable regression models examine the relationship between 1 outcome variable and multiple predictor variables. Commonly encountered types of multivariable regression in the literature include linear regression, logistic regression, and Cox proportional hazards regression. Consultation with a statistician can aid in choosing the type of regression analysis to perform.

Regardless of what type of multivariable regression analysis is chosen, an essential component to reduce bias in observational research using multivariable regression involves choosing confounding variables for adjustment. Confounders are variables that influence both the predictor and outcome variable contributing to a false association of the 2. Utilizing causal diagrams or models of how variables interact can assist with determining which variables to adjust for within models. Additionally, consulting a statistician at this phase can be invaluable to determining the number and types of variables to include within models to improve model fit while avoiding “overadjustment” or over selection of variables to be adjusted for in models.

Examples of statistical adjustments within research are numerous. One example of the use of statistical adjustments can be found in the study by Thomson et al of hospital outcomes including acute respiratory failure, ICU transfer, and LOS based on antibiotic exposure in neurologically impaired children with aspiration pneumonia.⁸  In this study, the research team used statistical adjustment to account for confounding related to illness severity and medical complexity. Another example can be found in the work by Kaiser et al, which used statistical adjustments for patient characteristics within multilevel models examining asthma pathway implementation across a national sample of hospitals and subsequent rates of early systemic corticosteroid administration, triage assessments, chest radiography use, hospital admission and transfer practices, and LOS.⁹ 

Propensity scoring is a statistical technique to assess a patient’s probability or likelihood of receiving a specific exposure. Propensity scores are generated for each patient¹⁰  based on the combination of important patient and clinical characteristics that are potentially associated with both exposure and outcome and can be used in multiple ways, including covariate adjustment, inverse probability of treatment weighting, or matching.¹¹  For example, in propensity score matching, each patient is assigned a propensity score (ie, the probability that they received 1 of the treatments being studied), and patients from the treatment groups are “matched” based on the similarity of their scores. When treatment groups are “well matched”, then the important characteristics used in the model to derive propensity scores tend to be balanced, mimicking randomization that occurs within randomized controlled trials. The ultimate goal of this process is to ensure equivalence between groups being compared. Once patients have been matched, modeling can be performed to examine the impact of the intervention on the outcome. Recent examples of propensity scoring can be found in the work of Lipshaw et al, exploring the use of antibiotics for children presenting to the emergency department for suspected community-acquired pneumonia and treatment failure (ie, readmissions, antibiotic changes) and in Parikh et al’s study comparing the effectiveness of dexamethasone versus prednisone in asthma management on LOS, readmissions, ICU transfers, and costs.^12,13  Compared with multivariable regression, propensity methods can achieve better balance of potential confounders and can incorporate more variables when determining propensity scores. Choosing between these methods can be complex and should be made in collaboration with a statistician.

Within the next section and in Table 2 we explore examples of important covariates to consider when conducting observational studies.

Severity of illness (SOI) can be difficult to define but is an important concept to address in many observational studies. There are several approaches that can be used to address SOI. The first approach is to use clinically defined parameters, such as vital signs and laboratory parameters, ICU utilization or transfer, comorbid conditions, or surgical and procedural interventions. Notably, defining SOI for many conditions can be challenging when relying solely on administrative data as important metrics may not be available (eg, vital sign parameters, laboratory results). Supplementing administrative data with information from chart review can enhance this approach to SOI. Alternative approaches to defining SOI include the use of metrics, such as the APR-DRG SOI or Hospitalization Resource Intensity Scores for Kids (H-RISK).¹⁴  The proprietary APR-DRG SOI uses 4 levels of severity that are defined based on demographics, diagnoses, and procedures. Since SOI levels are not comparable across APR-DRG groups, the H-RISK was developed to assign relative weights to each APR-DRG and SOI. Use of these latter 2 approaches can be found within studies by Synhorst et al, which used restriction based on APR-DRG SOI (ie, excluded those with SOI levels 3 [major] and 4 [extreme]) to limit the cohort and in Markham et al, which incorporated H-RISK within statistical models of pediatric hospitalizations for children with complex chronic conditions during the early coronavirus disease 2019 pandemic.^15,16 

Thoughtfully addressing sociodemographic factors is often necessary to create meaningful comparisons within observational studies. An extensive body of research describes disparities in child health outcomes based on the social construct of race and ethnicity across a variety of conditions and diseases. It is widely accepted that race and ethnicity represent social dimensions without a grounding in genetic or biologic mechanisms and that these factors are essential to consider in the context of observational studies. However, using race and ethnicity data are laden with challenges, including limitations with how data are collected (eg, observer, survey, self-reported, etc.), reported (eg, aggregate versus discrete groupings), and modeled (eg, interactions exist between race, ethnicity and socioeconomic status contributing to confounding and collinearity and difficulty disentangling the impact of individual factors). Recent publications including that by Cheng et al and Flanagin et al highlight the importance of considering race, ethnicity, and socioeconomic status and provide guidance for incorporating these factors within observational studies.^17,18 

Socioeconomic status (SES) refers to the combined economic and social standing of an individual. As with SOI and race and ethnicity, defining SES within observational studies can be challenging and the types and variety of metrics available varies based on individual data source. For example, in the Pediatric Health information System (Children’s Hospital Association, Lenexa, KS) database available variables to estimate SES include median household income and the COI,¹⁹  which are both based on a patient’s residential Zip code. The recent work of Krager et al using the COI 2.0 highlights how a multidimensional, composite measure of SES can be applied to better understand the impact of neighborhood conditions on hospitalizations for ambulatory care sensitive conditions.²⁰ 

Understanding and addressing sociodemographic and clinical differences within observational studies is necessary to reduce bias. A variety of techniques are available to researchers to address important covariates and to improve the understanding of observational study outcomes.

1. 1

   Many socioeconomic and clinical covariates have the potential to falsely influence study conclusions if unequally represented in study populations.

1. 2

   Methods to improve equal representation of covariates include selection, restriction, stratification, multivariable regression, and propensity scoring.

1. 3

   Common categories of covariates needing statistical adjustment in observational studies include demographics, clinical characteristics, and social factors.

1. 4

   Reducing bias in observational studies is dependent on what you can observe or measure. Results can be biased if there is an important covariate that goes unmeasured (and therefore unadjusted for in the analysis).