Missing Outcome Data in Recent Perinatal and Neonatal Clinical Trials Free

We searched Medline (via PubMed) for RCTs that enrolled newborns or their birthing parents, reported outcomes in the children, and were published between January 1, 2020, and December 31, 2022. The search was limited to selected high-impact general medical journals (Lancet, New England Journal of Medicine, Journal of the American Medical Association, British Medical Journal), and pediatric specialty journals (JAMA Pediatrics, Pediatrics, Lancet Child and Adolescent Health). We did not intend to perform a systematic review of imputation practices in all recently published perinatal and neonatal trials. Instead, we examined a small sample of target journals for perinatal and neonatal RCTs with high impact factors to examine best-case scenarios of current imputation practices.

Our search strategy is summarized in Supplemental Table 5. The inclusion criteria for our survey were: Trials must have enrolled newborn infants randomized before 28 days after birth, and for trials that enrolled pregnant women, outcomes had to be measured in their children at any time after birth. Data extraction from eligible trials included the numbers of participants who were randomized and numbers included in primary analyses for primary outcomes. When studies provided information on missing primary outcome data for both mothers and children, we extracted both data sets and included the larger amount of missing primary outcome data in our analysis. For RCTs with missing primary outcome data we examined:

1. whether authors reported the missing data in their results;

2. what statistical method they used for the primary analysis; and

3. whether they performed any sensitivity analysis accounting for the missing data, and if so, what imputation methods were used.

Data screening and extraction were conducted by authors in pairs (J.Z. and Y.L., G.L. and Q.M.) and in duplicate, with discrepancies resolved by discussion or adjudicated (L.T.). Continuous data were described by means and SDs, and categorical data by counts and percentages. We compared the trials published in general medical journals with those published in pediatric journals using χ² or Fisher’s exact tests.

A total of 87 eligible perinatal and neonatal RCTs were included in this survey (Supplemental Fig 1). Of these, 77 (89%) trials had missing primary outcome data. The mean percentage of randomized participants who had missing primary outcome data were 11% (SD 16%). The majority of missing primary outcomes were dichotomous (n = 61, 79%), and the remaining missing outcomes were continuous (n = 16, 21%). The dichotomous outcomes included death, either as a single outcome or as component in a composite outcome (n = 21, 27%); infection (n = 9, 12%); and a variety of other endpoints. Infant weights were the most common continuous primary outcome (n = 9; 12%) to be missing.

Table 1 summarizes our findings in the 77 trials with missing primary outcome data. The authors of 9 RCTs (12%) did not report the missing data in their results or comment on the missing data. Sixty-one study teams (79%) used a complete-case analysis as their primary approach, which meant that they excluded participants with missing primary outcome data from the analysis. Only 38 (49%) trial reports included any sensitivity analysis using a variety of imputation methods (Table 1). Reported approaches to the main analysis differed between trials published in general medical journals when compared with those published in pediatric journals. Sensitivity analyses were used more often in the former (Table 1).

Statisticians distinguish between data that are missing completely at random, data that are missing at random, and data that are missing not at random.¹⁰ An example of data that are missing completely at random may be an infant’s blood sample that is subsequently lost in a usually well-run laboratory. In this scenario, the blood test results are missing completely independently of the infant’s characteristics, the usual performance of the study setting, or the study team’s adherence to the study protocol.

Missing at random data can sometimes be deduced from other observed information about the participant or setting and statistical adjustments can be made accordingly. In this scenario, the missing data should likely be independent of any unobserved information for that participant. For example, an infant’s ventilation data may be missing because of known sloppiness at the study site.

In contrast, if data are missing not at random, bias can still arise from unobserved characteristics of the participant or setting, even after adjustments have been made for all observed variables. For example, outcome data may be missing in a delivery room resuscitation trial for a certain day of the week, possibly related to a shift schedule. These data are likely missing not at random, because their completeness may depend on the unobserved and likely variable skills or number of personnel in the delivery room.

Complete-case analysis was frequently used by reports in our survey as the only approach to the analysis of the primary outcome. Authors who restricted their analysis to participants with complete outcomes may have implicitly, but often incorrectly, assumed that outcome data in the excluded participants are missing completely at random. Understanding and exploring the reasons for missing data allow reassurance of the comparability between groups and inferences about treatment effects. Only data that are missing completely at random allow researchers to assume participants with complete outcome data remain a representative subset of the total study sample. However, it is generally unrealistic to assume that data are missing completely at random.³ Besides, unfortunately, there is no easy way to verify post hoc whether the data may be in truth missing at random or even missing not at random.^11,12 Therefore, sensitivity analyses are required to evaluate the impact and robustness of various assumptions of why data are missing.¹³ 

Missing outcome data increases the risk of bias, decreases statistical power and precision, and reduces the representativeness of the sample population, thereby weakening the applicability, reliability, validity, and interpretability of the trial results. Missing data that could affect trial conclusions may occur in trials of treatments for various diseases.^14,–17 Up to 33% of 235 RCTs with statistically significant results for their dichotomous primary outcomes were reported to lose statistical significance under the following plausible assumption: “The incidence of events in those lost to follow-up relative to those followed up is higher in the intervention than control group.”¹⁸ All these 235 RCTs had been published in leading medical journals.¹⁸ 

Imputation is defined as “the practice of filling in missing data with plausible values.”¹⁹ Methods to perform this include single imputation, multiple imputation, inverse probability weighting, and model-based imputation. These 4 types of imputations are described and discussed in Table 2. In our survey, among the 38 trial reports with sensitivity analyses, a single imputation method was used in 15 (40%), a multiple imputation technique in 22 (58%), an inverse probability weighting method in 3 (8%), and a model-based method in 2 (5%) trials. The authors of 3 trials (8%) did not specify the type of imputation methods they used. Some trials used >1 imputation method.

To illustrate potential differences resulting from a complete-case analysis and several single imputation methods (mean imputation, and worse/best case scenario method) in the presence of missing primary outcome data, we show a hypothetical RCT in Table 3. In this hypothetical example, when primary outcomes are known for all 100 infants, we obtain the true estimate of the treatment effect of a relative risk (RR) of 0.90, with 95% confidence interval (CI) of 0.54–1.49. However, if we assume that 10 of the 50 infants in each group of the trial have missing primary outcome data as described in Table 3, the different methods of analysis produce different results. For example, the best-case scenario method, which assumes that all patients with missing data in the placebo group experience the primary outcome event, yields the most biased and favorable estimate of the treatment effect (RR 0.63, 95% CI 0.37–1.04). In contrast, the worst-case scenario method, which assumes that all patients with missing data in the intervention group would experience the primary outcome event, results in the most biased and unfavorable estimate of the treatment effect (RR 1.79, 95% CI 1.06–3.02). This hypothetical example illustrates how sensitive the results are to the method of handling missing outcome data. The example is not intended to show which method works best. The best way to assess the performance of imputation methods is through simulation based on individual patient data, which could inform the selection of imputation methods in a specific context for handling missing data.³ 

The appropriate choice of imputation method for an individual trial depends on the assumptions about the reasons for missing data. However, it is often difficult to decide whether the data are missing at random or missing not at random. Although the mean imputation method may be used to deal with missing baseline covariates in RCTs,²⁹ the multiple imputation and mixed-model methods are preferred solutions to missing at random data because they have shown superiority to single imputation methods in overall performance and reduction of bias.³⁰ Statisticians do not agree on the optimal choices of methods for all circumstances.^{12,24,28,30,31} Perinatal and neonatal trialists should consult their trial statistician about the best approach. More research is needed to better inform the selection of imputation methods in specific contexts. Indeed, it is strongly recommended to conduct sensitivity analyses using different imputation techniques to assess or support the robustness of the primary approach in clinical trials.^3,4 Inevitably, selecting imputation methods involves guesswork. Therefore, we strongly recommend that trialists should devote substantial energy and resources during the design and conduct of the trial to prevent or at least minimize missing primary outcome data.

Among the 38 trials that performed sensitivity analyses in our survey, most did not find differences between the results of sensitivity analyses and those of their primary analysis (Table 1). However, in a trial of skin emollient applied from 2 weeks of age to prevent atopic dermatitis at age 1 year, the risk of atopic dermatitis was significantly increased in the emollient group when multiple imputation was used (risk difference 5.9%, 95% CI 2.0%–9.7%), whereas the primary approach using the best-case imputation method yielded a nonsignificant treatment effect (risk difference 3.1%, 95% CI −0.3% to 6.5%).³² This example supports the importance of performing sensitivity analysis to address missing outcome data.

Evaluating trial quality requires assessment of risk of bias because of missing data as part of the methodological considerations, as in the Cochrane Risk of Bias Tool 2.³³ Moreover, reporting guidelines for study protocols and statistical analysis plans mandate that investigators detail how they will handle missing data.^34,35 However, the CONsolidated Standards of Reporting Trials guideline does not currently and explicitly state the handling of missing data in reporting of trial results. This anomaly should be reconsidered and incorporated into future CONsolidated Standards of Reporting Trials iterations.

The majority of trials (70 of 77, 91%) with missing primary outcome data in our survey reported use of the intention-to-treat (ITT) principle for their primary analyses. A trial with a complete-case analysis in which missing data are ignored can still claim adherence to an ITT principle if participants are analyzed according to the group to which they are assigned. An ITT principle with a complete-case analysis does not contain any strategy to address potential problems of missing data, or protect against bias from missing primary outcome data. Hence, combining ITT analysis with different data imputation methods is recommended.^36,37 

A common practice during trial design is to empirically increase the planned sample size to account for anticipated dropouts. This practice, however, is flawed because it cannot account for the bias that results from missing data, especially from data that are missing not at random. When the bias resulting from missing data is similar to or even larger than the expected intervention effect size, it is impossible to accurately estimate a true intervention effect, even with a large sample size.¹ In short, increasing the sample size cannot correct for bias. Therefore, to calculate a sample size, it is suggested to estimate how the intervention effect could become attenuated because of dropout and nonadherence over time, or to generate a sophisticated statistical procedure that accounts for missing data and the associated uncertainty.³ 

Study participants or their representatives may withdraw from the trial intervention, refuse further data collection, or even request that already collected data not be included in any analyses. Data withdrawal leads to missing data and little may be known about the reasons for withdrawal, although these are important for exploring the mechanisms of missing data and the potential impact on conclusions.

Although parents’ wishes to withdraw from a trial intervention must be respected, they should always be asked to consider allowing data collection to continue, in particular for the primary outcome. Those data could then be included in the ITT analysis. We have previously provided detailed suggestions about the handling of data withdrawal.³⁸ 

The foremost strategy to handle missing data is to prevent missing data, especially during the design and conduct of the trial. Many recommendations stress this.^3,4,39,40 However, suggested strategies are rarely, if ever, specific to the unique situation of follow-up in perinatal and neonatal trials in which the participant is the parent–child dyad.

Both perinatal characteristics of the child (including gestational age, birth weight, and medical complications), as well as caregiver characteristics (including sociodemographic status, education, and parental perception of the child’s medical and developmental progress), are associated with likelihood of attrition.⁴¹ Parents report structural barriers to follow-up such as travel distance to appointments, problems related to finding child care for siblings, and anxiety surrounding developmental testing of the child.⁴² 

To limit attrition in perinatal and neonatal studies, research teams should engage families in discussion about the importance of long-term follow-up beginning from the first encounter. Study team members should make efforts to establish strong relationships with the parent participants.⁴³ In neonatal trials, trust is further enhanced when study team members understand both the research study and neonatal care practices and unit culture. Frequent interactions (including phone calls and birthday or holiday cards) provide opportunities to maintain these relationships while also updating family contact information. Additional strategies include flexible study visit scheduling (including evenings and weekends), free parking or transportation, availability of home study visits, child care for siblings, and provision of small age- and developmental status-appropriate gifts for the child, in addition to fair compensation for time spent attending the visit for the guardian. In study design phases, investigators can engage representative families in planning the protocol and selection of outcomes and measures, perform feasibility assessments to understand potential reasons for withdrawal, allow flexible treatment regimens, avoid outcome measures with likelihood of low response rates, and select assessments that provide relevant and helpful information for families about their children’s medical or developmental progress. In today’s era of telemedicine, investigators could also consider validated virtual assessments to evaluate outcomes.^44,45 Finally, as study results become available, these must be shared with families at a sixth- to eighth-grade reading level. Such feedback provides opportunities for investigators to provide information, maintain relationships, and express gratitude to study participants.

These suggested strategies are summarized in a Population, Intervention, Control, Outcome, Time frame, Study design, and Analysis format in Table 4, aiming to enhance prompt understanding for readers.

The handling of missing primary outcome data has been largely inadequate in recent perinatal and neonatal trials. This is concerning because missing data may result in misleading inferences about treatment effects. Although carefully conducted imputations of missing data are a necessary component of the analysis of clinical trials, preventing missing data as much as possible remains the best approach to reduce bias from attrition.

CI

confidence interval

ITT

intention to treat

RCT

randomized controlled trial

RR

relative risk