Heterogeneity and Gaps in Reporting Primary Outcomes From Neonatal Trials

The study’s steering group, composed of 6 members from The Hospital for Sick Children, Imperial College London, and Cochrane Neonatal (Supplemental Information), provided project oversight.

Full-text neonatal clinical trials published between 2015 and 2020 in English that:

1. investigated neonates requiring care during NICU admission;

2. had a sample size of ≥100 in each arm of the trial;

3. were either any type of randomized controlled trial or a cluster-randomized trial; and

4. reported either single, composite, or multiple primary outcome(s) with at least 1 primary outcome included from the neonatal COS (Table 1).

For composite outcomes, at least 1 component of the composite needed to be from the neonatal COS.

Trials were identified in a 2-step process; first, through a previous systematic review (search dates: July 1, 2012–July 1, 2017).⁷  Second, an updated search was conducted on the Cochrane Central Register of Controlled Trials to capture trials published between January 1, 2017, and up to January 1, 2020 (search date: June 2020). Trials identified through the updated search were limited to those published in the following journals: Pediatrics, New England Journal of Medicine, Journal of Pediatrics, Archives of Disease in Childhood: Fetal and Neonatal Edition, PLoS One, Journal of the American Medical Association, The Lancet, JAMA Pediatrics, and The Lancet Child and Adolescent Health. The updated search was limited to these journals because the first 7 published 3 or more trials in the original systematic review, and between them accounted for 59% of all neonatal trials in the original sample; the last 2 were searched to account for 2 newer, highly relevant journals in neonatology. For trials identified in the updated search, we used the Cochrane Screen4Me workflow¹⁷  to exclude studies that were not randomized controlled trials; all documents were then uploaded to Covidence¹⁸  and duplicates were removed. The updated search strategy (Supplemental Table 5) is reported per the Preferred Reporting Items for Systematic Review and Meta-Analyses checklist (Supplemental Table 6).¹⁹  Targeted searching in Google Scholar was conducted to identify follow-up studies of the identified trials within the target date range. Follow-up studies that already had their original trial included in the study sample were excluded. No citation screening was conducted.

All 76 trials, identified in the previous systematic review,⁷  were screened for inclusion independently and in duplicate by 2 reviewers (E.S., C.R.). Discrepancies were resolved through discussions between reviewers, and if needed, a third reviewer (A.M.) was consulted. For records identified in the updated search, a prescreening was done to exclude those published outside of the specified time frame and/or not in target journals of interest (A.M.). Independent and duplicate screening of all remaining studies was completed by 4 reviewers (M.O., J.W., C.G., R.F.S.) who also undertook data extraction to identify and verify the trial primary outcome as part of the neonatal COS; conflicts were resolved by a senior reviewer (M.O.). Multiple reports from the same study were not included.

A standard data extraction form was created on the Research Electronic Data Capture platform.²⁰  Outcome reporting was assessed through items selected from the CONSORT 2010 statement¹⁵  and the newly developed CONSORT-Outcomes 2022 extension¹⁶  (Supplemental Table 7). CONSORT-Outcomes 2022 and selected items relevant to outcome reporting from CONSORT 2010 were combined resulting in a 38-item data extraction tool. For items that address >1 aspect of reporting, the item was split into subparts to capture reporting comprehensiveness (eg, for CONSORT 4b, which looks at setting[s] and location[s] where the data were collected, it was split into CONSORT 4b[i] for setting[s] and CONSORT 4b[ii] for location[s]). Item rating options are detailed elsewhere (Supplemental Table 8). In addition to the ratings, raters were asked to extract verbatim text from the trial or applicable supplemental information to justify their rating. The data extraction form was piloted by members of the steering group (M.O., N.J.B., J.W., C.G., R.F.S.) and refined before use as described elsewhere (Supplemental Information).

Comprehensiveness of outcome reporting in included trials was assessed by raters recruited from the Cochrane Neonatal authors database through e-mail. Details on rater identification and rater piloting is outlined in Supplemental Information; rater registration form is included in Supplemental Figure 3. Each included trial was assessed by at least 3 raters.

After raters completed their assessments, the steering group reviewed and adjudicated all scores, particularly the reporting items with obvious discrepancies among the raters and the items rated as “unclear if reported.” Descriptive quantitative methods (counts and frequencies) were used to analyze the final data on Microsoft Excel (A.B.). In the overall assessment calculations, “not applicable” items were excluded.

Figure 1 details the number of records identified in the search, reasons for exclusions, and the final number of trials included. A total of 37 neonatal trials were included. The trial used in the rater pilot exercise²¹  was removed from the final study sample, leaving 36 trials for the outcome-reporting assessment.

Table 1 details the characteristics of the 36 trials; additional details are provided (Supplemental Table 9). Trials had a median sample size of 456 (interquartile range 308–862), and were most commonly national multicenter trials (47%) conducted in Europe (30%). Composite outcomes were the most common trial primary outcome (56%). Ten of 12 neonatal COS primary outcomes were represented as primary outcomes, with survival/death/mortality being the most common (64%); none of the included trials used quality of life or adverse events as their primary outcome.

Seventy-two systematic review authors from the Cochrane Neonatal review group registered through the online form to participate as a rater in the study. The pilot exercise was completed by 46 raters. One rater’s pilot score did not meet the criteria of agreement with other raters and was therefore excluded. A total of 45 raters were sent materials to complete the outcome-reporting assessment for 3 or 4 trial reports each. Of these, 39 raters (Table 2; Supplemental Table 10) completed outcome-reporting assessments.

Discrepancies in rating assessment were observed for all 38 reporting items. The 4 items accounting for the most discrepancies were those related to minimal important difference (7a.1), reporting of absolute and relative effect sizes (17b), reliability of study instruments (6a.8[ii]), and description of study instrument(s) used to assess the outcome (6a.8[i]). Through steering group discussions, all apparent discrepancies were resolved, and a final reporting assessment was adjudicated to each reporting item for each trial.

Figure 2 portrays the variability of outcome-reporting comprehensiveness across the trials, calculated as the percentage of fully reported, partially reported, and unreported. The median percentage of fully reported items across trials was 75% (range 39–94%). Complete reporting was found in 22 (63%) reporting items for >80% of included trials.

Table 3 details the reporting frequency and percentage for the 13 CONSORT 2010 items relevant to trial outcomes. Not all items are applicable to all primary outcomes (eg, composite outcome [6a.6] is not applicable to single primary outcomes) or intervention (eg, blinding [11a] was not done in a trial). Reporting frequencies for each item were calculated after excluding trial reports where the item was not applicable; the denominator for each item is reported in Table 3.

CONSORT 2010 items were generally well reported, with 10 of 13 (77%) items fully reported in >80% of trials. Two of these 10 items were completely reported in all trials:

1. statistical methods used to compare groups for the primary outcome (item 12a); and

2. number of participants who were randomly assigned, received intended treatment, and were analyzed for the primary outcome (13a).

The 3 items reported in <80% of trials were:

1. changes to trial outcomes after trial commencement (6b; 31%);

2. blinding (11a; 66%); and

3. absolute and relative effect sizes (17b; 71%).

Table 4 details the reporting frequency and percentage of the 25 CONSORT-Outcomes 2022 items. Not all items were applicable to all 36 trials (eg, study instrument reliability [6a.7(ii)] is not applicable to mortality). As before, the number of trial reports that the item was not applicable to was excluded from the outcome reporting frequency and percentage calculation; the denominator for each item is noted in Table 4. Twelve (48%) items were fully reported in >80% of trials, of which 3 items were completely reported in all applicable trials: (1) specific measurement variable (6a.2[i]), (2) time point(s) used for analysis (6a.4), and (3) defined individual components for a composite outcome (6a.6). Although 9 trials (25%) reported outcomes that were not originally specified in a trial registry or protocol, 27 (75%) trials either did not identify or report additional outcomes that were not prespecified (6a.7).

This detailed analysis of primary trial outcome reporting in 36 recent neonatal trials identified important reporting gaps and elicited contributing factors to inadequate reporting. Although many key trial items were well reported, descriptions of blinding of the outcome assessor, minimal important difference between treatment groups, outcome data missingness, and how outcome multiplicity was dealt with in the analysis were insufficiently reported.

Overall, 10 (77%) CONSORT 2010 items were fully reported in >80% of included trials, whereas 12 (48%) of the new CONSORT-Outcomes 2022 items had been reported in >80% of trials. Key items were often insufficiently reported; <20% reported on the instrument’s reliability and validity, and only 40% reported qualifications or trial-specific training needed to administer the study instrument. Because the included trials were published from 2015 onwards, it is likely that trialists had access to CONSORT 2010 in drafting the trial report. Though CONSORT-Outcomes 2022 was still in development during the publication of all included trials and conduct of the current study, it is encouraging that close to half of CONSORT-Outcomes 2022 items are addressed in >80% trial reports; however, there is still room for improvement. Future studies may use our results to examine whether reporting improves over time.

Awareness and understanding of key concepts related to trial outcomes is fundamental for trialists to be able to comprehensively report their findings, and for research users to assess reporting quality. Our review of the extracted verbatim texts and discrepancies among the 39 raters’ assessments revealed that a small number of items, including blinding, minimal important difference, missingness, and management of outcome multiplicity, were confusing to raters. Raters often confused minimal important difference with sample size; although related, these are distinct concepts in the design and reporting of trials. Along with the CONSORT-Outcomes glossary (Supplemental Table 11), we provide the 5 core elements of a defined outcome (Supplemental Table 12) as resources to understand key concepts related to outcome reporting.

Raters’ scores also showed how frequently information in trial reports is simply missed; their assessments may resemble those of peer reviewers for academic journals. This points to a need for training and a structured reporting format for trials that everyone, including expert peer reviewers, could use during the peer review process.²²  Evidence shows that, although journals support authors and peer reviewers by encouraging the use of reporting guidelines and checklists to guide peer review,^23–25  currently, “instructions to authors” are variable between journals, and guidance for peer reviewers is heterogenous. Efforts to improve and provide clear standards for reporting outcomes to both authors and peer reviewers are needed to improve outcome reporting.

Insufficient reporting of key trial outcomes highlights weaknesses in current publication practices among academic journals. Because of various constraints, such as figure and word count limitations, trialists may leave out critical information to meet submission requirements. One reason for apparent limited outcome reporting in published trials is that information could be published in protocols or supplemental information. In the context of limited journal word counts, supplemental information is useful for obtaining further context on the results and methods that may not be found in the main text.²⁶  When available, we provided the supplemental information along with the trial report to our raters for their assessment. Referrals to appendices to the trial report and a rating, “They refer to another document,” were both considered adequate reporting. In our sample, trials without any available supplemental information (n = 16; 44%) generally had more unreported items; the 5 trials with the most unreported items had no accompanying supplemental information. We conclude that supplemental information is an underused tool to attain full trial reporting.

The identified underreported items reemphasize the need for reporting guidelines like CONSORT and its extensions. Uptake of these guidelines has been shown to improve reporting^27,28 ; CONSORT 2010 items are increasingly well reported, suggesting that they are effective.^29,30  Journals could endorse use of reporting guidelines as a standard for submission, and identify the main reporting guidelines for trials they send out for review.^31–34  Journals could also mandate trial registration,^34,35  implement guidance for what to include in supplemental information while considering its usefulness to different stakeholders,²⁶  and make statistical analysis plans and changes to information included in a registry readily available for their readership. Such incentives in the publication process that encourage comprehensive reporting have been used before and showed a positive impact.²⁷  Uptake of the neonatal COS, combined with use of reporting guidelines, could facilitate meta-analyses and translation of results into clinical practice, a positive trend observed in other fields.³⁶ 

Beyond trialists, journal editors, and peer reviewers, knowledge users and consumers of research need to be aware of the pervasive, suboptimal reporting in research.³⁴  There is a need for transparency in reporting to reduce bias and research waste, because clinical decisions around fitting treatment options that impact patients and families are based on published trial results. Though sometimes not aware of a trial’s quality, readers deserve to know that there are ways trialists and journals can enhance reporting, such as through understanding and endorsement of the “minimal reporting items” in reporting guidelines. Readers should look carefully at what outcomes have been reported and how they have been reported. We urge readers to consult other accompanying, accessible documents in tandem with a trial report, such as the supplemental information, trial protocol, statistical analysis plan, and trial registration, to gain a comprehensive overview of what they are reading, and to be able to interpret the results presented. General readers’ awareness of these issues may propel transparency in research and increase buy-in from all stakeholders within the research enterprise.

Strengths of our study include the involvement of an international group of clinicians, systematic reviewers, and trial experts identified through Cochrane Neonatal who assessed outcome reporting in recent neonatal trials, which allowed us to develop a better understanding of what is considered well reported or inadequately reported. We used items from empirically developed reporting guidelines using robust consensus methodology to rate outcome reporting. Although we used CONSORT-Outcomes 2022 before its publication in December 2022, it provided us insight to current neonatal trial reporting because it codifies best reporting practices. Focusing on Core Outcomes in Neonatology primary outcomes in recent neonatal trials ensured that we examined the reporting of outcomes identified as important to key stakeholders in neonatology.

Limitations first include our restriction to trial primary outcomes; we cannot guarantee our findings apply to reporting of secondary outcomes. In adjudicating whether an item was reported or not, we took a liberal approach, meaning if any element of the reporting item was addressed, we scored it as reported; however, we found very few examples of optimal reporting. Examples of optimal reporting of each item to guide prospective writers of neonatology trials are needed. Second, we were unable to assess reporting of quality of life and adverse events, because these were not used in our sample. We note that these are rarely used as primary trial outcomes, yet their reporting deserves dedicated attention. Third, we recognize the potential of language bias because we only included studies published in English. Fourth, the journals that the included trials were published in endorse the use of reporting guidelines, which is likely a contributing factor as to why CONSORT 2010 items were generally well reported. With this in consideration, our findings may overestimate the true completeness of outcome reporting in neonatal clinical trials, because many neonatal studies are not published in the journals to which we restricted our search. Outcome-reporting comprehensiveness may differ in journals that do not endorse reporting guidelines.

Reporting of primary outcomes in neonatal trials often lacks key information needed for interpreting results, knowledge synthesis, and evidence-informed decision-making in neonatology. Use of existing outcome-reporting guidelines such as CONSORT 2010 and CONSORT-Outcomes 2022 by trialists, journals, and peer reviewers will increase usability of neonatal research, provide more opportunities for evidence synthesis to inform decision-making at the bedside, reduce research waste, and improve child health outcomes.

We thank the Core Outcome Reporting in Neonatal Trials Study Group (Supplemental Information and Supplemental Table 10) for their contributions. We also thank Colleen Ovelman (managing editor, Cochrane Neonatal) for her assistance with the updated search strategy.

COS

core outcome set

CONSORT

Consolidated Standards of Reporting Trials