Prognostic Models Predicting Mortality in Preterm Infants: Systematic Review and Meta-analysis Free

To obtain an update of the systematic review by Medlock et al⁷  from 2011, the same search strategy was used. Medline was searched for all articles from May 2010 (last search date of Medlock et al⁷ ) up to June 2020 by using a search that followed the general form: “prediction model AND preterm AND infant AND mortality.” The detailed search strategy is shown in Supplemental Table 6. The current review is reported according to the Preferred Reporting Items for Systematic Reviews and Meta-analyses statement.¹⁰ 

Inclusion and exclusion criteria were similar to the criteria used in Medlock et al.⁷  The criteria following the critical appraisal and data extraction for systematic reviews of prediction modelling studies (CHARMS) checklist¹¹  are given in Table 1. In brief, all prognostic models that aim to predict mortality at any time point in infants born at <32 weeks and/or <1500 g birth weight were included. Studies were classified as pre- or postsurfactant on the basis of the authors’ report of surfactant use, whereafter presurfactant studies were excluded. In studies in which surfactant use was not reported, surfactant was assumed to be in routine use after 1990.

Titles and abstracts were independently screened by 2 authors each (P.E.v.B., P.A., W.O., and E.S.) and included if considered relevant. Before full text screening, all studies included in Medlock et al⁷  were added. In addition, Medlock was contacted for external validation studies that were excluded from their 2011 review, and these were added as well. Subsequently, full texts of the selected articles were screened in duplicate for final inclusion by 2 authors (P.E.v.B., P.A., W.O., and E.S.). Likewise, data extraction and ROB assessment were conducted in duplicate (P.E.v.B., P.A., W.O., and E.S.). In case of discrepancies, a third reviewer was involved to establish consensus.

Eligible articles were categorized into 2 groups: development studies and external validation studies, with separate data extraction forms for each group. Relevant items were extracted from each selected article by using the domains described in the CHARMS checklist, which included information on population, candidate predictors (only for development studies), outcome to be predicted, model development (only for development studies), and model performance.¹¹  If an article described the development or external validation of multiple (existing) models, separate data extraction for each model was conducted for each model. Additionally, a ROB and applicability assessment were performed by using PROBAST.^8,9  PROBAST is organized into 4 domains (participants, predictors, outcome, and analysis) and contains a total of 20 signaling questions to facilitate structured judgment of ROB. Signaling questions are answered as “yes,” “probably yes,” “no,” “probably no,” or “no information.” A domain in which all signaling questions are answered as “yes” or “probably yes” should be judged as low ROB, whereas a “no” or “probably no” on 1 or more questions in a domain flags the potential for bias. Insufficient information on 1 or more questions might result in unclear ROB as well as in low or high ROB, depending on judgment of the reviewers. Applicability of the study to the review question is assessed for the 3 domains participants, predictors, and outcomes and is rated as low, high, or unclear, with low concern regarding applicability if the review question and the study are a good match. To achieve consistent data extraction and ROB assessment, the standardized data extraction forms were piloted, modified, and finalized after discussion with all authors. The full list of final extracted items is available on request.

Details of the protocol for this systematic review were registered on PROSPERO (IDCRD42019141434).¹²  In the protocol, prediction of mortality 1 year after birth was registered as the maximum time span of prediction, but in the final review, no such maximum was used, meaning all articles on mortality were included, regardless of the time point at which mortality was predicted, giving a comprehensive overview of all available models. Consequently, the current review provides a comprehensive list of prediction models for mortality. Furthermore, in the protocol, it was stated that the aim of the article was to give a narrative overview, but in the final review, a quantitative analysis was also added. During the selection process, it became apparent that certain models had been externally validated frequently, thereby allowing quantitative (meta-)analysis of model performance.

Results of development and external validation studies were summarized by using descriptive statistics. Prognostic models that were externally validated in at least 5 studies were analyzed quantitatively by using random effects meta-analyses. If researchers of a study performed multiple external validations of 1 model, the validation with characteristics most similar to the development study was used for meta-analysis. Furthermore, meta-analysis was performed in each subgroup separately, on the basis of whether the study population was extremely low gestational age or birth weight (ELGA/BW) (defined as a gestational age <28 weeks or birth weight <1000 g) or very low gestational age or birth weight (VLGA/BW) (applicable to all infants that were not ELGA/BW). Subgroup analysis was performed if at least 5 studies were included in a subgroup. If no C-statistic was reported despite presentation of the receiver operating characteristic curve, WebPlotDigitizer was used to reconstruct the curve and to calculate the area under the curve (ie, the C-statistic). Logit transformation for the C-statistics was used during meta-analyses to overcome the poor statistical properties of the normal distribution when the C-statistic was close to 0 or 1 or when sample sizes were relatively small.¹³  Between-study heterogeneity was quantified by using the I² statistic.^14,15  A rough guide to interpretation of the I² statistic is as follows: 0% to 40% might not be important; 30% to 60% may represent moderate heterogeneity; 50% to 90% may represent substantial heterogeneity; 75% to 100% is considerable heterogeneity.¹⁶  Furthermore, 95% confidence intervals (CIs) were calculated to indicate the precision of the summary performance estimate, and 95% prediction intervals (PIs) were calculated to provide boundaries on the likely performance in future model validation studies that are comparable to the studies included in the meta-analysis and thus can be seen as an indication of model generalizability.¹⁷  In addition, we calculated the probability that the C-statistic of the validated models will be larger than 0.70 and 0.80 in future validation studies. All analyses were performed in R version 3.5.2.

The initial search yielded 2159 unique articles, as shown in the flowchart in Fig 1. After title and abstract screening, 62 articles were provisionally selected for full text screening. All 41 articles identified by Medlock et al plus an additional 18 articles reporting on the external validation of existing models were added for full text screening. Out of those 121 articles, 60 articles, including 29 articles from Medlock et al, met the inclusion criteria and were selected for data extraction. The 30 articles from Medlock were excluded because the article did not concern an individual prediction model (n = 7), the study was performed in the presurfactant era (n = 9), the population was not applicable to our research question (n = 8), the outcome was not applicable to our research question (n = 2), no full text was available (n = 3), or the article was written in a foreign language (n = 1). From the 36 studies reporting on model development, 144 unique models were identified (Table 2). In the 34 studies reporting on external validation, 118 models were validated (Table 3). Of these 34 studies reporting on external validation, 23 studies were used for meta-analysis of the Clinical Risk Index for Babies (CRIB) (n = 15), CRIB-II (n = 12), Score for Neonatal Acute Physiology Perinatal Extension (SNAPPE) II (n = 6) and National Institute of Child Health and Human Development (NICHD) calculator (N = 5) scores.

Table 4 shows key characteristics of the study design, sample size, predictors, outcome, modeling method, and predictive performance of the included model development studies. The majority of the included studies originated from registry or retrospective cohorts (n = 32, 89%). Of all 144 models, 60 (42%) models used birth weight as their inclusion criterion, 52 (36%) models used gestational age as their inclusion criterion, and 32 (22%) models used both birth weight and gestational age as inclusion criterion. The number of participants used for developing the models varied from 57 to 29 180 (median 828), and the number of events ranged between 16 and 4448 (median 171). The median mortality rate was 13%, with an interquartile range of 9% to 28%. The number of events per variable (EPV) could be calculated for 120 (83%) models, ranged from 0 to 426 (median 10), and was <10 for 51% of the models. Although the majority of prediction models were focused on mortality during hospital admission (n = 72, 50%) and within 28 days after birth (n = 31, 22%), 7 other outcome measures were identified, including mortality before term age, or within 7, 120, 180 days, 1 year, 18 to 22 months, and 5 years. The C-statistic varied from 0.70 to 0.95, with a similar range in subgroups for VLGA/BW and ELGA/BW. For 36 (25%) models, both discrimination and calibration were reported, with 10 (25%) models presenting calibration by using a calibration plot and the majority presenting the resulting P value of a Hosmer–Lemeshow test (n = 35, 88%). In total, 84 of the 144 models (58%) were internally validated, most often by using a random split of the data into development and validation data sets (n = 42, 50%) or cross-validation (n = 18, 21%). For 64 (44%) models, insufficient information was presented to allow calculation of individual risks.

Figure 2 summarizes all predictors included in the final models. Variables concerning size and maturity of the infant and variables concerning birth and delivery were most often included (in 77% and 64% of the final models, respectively).

Figure 3 shows a summary of ROB and applicability for all models. Across nearly all models, ROB related to outcome and predictors was considered low. ROB related to the participants’ domain was high in 14% of the models because of inappropriate inclusion and exclusion criteria of participants; for example, including <50% of the eligible infants or exclusion of infants that died later than the prediction horizon. By contrast, ROB related to the statistical analysis was high in every single model, mostly because of inappropriate handling of missing data (100%), not presenting all relevant performance measures (96%), a low number of participants with the outcome in relation to the number of candidate predictors (51%), and no correction for overfitting when indicated (94%). In summary, the overall ROB was high across all models.

The concern of the model not being applicable to our research question was high in 29% of the models, mainly because of inclusion of participants different from those in our research question (eg, studies excluding outborn infants).

Table 4, shows key characteristics of the study design, sample size, outcome, and predictive performance of the included external validation studies. Although in 34 articles, 118 external validations were performed, the majority of the 144 developed models (n = 135, 94%) had not been externally validated. For some models, an external validation study was included in this review, but not the original development study, because the model was developed in the presurfactant era or in a population not applicable to this review but was externally validated in a time period or population that was applicable. In total, 18 different models were externally validated (Table 3). Median mortality rate was 12% (interquartile range: 10%–23%), which was comparable to the mortality rate in the development studies. The C-statistic was reported for 109 (92%) models, with a range of 0.56 to 0.97. For 26 (22%) models, both discrimination and calibration were reported, with 14 (45%) models presenting calibration using a calibration plot and the majority presenting the resulting P value of a Hosmer–Lemeshow test (n = 20, 65%). Figure 4 shows a summary of ROB and applicability by domain. Across almost all models, ROB related to outcome, predictors, and participants was low. By contrast, ROB related to the analysis was high in almost all models, mostly because of inappropriate handling of missing data (95%) and not presenting a calibration plot (92%). This resulted in an overall high ROB for the validation of 114 (97%) models.

The CRIB¹⁸  was validated most (n = 15), followed by the CRIB-II¹⁹  (n = 12), the SNAPPE-II²⁰  (n = 6), the NICHD model²¹  (n = 5) and the Apgar score²²  (n = 5) (Table 3). However, the Apgar score was unsuitable for meta-analysis because of substantial heterogeneity across the external validations, caused by differences in moment of prediction, prediction horizon, and type of Apgar score (conventional, specified or expanded score). Results from the studies on external validation of the Apgar score are presented without meta-analysis in Fig 5E.

At meta-analyses, estimated approximate average C-statistics across the included studies were 0.88 (95% CI: 0.83–0.91, I² = 91%) for the CRIB score (Fig 5A), 0.87 (95% CI: 0.81–0.92, I² = 94%) for the CRIB-II score (Fig 5B), 0.86 (95% CI: 0.78–0.92, I² = 90%) for the SNAPPE-II score (Fig 5C) and 0.71 (95%CI 0.61–0.79, i² = 84%) for the NICHD model (Fig 5D). The 95% PIs were 0.63–0.97, 0.59–0.97, and 0.60–0.96 for CRIB, CRIB-II, SNAPPE-II and NICHD scores, respectively. Based on the forest plot in Fig 5A, the study of Asker et al was an outlier in comparison with other studies and as such may be great source of heterogeneity. Exclusion of this study lowered the I² to 69% and improved the 95% PI to 0.80–0.94. The probabilities that the scores would achieve a discrimination >0.7 and >0.8 in future validation studies were 93% and 78%, respectively, for the CRIB score, 92% and 78%, respectively, for the CRIB-II score, 93% and 78% for the SNAPPE-II score and 57% and 10% respectively for the NICHD score. A calibration plot was presented for 2 external validation studies of the CRIB score for 1 external validation study of the CRIB-II score and for 1 external validation study of the NICHD score, showing poor and good calibration for the CRIB score^19,23  good calibration for the CRIB-II score²³  and moderate calibration for the NICHD score.²⁴  Subgroup analyses in a VLGA/BW population for the CRIB-II and SNAPPE-II scores showed similar results (Supplemental Fig 6).

In this systematic review, we summarized all available prognostic models for mortality prediction in liveborn very preterm infants. In total, 144 models from 36 studies on model development and 118 models from 34 studies on external validation were identified, revealing that there is an abundance of mortality risk prediction models for very preterm infants. ROB assessment showed high ROB in the majority of the models, most often because of inadequate (reporting of the) analysis. Furthermore, internal and external validation of these models is often lacking.

Four main identified methodologic flaws within the analysis domain need addressing. First, at development, 61 (51%) models had a number of participants with the outcome in relation to the number of candidate predictors (EPV) <10, resulting in high ROB according to PROBAST because of the risk of overfitting. With such a small EPV, it is recommended to account for overfitting and optimism to decrease the ROB,⁹  but this was scarcely done in the included models. Historically, and as such in PROBAST, sample size consideration have been based on the EPV; however, it has been recently suggested to also include the total number of participants, the outcome incidence in the study population, and the expected predictive performance.²⁵ 

Second, none of the included studies handled participants with missing data correctly according to PROBAST. Use of missing data as an exclusion criterion or excluding enrolled participants with any missing data from the analysis leads to biased associations and model performance.^26–35  Therefore, multiple imputation is recommended to handle missing data because it leads to the least biased results with correct SEs and P values.^{26–31,33–35} 

Third, information on both calibration and discrimination was presented for only 25% of the models. Calibration was most often assessed by using a Hosmer–Lemeshow test, whereas this statistical test indicates neither the presence nor the magnitude of any miscalibration and is known to be dependent on the sample size.⁹  Therefore, it is recommended to present a calibration plot instead, which unfortunately was hardly ever reported in the included articles.

Fourth, 84 of the 144 models (58%) were internally validated, most often by using a random split of the data into development and validation data sets (n = 42, 50%). However, this has been shown to be an insufficient way of data use and as an inadequate way to measure optimism.^36,37  Instead, bootstrapping or cross-validation is recommended to quantify overfitting of the developed model and optimism in its predictive performance.³⁸  Furthermore, the majority of the studies performing internal validation seemingly failed to replicate the exact model development procedure and thus may still underestimate the actual optimism and thus overestimate the actual performance of their model.^39,40 

Methodologic flaws identified within the ROB assessment of the participants domain included using a nested case control design without correction for baseline risk, inclusion of <50% of the eligible infants, and exclusion of all infants who died after 7 days. Within the applicability assessment, issues raised included exclusion of outborn infants, a study conducted in a high altitude NICU, and exclusion of all infants who died within 72 hours.

This review reveals that development of new prediction models for mortality in preterm infants is an ongoing practice. However, many models are of unknown value for daily practice because of lack of validation. Therefore, future emphasis should be shifted toward external validation and adaption of existing prediction models, which applies to a broader field of prediction modeling and has been stated before.^41,42  Ideally, these validation studies are performed by using prospectively collected data because validation studies have higher potential for ROB when participant data are from existing sources with data collected for a purpose other than validation or updating of prediction models. Consecutively, impact studies are warranted to quantify the effect of a prognostic model on physician’s behavior and patient outcome.⁵ 

In the majority of development studies, participants, predictors, and outcome were described sufficiently clear and did not introduce bias. Contrastingly, high ROB occurred in the analysis section of practically all studies because of inappropriate analysis methods or omission of important statistical considerations. Moreover, for almost 40% of the models, information to allow others to correctly apply the models in new individuals (ie, information on predictors and coefficients of the final developed model including intercept) was insufficient. Improvements in studies on mortality risk prediction in very preterm infants are needed and can be achieved from better (reporting of) analyses. A first step in that direction would be better adherence to the TRIPOD (transparent reporting of a multivariable prediction model for individual prognosis or diagnosis) statement⁴³  and consideration of PROBAST.^8,9 

This review showed that variables concerning size and maturity of the infant (in 77% of the models), variables concerning birth and delivery (64%), and maternal variables (41%) were most often included. Specifically, gestational age, Apgar score, birth weight, sex, multiplicity, antenatal corticosteroids, and ethnicity were used as predictors in >40 models. This reveals the importance of these variables in mortality risk prediction in preterm infants. Nevertheless, because the vast majority of these models were considered of low quality and calibration of these models was not reported, their actual in value in mortality risk prediction remains unclear.

At meta-analysis of the C-statistic, the CRIB, CRIB-II, and SNAPPE-II, all revealed excellent performance (C-statistic >0.85), comparable to a recently published meta-analysis.⁴⁴  However, considerable heterogeneity across the included studies was found (I² ≥ 90% for all models), which can originate from differences between study populations and study designs.^45,46  Important characteristics of the included studies, including inclusion criteria, moment of prediction, and time span of the outcome, are shown in Fig 5 A–C and indicate substantial differences in study population. However, it is difficult to draw conclusions on the defining sources of heterogeneity, meaning further research will be necessary. Although the 3 models revealed great discriminative performance, information on calibration is largely lacking. To provide a complete and accurate judgment of the performance of these models, information on calibration, ideally by providing a calibration plot, will be needed. Compared to these three models, the NICHD model lagged in its discriminative performance (average c-statistics 0.71), which may be related to the study population of extremely preterm infants.

Health care decisions for individual patients should be informed by using the best available evidence. Systematic reviews summarizing large amounts of information are very powerful tools to facilitate clinical decision-making but also to identify gaps in our knowledge or room for improvement. In our article, we clearly show a lack of evidence regarding the external validity of the majority of models, poor (reporting of) analyses, and absence of calibration plots in the majority of the models. The abundant availability of insufficiently validated models is not useful for clinical practice.⁴⁷  In our systematic review, the extensive ROB assessment revealed that the model published by Manktelow et al⁴⁸  had the highest quality among all 144 developed models. Furthermore, the external validity of the CRIB, CRIB-II SNAPPE-II and NICHD models has been assessed often and show good discriminative performance.^18–21  Unfortunately, information on their calibration is still lacking. Based on the currently available evidence, we consider these 5 prediction models to have the highest potential for use in clinical practice. A first step would be to (again) externally validate these models, but now also focus on calibration. Presenting discrimination will be sufficient when the aim is to distinguish high and low risk populations, but for individual prediction information on calibration is essential. During such external validation, the original model may require an update, thereby addressing the potential issue of miscalibration associated with differences in mortality rate between the development and validation population. Ideally, such external validations are followed by impact studies to quantify the effect of a prognostic model on physician’s behavior and patient outcome.

Since Medlock et al⁷  published their systematic review of models for the prediction of mortality in very premature infants in 2011, only 1 systematic review in Spanish has been published.⁴⁹  Large improvements of our review compared with both existing reviews are (1) the use of a standard tool for ROB assessment, which is an essential step in any systematic review^8,9 ; (2) the inclusion of articles externally validating models and meta-analysis of the models most often validated, giving additional insight in their quality and value for clinical practice; and (3) the vast amount of newly published prediction models since 2011, showing the need for an update to provide a comprehensive overview of prediction models for mortality in very preterm infants.

However, this study has several limitations, too. First, for some models, an external validation study was included, but not the development study, because studies developed in the presurfactant era or in the general NICU population that included very preterm born infants but also infants born >32 weeks’ gestational age were excluded. Second, PROBAST is a recently developed tool using contemporary expertise and knowledge, which was applied to models of which some were developed and published decades ago. Information currently necessary for assessment of bias (eg, calibration) was often not reported, leading to high ROB in the analysis domain across all models. Third, the majority of the included studies originated from developed countries, making this review less applicable to developing countries. In future research, validating prediction models in developing countries might require more attention because there is much to be gained with respect to postnatal mortality in preterm infants.

There is an abundance of mortality risk prediction models for very preterm infants. Improvement in studies on mortality risk prediction in very preterm infants can be achieved from improved (reporting of) analyses. Many of the models are of unknown value for daily practice because of lack of external validation. Meta-analysis on the widely used CRIB, CRIB-II, SNAPPE-II and NICHD scores revealed good discriminative performance of these scores, but calibration is currently unknown. Instead of developing additional mortality prediction models for preterm infants, the emphasis should be shifted toward external validation and consecutive adaption of the existing prediction models for mortality in preterm infants.

CHARMS

critical appraisal and data extraction for systematic reviews of prediction modelling studies

CI

confidence interval

CRIB

clinical risk index for babies

ELGA/BW

extremely low gestational age or birth weight

EPV

events per variable

PI

prediction interval

PROBAST

prediction model risk of bias assessment tool

ROB

risk of bias

SNAPPE

score for neonatal acute physiology perinatal extension

VLGA/BW

very low gestational age or birth weight