Validation of a Prediction Rule for Mortality in Congenital Diaphragmatic Hernia

The data were collected from 4 high-volume CDH centers treating ≥10 patients with CDH per year.¹⁹  These centers are part of the CDH Euro Consortium: Erasmus University Medical Center, Rotterdam, Netherlands; Radboud University Medical Center Amalia Children’s Hospital, Nijmegen, Netherlands; University Hospital Mannheim, Mannheim, Germany; and Bambino Gesù Children’s Hospital, Rome, Italy. The CDH Euro Consortium is a voluntary collaboration of European institutions that work together in research. This collaborative group also developed the CDH Euro Consortium management guidelines that are implemented in all participating centers.^1,20  Institutional review board approval was obtained from the Medical Ethics Committee, Erasmus University Medical Center in Rotterdam (MEC2016-109).

For the external validation of the CDHSG prediction rule, patients born before November 2011 were excluded because these patients were included in the CDHSG database and were used for the development of the original CDHSG prediction rule.¹⁵  We included all live-born infants with CDH who were born between November 2011 and November 2015. We reviewed the data of these patients from the local CDHSG database and added missing data from the medical files if available. The collected data were in accordance with the definitions used by Brindle et al¹⁵ : low birth weight (<1500 g); an Apgar score <7 at 5 minutes or the absence of an Apgar score; severe PH, defined as right-to-left shunt or estimated supra-systemic pulmonary pressures on the first echocardiogram; chromosomal anomalies, defined as any abnormalities in the chromosomal array; major cardiac anomalies, classified as all anomalies other than a patent foramen ovale; patent ductus arteriosus; an atrial septum defect; and a ventricular septum defect.

The data of each patient were entered in the CDHSG prediction rule to calculate a total CDH risk score, ranging from 0 to 8 (Table 1). This score was used to stratify the patients into 1 of the 3 risk groups; low (0), intermediate,^1,2  and high risk.^3–8 

For the implementation of prenatal variables in the CDHSG prediction rule, we included all live-born infants with prenatally diagnosed CDH who were born between 2008 and 2015. The predictors in the CDHSG prediction rule were reviewed. Most of the variables were used as binary variables. However, to further improve the model, birth weight was also tested as a continuous variable, and low Apgar score was defined as <5 at 5 minutes or <7 at 5 minutes. Missing Apgar score was left out because one of the centers never calculates an Apgar score for patients with CDH. Also, after discussion with an expert group consisting of pediatric intensivists, neonatologists, and prenatal specialists across participating centers, we decided that the variable chromosomal anomalies should always be in the model because of its major significance in the decision to start and continue treatment.

Additionally, candidate pre- and postnatal predictors were selected by the expert group. The first measured O/E LHR after 18 weeks’ gestation was included as a continuous variable. The presence of intrathoracic liver herniation on the last prenatal ultrasound was used as a binary variable. Also, the side of the hernia, fetal endotracheal occlusion, the presence of polyhydramnios,²¹  gestational age at diagnosis, and gestational age at birth were selected.

To describe the baseline characteristics of the patients with CDH, medians and interquartile ranges (IQRs) were used for continuous variables, and percentages were used for categorical variables. Comparisons between baseline characteristics and death before discharge were made by using the χ² test for categorical variables and the Mann–Whitney U test for continuous variables. Comparisons between centers were made by using Kruskal-Wallis and χ² tests.

For the external validation of the CDHSG prediction rule, multiple imputation was performed for missing data (Table 2), creating 100 databases by using fully conditional specification. Because the available data between centers were heterogeneous, we used “center” as a covariate in the multiple imputation. Then the CDHSG prediction score was calculated for each individual by using the final prediction rule, as used by Brindle et al,¹⁵  as well as the original equation, which was used to develop the CDHSG prediction rule (Table 1). The predictive performance was assessed by using calibration plots and the c-statistic (ie, the area under the receiver operating characteristic [ROC] curve). Also, the predicted outcome of the final equation was compared with the observed outcome in the study cohort from the pooled database.

For the new model, predictors were tested by using univariate analysis, assessing if a variable was associated with increased mortality. We corrected for center. The selected variables were put into a multivariable logistic regression model by using the stepwise backward method. In every step, the variable with the highest P value was excluded if P > .1, and this was repeated until all variables included in the model had P < .1. The model was evaluated with a calibration plot to assess the discriminatory abilities of the model, followed by bootstrapping to correct for the optimism of the model. We then calculated the predicted risk per patient and plotted the ROC curves to determine cutoff values of the predicted risk for 3 risk groups: low, intermediate, and high risk. SPSS Statistics version 24 (IBM SPSS Statistics, IBM Corporation, Armonk, NY) and R version 3.6.1, with the packages rms and mice, were used for the statistical analyses.

A total of 753 patients were diagnosed with CDH between January 1, 2008, and December 31, 2015. Eight patients were excluded because there were no patient characteristics available. Fourteen pregnancies resulted in an intrauterine fetal demise. Three hundred forty-three patients were born between 2011 and 2015, and their data were used for the validation of the original prediction rule. Six hundred twenty patients were included to develop the new rule. In 111 patients, the diagnosis was not prenatally known, and therefore they were excluded for the new rule. This postnatally diagnosed group had a mortality of 9%.

Baseline characteristics of both patient groups are shown in Table 2. In 70.3% of the patients in the cohort used for the validation, the first echocardiogram was performed within the first 24 hours of life. The overall mortality was 18%. In the group used for the new rule, 76.9% of the patients had their first echocardiogram performed within the first 24 hours of life. Their overall mortality was 23%. In both groups, the baseline characteristics of the patients who survived were significantly different from those of patients who died, except for sex (Table 2). These characteristics also differed significantly between centers, as presented in Table 3.

The outcome of the CDHSG prediction rule after multiple imputations is shown in Table 4. Forty-six percent of the patients were grouped in the low-risk group (score 0), with an observed mortality of 4%, and 38% of patients were grouped in the intermediate group (score 1–2), with a mortality of 22%. The high-risk group (score 3–8) was smaller, containing 16% of the patients, with a mortality of 66%. The discrimination of the model was moderately strong, with a c-statistic of 0.784 for the original equation and 0.790 for the final CDHSG prediction rule.

Subsequently, to develop a new rule, the original prediction rule was modified. First, logistic regression was performed within the large data set by using a backward elimination algorithm. Missing data were imputed. O/E LHR, side of the hernia, gestational age at birth, fetal endotracheal occlusion, polyhydramnios, Apgar score <5 at 5 minutes, and gestational age at diagnosis were excluded from the model with backward elimination. Although chromosomal anomalies had a P value >.1, we forced it into the model (Table 5). The new model contains birth weight as a continuous variable and intrathoracic herniation of the liver, major cardiac anomalies, chromosomal anomalies, Apgar score <7 at 5 minutes, and severe PH as binary variables (Table 1). An evaluation of the model in a calibration plot revealed good discrimination of the model, with a c-statistic of 0.859. When correcting for the optimism of the model, estimated ∼1.4%, the c-statistic is 0.846. In Supplemental Fig 1, the ROC curve of the new model is shown. We then stratified the patients into 1 of the 3 groups: low, intermediate, and high risk of mortality. When using <10% (mild), 10% to 50% (moderate), and >50% (severe) risk of mortality as cutoff points, the cutoff between the mild group and the moderate group revealed a sensitivity of 90.8% and a specificity of 55.4%, whereas the cutoff between the moderate and the severe group revealed a sensitivity of 49.3% and a specificity of 93.5%.

The disease severity, by using the rules per center, is presented in Supplemental Tables 6 and 7.

In this study, we externally validated the CDHSG rule in the European population. We found that the rule had fair discrimination, but also room for optimization, compared with the internal validation of Brindle et al.¹⁵  Bent et al¹⁶  also validated the rule in a large group of patients with CDH born in California and found an underestimation of mortality in the patients with a score of 1. We did not find this in our population. This might be explained by the difference in health care systems in Europe and the United States. In Europe, centralized care is more common and many patients with CDH are born in high-volume centers. It is increasingly recognized that centralized care improves outcome in these patients.¹⁹  This might also explain the lowest mortality in patients born in the largest center of our study. Furthermore, in Europe, many CDH centers collaborate in the CDH Euro Consortium, which has developed a standardized treatment protocol, increasing survival from 67% to 88%.^20,22 

Although it seems valid to use the model, some variables were not useful or difficult to apply in our population. In 1 of the 4 centers, Apgar scores were never measured because the medical team felt it was not a useful tool in this patient group. Patients with CDH will have a lower Apgar score because they are intubated directly after birth.²⁰  Brindle et al¹⁵  theorized that the absence of an Apgar score implies an infant is sicker, but this was not applicable to our cohort. Also, the measurement of PH on echocardiography is not standardized, and different definitions for PH are being used. Brindle et al¹⁵  used right-to-left shunting or estimated suprasystemic pulmonary pressures. In Europe, the presence of PH is often defined as pulmonary pressures higher than two-thirds of the systemic pressures.^20,23,24  Furthermore, the timing of the measurement differs between centers. Brindle et al¹⁵  used the earliest echocardiogram in the model, whereas Bent et al¹⁶  used PH at discharge. Presumably, the incidence of severe PH was underestimated in the study by Bent et al¹⁶  because many patients with PH would have already died, and in others, pulmonary pressures would have decreased.²⁵  The registered incidence of PH in nonsurvivors is only 33.5% in the study by Bent et al¹⁶ , whereas it is >50% in our cohort and >60% in the CDHSG population, supporting this assumption.^15,16 

To improve the power of the original prediction rule, combining pre- and postnatal variables is presumably superior. Prenatal variables have been found to adequately predict lung hypoplasia and the need for extracorporeal membrane oxygenation (ECMO) but are less reliable as a marker for PH.^8,9  To predict mortality, postnatal variables are still essential. Different prenatal variables were tested in the model, and eventually, only the position of the liver was a significant variable. Surprisingly, the O/E LHR was not of additional value to the model, whereas in earlier studies, the O/E LHR did have a role in predicting survival. It is a more reliable prenatal predictor than the lung-to-head-ratio because it is a stable variable during pregnancy.^6,26  However, different O/E LHR measurement techniques are being used between centers, and there is a learning curve in the examination of the O/E LHR.^27,28  In 2 of the centers, the tracing method was always used, whereas in the others, also the longest-axis-diameter method was used. The longest-axis-diameter method overestimates the O/E LHR up to 34% and has a larger interobserver variability.^29–31  In addition, the O/E LHR can be calculated with multiple calculators (ie, www.totaltrial.eu or www.perinatology.com), resulting in different ratios. In our study, the method of calculation varied. Measuring lung volumes on MRI holds promise.³²  However, in many centers, it is not possible to use MRI for this purpose because of costs and lack of availability. For this study, analysis of fetal lung volume resulted in too many missing data. Another prenatal predictor is stomach position.⁷  This measure, however, is not implemented in standard prenatal care and could not therefore be analyzed for this study. More consistent measuring techniques and reporting would probably make these different prenatal variables more suitable for use in outcome prediction.

For the modified model, we also made some changes to the original variables. “Missing Apgar score” was taken out. On the other hand, we kept chromosomal anomalies in the model, although its association with mortality was not significant (P > .1). Possibly, some patients had a clinically insignificant abnormality on array; however, because these data are retrospective, it was not possible to reliably select only clinically relevant cases. Therefore, we choose to include all patients with abnormalities on chromosomal array. CDH is associated with numerous chromosomal anomalies, and often the associated anomaly has a major impact on mortality.^33,34 

The strength of our study is the large population of patients with CDH and the amount of prenatal data available. Often, the implementation of prenatal data in a prediction model is difficult because of the amount of missing data.¹⁵  Also, when using a voluntary database, or a database based on coded diagnoses, the accuracy of the data are difficult to interpret. We were able to go back to the original patient files when needed. However, we did have some missing data, as shown in Table 2, which we corrected for using multiple imputation, a statistical tool often used in this setting. However, it is possible that some predictors were not significant in the model because of the large amount of missing data, such as the presence of polyhydramnios and gestational age at birth. Furthermore, because all 4 centers are part of the CDH Euro Consortium, postnatal clinical management is similar in all centers, increasing the reliability of an early prediction model, but it also potentially limits the generalizability of our study and may contribute to an optimistic assessment of model performance. Therefore, our new model needs additional external validation in a more heterogeneous group like the patients in the CDHSG database to prove its generalizability. Also, because prenatal data are necessary for the new rule, other settings with imperfect prenatal care provision may not benefit as much from this new model. The original CDHSG model is easy to apply at the bedside. For our model, a more complicated calculation is necessary.

In a population with a rare congenital defect with high mortality and morbidity, it is important to reliably predict outcomes. Prenatally, this can guide parents and clinicians in decisions regarding perinatal management and in the referral to high-volume centers for the delivery in areas with a low-density population. Postnatally, adequate prediction can help parents to better understand the course of their child’s illness. In addition, it can also be used for standardized reporting and benchmarking between centers. A good postnatal prediction model can potentially improve care for specific groups of patients with CDH. This model can act as a practical tool when stratifying patients into risk groups such as mild, moderate, and severe. The mild group of patients, identified with a highly sensitive threshold, could receive less-aggressive treatment, such as a spontaneous breathing trial at birth.³⁵  On the other hand, the severe group, identified with a highly specific threshold, could potentially benefit most from more aggressive experimental therapies, and the true benefit of these therapies could be detected earlier because there would be no dilution of effect due to the inclusion of patients with lower risk. For all these reasons, there is a need for a reliable prediction model that can be applied on the first day of life.³⁶  The power of a prediction model is dependent on the availability of the information needed for the calculation. Prenatal data are not readily available in some areas of the world. However, the only prenatal parameter significant in our model, the prenatal position of the liver on ultrasound, is reasonably easy to evaluate.

We have successfully validated the CDHSG prediction rule within a European population. We also developed a modification of the original rule, implementing prenatal variables, with apparent improvement of the predictability of mortality. Standardization of the measurements of prenatal variables, such as the O/E LHR, and the postnatal variable PH could potentially increase their predicting value and further improve these models. Validation of this modified rule is needed to evaluate its generalizability.

We thank Dr R. Crisafulli for the data collection. We thank Prof K. Allegaert and Dr R. de Jonge for their useful contribution to the article.

CDH

congenital diaphragmatic hernia

CDHSG

Congenital Diaphragmatic Hernia Study Group

ECMO

extracorporeal membrane oxygenation

IQR

interquartile range

O/E LHR

observed-to-expected lung-to-head ratio

PH

pulmonary hypertension

ROC

receiver operating characteristic