Single-Examination Risk Prediction of Severe Retinopathy of Prematurity

This study was approved by the institutional review boards at the coordinating center (Oregon Health & Science University) and at each of the 7 study centers (Columbia University, University of Illinois Chicago, William Beaumont Hospital, Children’s Hospital Los Angeles, Cedars-Sinai Medical Center, University of Miami, and Weill Cornell Medical Center) and was conducted in accordance with the Declaration of Helsinki. Written informed consent was obtained from parents of all enrolled infants.

As part of the multicenter Imaging and Informatics in Retinopathy of Prematurity (i-ROP) cohort study, 842 unique patients (BW <1501 g or GA <31 weeks) were screened multiple times for ROP between January 2012 and July 2020. During each examination, retinal fundus images were captured via a RetCam (Natus, Pleasanton, CA). Patients were clinically examined at the bedside but also received image-based ROP diagnoses, which were determined by a consensus of 3 ROP experts using the full International Classification of ROP criteria.⁴  Patients’ retinal images were required to have expert consensus agreement that their quality was acceptable for diagnosis; 33 images did not meet this criterion. Clinical comorbidities and demographics were recorded for all patients’ examinations (Table 1, Supplemental Table 4). Statistical significance, as applicable, was determined by using Welch’s 2-sample t test and was defined at a cutoff of P ≤ .05.

Each eye examination was represented by a single RetCam image centered on the macula, which is approximately the field of view of zone I. Images were analyzed by i-ROP DL, an algorithm developed to detect plus disease (a manifestation of severe ROP).¹⁴  i-ROP DL provided a softmax probability of each image having normal, preplus, or plus disease vasculature (ie, it approximated the probability [P()] of each class, in which values range between 0.0 and 1.0 but must sum to 1.0 across all classes). From these values, a VSS, ranging from 1.0 to 9.0, was developed: VSS = P(normal) + 5 × P(preplus) + 9 × P(plus).

The VSS has been shown to independently correlate with more posterior disease (zone), higher stage, and higher extent of stage 3 ROP, in addition to plus disease (all the components of the International Classification of ROP criteria).^15–19  On the basis of previous work, the 32 to 33 weeks’ PMA imaging window was identified as potentially predictive of TR-ROP.^16,17  Thus, the first eye examination in this window was used for each patient. Because the goal was to develop a predictive (rather than diagnostic) model, infants who were diagnosed with TR-ROP within this window were excluded from the training data set (specifically, if they developed TR-ROP within 7 days of the first examination to occur within the 32 to 33 weeks’ PMA window). The held-out test data set (a subset of examinations from the i-ROP data set that were only used for model evaluation) contained all infants eligible for ROP screening, regardless of if or when they developed TR-ROP. Patients were mutually exclusive to the training (n = 376 patients) and test (n = 444 patients) data sets. The training data set contained 58 eyes that eventually developed TR-ROP and 660 eyes that did not.

BW, GA, and VSS were evaluated via recursive feature elimination by using multiple ElasticNet models trained by using Sci-Kit Learn in Python.²⁰  ElasticNet is a type of logistic regression in which a mixture of L1 and L2 regularization is used.²¹  L1 and L2 regularization is useful for feature selection and when collinear and codependent features are included in a model, respectively, and help to improve model generalizability. The ElasticNet mixing parameter was tuned via fivefold cross-validation by using 11 evenly distributed operating points from 0.0 to 1.0. Values of 1.0 and 0.0 are equal to L1 and L2 regularization, respectively. Because of the class imbalance (ie, eyes that eventually developed TR-ROP versus those that did not), the area under the precision-recall curve (AUPR) was the primary measure of model performance rather than the area under the receiver operating characteristic curve (AUROC) because the AUROC may be too optimistic, that is, a random classifier theoretically has an AUROC of 0.5 but an AUPR only equal to the proportion of positive cases divided by the total number of cases.

The performance of the model with the highest AUPR was assessed via the F_β score by using fivefold cross-validation across 101 evenly distributed operating points from 0.00 to 1.00. Whereas the F₁ score (β = 1) attempts to balance the proportion of false-negatives to false-positives, increasing β (eg, F₂, F₃, etc) prioritizes minimizing false-negatives over minimizing false-positives. The F₂ score is commonly used to slightly prioritize minimization of false-negatives. To minimize false-negatives, β was set to 4. The mean operating point (minus 1 SD) that maximized the F₄ score was selected and used to evaluate both test data sets.

This model was then evaluated on the held-out i-ROP test data set and on an independent data set that was collected between September 2015 and June 2018 from 132 unique patients born at a hospital in Salem, Oregon (Table 1). Data collection and exclusion criteria were similar to those for the i-ROP data set. Retrospective evaluation of these data was performed under a waiver of consent from the Oregon Health & Science University Institutional Review Board. Because patients are referred for treatment (not individual eyes), test data set evaluations were conducted at the patient-level (ie, if 1 or both eyes were predicted to develop TR-ROP, the patient was labeled as such). The i-ROP test data set contained 74 patients (132 eyes) who eventually developed TR-ROP and 370 patients (729 eyes) who did not. The Salem data set, contained 7 patients (14 eyes) who developed TR-ROP and 125 patients (248 eyes) who did not. The main outcome measures were sensitivity, specificity, positive predictive value (PPV), negative predictive value (NPV), and their corresponding 95% confidence intervals (CIs), evaluated independently by using the conservative Clopper-Pearson method, as suggested by Ying et al.²² 

In a secondary analysis, the maximum VSS between eyes for patients in the i-ROP test data set who screened positive was managed over time. On the basis of previous work, the VSS has potential for use as a monitoring tool to detect disease progression. The change in VSS over time for patients who screened positive and eventually developed TR-ROP was compared with that for those who screened positive but did not develop TR-ROP. Statistical significance was set at a cutoff of P ≤ .05 and was determined by using an analysis of variance and a Welch’s 2-sample t test.

Table 1 displays the relevant demographics and VSSs at 32 to 33 weeks’ PMA and clinical outcomes in the 2 data sets used in this study. In both data sets, eyes that developed TR-ROP tended to have higher VSSs at 32 to 33 weeks, and infants who required treatment in 1 or both eyes tended to have lower BW and GA.

ElasticNet was tuned via fivefold cross-validation for all combinations of BW, GA, and VSS. An ElasticNet model with an L1 ratio of 0.4, by using the predictors GA and VSS, had the highest AUPR (0.35 ± 0.11; Table 2, Fig 1). A random classifier would have an AUPR approximately equal to 0.08 (the proportion of TR-ROP cases in the training data set).

The operating point was tuned for increased sensitivity (so that all cases of TR-ROP would be identified) before we evaluated performance on the test data sets. The maximum F₄ score ± SD (0.74 ± 0.12) occurred at an operating point of 0.33 ± 0.08. To further increase sensitivity, this operating point was lowered by 1 SD to 0.25.

The model was then evaluated on the held-out test data set from the i-ROP database (Table 3). It identified all infants who eventually required treatment (sensitivity: 100.0% [CI 95.1%–100.0%]; PPV: 28.1% [CI 22.8%–34.0%]) while correctly identifying nearly half of the infants who never would (specificity: 48.9% [CI 43.7%–54.1%]; NPV: 100.0% [CI 98.0%–100.0%]). For infants who developed TR-ROP, the average number of weeks ± SD to TR-ROP diagnosis was 3.7 ± 2.7 weeks (range: 0.1–11.0 weeks) after prediction.

The model was also evaluated on an independent test data set collected from a hospital located in Salem, Oregon (Table 3). Again, it correctly identified all infants who eventually required treatment (sensitivity: 100.0% [CI 59.0%–100.0%]; PPV: 22.6% [CI 9.6%–41.1%]), and specificity increased to 80.8% (CI 72.8%–87.3%) (NPV: 100.0% [CI 96.4%–100.0%]). The average time ± SD to TR-ROP diagnosis, after prediction, was 3.4 ± 2.1 weeks (range: 0.1–5.0 weeks).

Among positive predictions in the i-ROP test data set (Table 3), the average VSS was managed over time. Patients who developed TR-ROP appeared to have a greater change in average VSS compared with those who screened positive but never required treatment (P ≤ .05), suggesting that specificity could be further improved by analyzing change in VSS over time (Fig 2).

We tested whether incorporation of an artificial intelligence–based assessment of vascular severity could improve the performance of ROP risk prediction models. We found that using just GA and VSS (obtained during a single eye examination at 32 to 33 weeks’ PMA) can identify all infants who are at risk for developing TR-ROP nearly 1 month before diagnosis and simultaneously rule out more than half of the low-risk population. With further validation, implementation of this model could reduce the number of ROP examinations and associated physiologic stress for low-risk infants. Finally, quantitative monitoring of vascular severity may lead to earlier and more consistent diagnosis of TR-ROP in infants who are at the highest risk, thus minimizing the overall risk of adverse outcomes.

This hypothesis was based on previous work that revealed that a DL-derived VSS may identify high-risk eyes as early as 1 month before TR-ROP diagnosis.^16,17  This proved to be accurate because the AUPR of the VSS at predicting TR-ROP was 0.07 points higher than the BW or GA univariate models, or the combination thereof (Table 1). This suggested that diagnostic prediction might be higher if a combination of the VSS and GA and/or BW were to be used in a risk model. After optimizing the operating point of the highest-performing algorithm (GA + VSS) for increased sensitivity (to avoid missing cases of TR-ROP), the model correctly identified 100% of infants who developed TR-ROP in 2 separate populations.

The intended use population and the potential impact of the PPV and NPV in each target population must also be considered. In the i-ROP data set, consistent with a population of infants from academic medical centers (who may be higher risk than those in the average NICU), the specificity of the model was 48.9%, compared to 80.8% in the Salem, Oregon, hospital, where the incidence of TR-ROP was lower. Even in the higher-risk population (i-ROP), these results suggest that, by 32 to 33 weeks’ PMA, half the population could be accurately identified as low risk and no longer require frequent examinations. The Salem, Oregon, population suggests that this proportion may be substantially higher in community ROP screening programs.

We also found that using the VSS to monitor disease progression may further enhance early detection of incident TR-ROP in infants who screen positive (Fig 2). This is consistent with previous work revealing that quantitative monitoring of vascular severity may be useful not only for screening but also for quantitative diagnosis and determining if the disease is stable, progressing, or regressing.^14–19  This could lay the framework for a new model of ROP screening in which low-risk infants receive less examinations and high-risk infants receive earlier and more precise diagnoses. To this point, it may be worth investigating the roles of oxygen exposure, intraventricular hemorrhages, sepsis, necrotizing enterocolitis, thrombocytopenia, and other previously associated risk factors to further increase specificity, although they may complicate this model and/or introduce confounding effects.

This model may also be easier to implement than previous ROP risk models. The performance of the GA + VSS model is comparable to the initial performance measurements of the Children’s Hospital of Philadelphia ROP model, which used a combination of BW + GA + weight gain to predict future occurrences of type II ROP and TR-ROP.¹²  Both models achieved 100.0% sensitivity in predicting TR-ROP and had similar specificities. However, when the Children’s Hospital of Philadelphia ROP model was applied to an external validation cohort of infants admitted to 30 hospitals across North America, the operating point had to be lowered to achieve 100.0% sensitivity, consequently reducing specificity to just 6.8%, which is too low to have a substantive impact on screening protocols.¹³  Another advantage of the proposed model is that it only requires data from a single examination. In general, GA is known with high precision, except in low- and middle-income countries (LMICs), where dating pregnancies may be less reliable. In these settings, it may be worth exploring a model that uses BW + VSS instead because Table 1 suggests almost comparable performance. However, a retinal fundus photograph obtained at 32 to 33 weeks’ PMA is also required, and herein lies the main barrier to implementation at this time. Images are not part of the standard of care, and digital fundus cameras can be expensive, so images are not often obtained.^2,4  As cameras drop in price and smartphone-based cameras become viable alternatives, it may be that future studies validating this concept reveal that the clinical benefit of earlier detection of high-risk infants, along with the reduced screening burden, outweighs the cost of implementing routine imaging.^23–25  Nonetheless, this remains a barrier to implementation and the main disadvantage of this method.

Additionally, this model is not likely to generalize well “out of the box” to populations different from the North American screening population. In many LMICs, the epidemiology and demographic risk factors are different, and the model would need to be retuned on the basis of local disease epidemiology.^9,26,27  For example, high-risk infants could be less premature and a time point other than 32 to 33 weeks’ PMA may be more predictive. There is, however, evidence that the i-ROP DL system accurately diagnoses TR-ROP in an Indian ROP telemedicine program, suggesting that the technology is effective in that context and thus may be translatable.¹⁹ 

Regardless, this model has potential to create a paradigm shift, transitioning from ophthalmology-led to neonatology-led ROP screenings, because the only required inputs are GA and a fundus photograph (not a complete ophthalmoscopic examination). Such a paradigm shift could, in addition to reducing the number of examinations needed for low-risk infants, dramatically reduce the number of examinations for which an ophthalmologist is needed. This could lead to better use of scarce resources, especially in rural regions and LMICs, where this is a significant issue.^26,27 

We have trained and optimized an interpretable, parsimonious model for the prediction of TR-ROP. In 2 separate validation cohorts, we demonstrated that a single examination at 32 to 33 weeks’ PMA detected all infants who eventually developed TR-ROP and more than half of those who did not. Implementation of this model could lead to significantly fewer ROP examinations for low-risk infants, better use of ROP screening resources, and earlier recognition of TR-ROP disease progression. Future work will validate this concept in LMICs, where the potential added value may be even greater given the increasing prevalence of disease and scarcity of resources, with the goal of reducing or eliminating blindness due to ROP.

AUPR

area under the precision-recall curve

AUROC

area under the receiver operating characteristic curve

BW

birth weight

CI

confidence interval

DL

deep learning

GA

gestational age

i-ROP

Imaging and Informatics in Retinopathy of Prematurity

i-ROP DL

Imaging and Informatics in Retinopathy of Prematurity Deep Learning

LMICs

low- and middle-income countries

NPV

negative predictive value

PMA

postmenstrual age

PPV

positive predictive value

ROP

retinopathy of prematurity

TR-ROP

treatment-requiring retinopathy of prematurity

VSS

vascular severity score