Assessing and Validating a Model of Study Completion for a Prospective Cohort of Healthy Newborns Free

Participants in the current study were drawn from a single tertiary-care urban hospital in Philadelphia, PA, as part of an original study to assess the efficacy of a smartphone application to detect and measure neonatal jaundice in the first week of life among singleton births >35 weeks’ gestational age. Full details of the original study are described elsewhere.¹²  Exclusion criteria for the original study included any infants who were transferred to the NICU or had undergone phototherapy treatment. In short, mothers were approached between July 2015 and March 2019 during the birth hospitalization to enroll their newborns in the study. Consenting mothers were asked to complete a brief instrument capturing sociodemographic characteristics and contact information. The study visit was scheduled to occur at an outpatient laboratory within walking distance of the birth hospital between 2 and 6 days of life and was scheduled to coincide with the newborns’ first well visit with their pediatrician. Following enrollment, multiple reminder texts were sent to participants instructing them of their follow-up visit; additional attempts and phone calls were made in further efforts to contact participants in the absence of affirmative text responses. At the scheduled outpatient laboratory visit, the smartphone application to assess for jaundice was used, immediately after which a serum bilirubin was obtained. Data for this study were abstracted from the electronic health record (EHR) and participant study records from the original study retrospectively; any participants who completed their study in hospital were also excluded from analysis. The current study of mother-infant dyads was approved by the university’s institutional review board.

Variables that were identified as empirically important to our study outcome based on their clinical relevance or evidence from previous studies were abstracted from 1 of 2 sources: the EHR or primary study materials. Data abstracted from the EHR included maternal age, parity, delivery mode, length of stay, insurance status, discharge day, and follow-up health care provider location. Follow-up location was recoded into a binary variable of follow-up “colocation,” in which “colocation” consisted of visits occurring in 1 of 3 physician practices across the street from or in the same building as the laboratory where the study visit occurred, and “no colocation” for all other locations. Original primary data from the parent study used for this substudy included infant race/ethnicity as specified by the mother and whether an alternate phone number was provided. Of note, information on maternal race was not collected. Participants in the original study were given $50 gift cards for participation; if their follow-up visit was not colocated, they were given an additional $10 gift card to cover the cost of transit and/or parking for the study.

Our primary outcome for all analyses was study completion for the newborn jaundice study, defined as completing both the smartphone application procedure and blood collection at the scheduled study visit.

Variables identified via the literature as having a possible role in study completion including maternal age, parity, delivery mode, length of stay, maternal and infant insurance status, discharge day, follow-up location, infant race/ethnicity, and provision of an alternate contact number were abstracted from the EHR or primary study materials; univariate and bivariate analyses were conducted. After assessing for evidence of collinearity and correlation in several variables of concern (maternal and infant insurance, delivery mode, and length of stay), we determined to include only maternal age, parity, delivery mode, maternal insurance status, discharge day, follow-up location, infant race/ethnicity, and provision of an alternate contact number in our final models, excluding length of stay and infant insurance status. Estimates are shown as odds ratios (ORs) with corresponding 95% confidence intervals (CIs).

To evaluate the predictive validity of the final study completion model, we calculated the following measures of accuracy: sensitivity, specificity, positive and negative predictive values, and percent correctly classified. Receiver operating characteristic curves, which plot the true positive rate of the model against the false-positive rate of the model at various thresholds, were used to compare the predictive power of the model. We report the area under the receiver operating characteristic (AUROC) curve, to quantitatively assess performance of the model. Internal validation of the model was then conducted using k-fold validation, with 5 folds (n = 100-101 per fold) to ensure that each testing fold had a sufficient sample size.^13–15  The resultant predictive outcomes from the cross-validation model were compared with the predictive outcomes from the full predictive model via confusion matrix, which compares actual versus predicted outcomes using cross-tables, as well as via comparison of measures of predictive validity (eg, sensitivity, specificity) between the full and cross-validated models.

All statistical analyses were conducted using SAS 9.4 (SAS Corporation, Cary, NC).

Of the 558 participants in the original study, 501 (89.8%) met the stated inclusion criteria for this study. Among those included, 60.7% (n = 304) went on to complete their study visit (study completers), whereas 39.3% (n = 197) did not complete the study visit (study noncompleters). In our sample, 47.9% (n = 240) reported infant race/ethnicity as non-Hispanic Black/African American. The median age of mothers was 28 years (mean, 28.2), median parity level was 1 previous child (mean, 1.2), and median length of stay was 2 days (mean, 2.2). Approximately 81.8% (n = 410) delivered via vaginal birth.

Bivariate analyses of study completers versus study noncompleters revealed statistically significant differences based on maternal length of stay (2.1 days vs 2.3 days, P = .01) vaginal versus cesarean section delivery (84.5% vs 77.7%, P = .05), follow-up colocation versus no colocation (75.7% vs 53.3%, P < .001), and provision of an alternate phone number versus no alternate number (49.0% vs 36.0%, P = .004). Full descriptive statistics and bivariate analyses can be found in Table 1.

Of the variables included in our logistic regression model, only follow-up colocation, provision of an alternate phone number, and delivery mode remained statistically significant predictors of study completion. Holding all other covariates constant, infants who had follow-up colocation were just under 3 times as likely to be study completers versus those that did not (adjusted OR [aOR], 2.99; 95% CI, 2.01–4.46); mothers who provided an alternate phone number were just under twice as likely to be study completers compared with those who had not (aOR, 1.99; 95% CI, 1.34–2.96); and those who delivered via vaginal birth were just over 1.5 times as likely to complete the study (aOR, 1.65; 95% CI,1.01–2.70). Full model information can be found in Table 2.

The final model had an AUROC of 0.68 (95% CI, 0.63–0.73), indicating statistically better than chance prediction of study completion. Sensitivity of the final model indicates that 69% of study completers were correctly predicted to complete the study, and specificity indicates that 62% of those study noncompleters were correctly predicted not to complete the study.

Cross-validation of the model showed accurate performance with little difference between the predictive model and the cross-validated model, which had an average AUROC of 0.65, a sensitivity of 68%, and a specificity of 60%. The confusion matrix, which shows differences in predicted and actual outcome between the predictive and cross-validated model, can be found in Table 3. These results showed that classification of final and cross-validated models had near-total concordance, with only 5 predictions shifting in the cross-validation. Full diagnostic measures of the predictive and cross-validated models can be found in Table 4 for comparison. An overlay of the final model AUROC curve and the cross-validation AUROC curves is shown in Fig 1 and demonstrates visual similarity, noting that the range of cross-validation values (0.59-0.72) only slightly differs from the CI of the full model (95% CI, 0.63–0.73).

Among mother-infant dyads enrolled in a prospective cohort study on neonatal jaundice, we identified a few factors predictive of higher odds of study completion. In our bivariate analysis, we identified maternal length of stay, delivery mode, follow-up colocation, and provision of alternate phone number as predictors of completion. In our final model, delivery mode, physical colocation of the first infant follow-up visit with the study visit, and provision of an alternate phone number retained significance; no other variables included were significant predictors. These findings offer guidance to researchers, particularly in the design phase of studies, who conduct hospital-based studies involving prospective recruitment of healthy newborns and their families.

We identified delivery mode as a potentially important and actionable predictor of study completion. Our findings suggest that delivery via cesarean section may be slightly predictive of noncompletion. Given that a full recovery following cesarean section can take significant time after hospital discharge and may cause reduced mobility and increased pain, as well as other physical and mental health issues,¹⁶  women who undergo a cesarean section delivery may require additional accommodations for study completion, particularly when the study requires follow-up visits. Additionally, it has been noted that timing of first outpatient well visits for those delivering via cesarean section is later than those delivering via vaginal birth,¹⁷  which may corroborate evidence of issues of mobility and may indicate a need for improving the ease of participation for mothers delivering via cesarean section.

Additionally, colocation of the first infant follow-up and the study visit was an important highly predictive factor of study completion. It is likely that part of the explanation for these findings is that the coordination of the study visit with the outpatient visit saves significant time and effort for families wanting to participate and allows for a more uniform and personalized approach (eg, the research team member maintains consistent communication leading up to the study visit and conducts the study visit first, then subsequently guides the mother-infant dyad to their well visit) compared with those following up at an office that is remote from the study visit. This finding is also especially notable considering that participants who did not have colocated follow-up were given additional financial compensation for the cost of travel. This suggests that additional barriers to participation were not sufficiently addressed by additional financial incentives. In addition, one must also consider that local context may play a role. Nearly one-quarter of Philadelphians rely on public transit¹⁸ ; given the already difficult task of transportation after giving birth, this may represent an additional burden to study completion beyond financial compensation. Although it is difficult with our current data to confirm that a significant proportion of our sample did rely on public transit, data from other studies have suggested that transit may play an important role in hospital access, and particularly for low-income populations,^19,20  which is important to consider given that nearly 70% of our sample was uninsured or on Medicaid.

Other cohort studies of longer duration have shown that consistent communication is an important predictor of study completion²¹ ; in our sample, the mother providing an alternate phone number may have facilitated easier/more consistent communication by offering multiple points of contact, given the reliance on phone communication methods in this study as described in the Methods. It is also possible that by obtaining contact information directly from the participants (as opposed to via extraction from EHR data) may have led to improved accuracy of contact information compared with what can be obtained from EHR data, which may be prone to error or inaccuracy. Although outside the scope of our current study, future research on study design and implementation may investigate whether contact information provided by study participants to research coordinators is more accurate than EHR-derived contact information. It is also possible that provision of an alternate contact number may in part be a function of social support because individuals without social support may not have a second point of contact to provide, though we did not collect information on social support.

Our findings highlight potential barriers to study completion in otherwise willing participants. These findings strengthen existing evidence and show that some factors identified in high-risk infants, such as staff-participant communication, are also applicable to studies with healthy newborns. As a means of improving retention, our findings support obtaining alternate phone numbers directly from participants when possible and colocate study visits with other necessary clinical care. Although not directly supported by our analysis, we argue, based on informed speculation from our findings regarding predictiveness of delivery mode as well as other literature on the topic, that researchers focusing on well-newborn populations should also seek to address potential barriers to participation a priori during study design, (eg, how the burden of cesarean section delivery or the use of public transit may hamper participation after enrollment). In addition, future studies should collect completion and retention data as well as sociodemographic and other study factors previously identified in the literature to evaluate study recruitment and retention practices. Accurate collection of these data in real time can help to improve researchers’ abilities to prospectively identify and address potential socioeconomic or structural barriers participants may face in participating in research, which may also improve equity of participation in research.

Finally, as noted in the Results, many factors previously identified as potentially predictive (eg, race/ethnicity, insurance status, parity) were not predictive in our models. This contradiction should be considered in combination with our diagnostic statistics, which indicate our model is better than chance at predicting study completion, but not perfect. It is possible that our findings represent a refutation of previous research models predicting study completion. However, based on our experience as clinicians and epidemiologists, we suspect it is more likely that prediction models, and predictive indicators, may vary substantially from population to population, even when those population-level differences are minor (eg, primarily healthy newborns versus high-risk newborns). Thus, before using these models in other settings, investigators should consider whether variables such as race/ethnicity, insurance status, and parity might modify the observed associations, with careful consideration of how this could affect external validity of the models.

These findings should be considered in light of several strengths and limitations. First, our methods allowed us to assess the predictive accuracy of our model, providing additional confidence in our results going a step further than many studies that look at factors of study completion. Second, our large sample of healthy newborns represents a relatively understudied subpopulation in the context of neonatal research.

One important limitation of this study was the absence of potentially important but unmeasured predictors of study completion (eg, maternal health literacy, attitudes toward research, maternal race/ethnicity [particularly as a proxy for racism]),^3,6  which may have contributed to the observed modest predictive accuracy of our model. Taken in the context of previous studies with conflicting results on predictors of study completion, results of this study highlight the need for a nuanced approach to prediction and the general difficulty of constructing predictive models for study completion. The positive and negative predictive values reported from our analyses should be interpreted cautiously given that these measures are sensitive to prevalence of the outcome; as such, we recommend focusing on sensitivity and specificity.

Because our data came from a single hospital, generalizability is also a potential concern. However, this single urban tertiary-care hospital serves a diverse patient population from all parts of Philadelphia and surrounding counties such as Bucks, Montgomery, and Delaware.²²  We also note that given the period of the current study (occurring within the first week of life), our findings may not generalize to well-newborn research requiring follow-up over on longer time scales. Furthermore, because phototherapy was an exclusion factor for the original study, it is likely that severity of hyperbilirubinemia was associated with eligibility for the current study and thus could cause some degree of selection/sampling bias. However, the outcome of our current study (study completion) was agnostic to neonatal bilirubin level, making selection bias less likely to present as a major issue. Although outside of the scope of this manuscript, methods for quantitative assessment of selection bias are available and may be an area of focus for future research.^23,24 

We identified several factors predictive of study completion in a cohort of healthy newborns. These factors emphasize the importance of good participant-coordinator communication and more patient-centric approaches in clinical research to improve retention, such as those aimed at reducing participant burden. Additionally, our findings present future researchers with cost-effective guidance on study design, which may improve participant retention among diverse urban populations of healthy newborns. Future work should aim to prioritize these and other factors that maximize completion while continuing to collect information important to evaluating retention strategies. In particular, evaluating and addressing potential socioeconomic barriers to study completion may increase equity in prospective clinical research.

The authors thank the sitewide principal investigator for the original study, Dr James Taylor, as well as previous research coordinators Mitali Panchani and Rachel Powell for their invaluable help in collecting the data necessary to conduct this analysis.