Perinatal Risk Factors and Outcome Coding in Clinical and Administrative Databases

The California OSHPD provides a linked vital statistics birth, newborn discharge, and maternal delivery data file. Data for these files are submitted directly to OSHPD from designated hospital staff or a designated reporting agent at minimum twice yearly. Data for the CPQCC database are submitted directly to CPQCC in real time from bedside nurses and dedicated data abstractors who undergo yearly training, with logic and range checks at the time of data entry, as well as confirmation when records exceed defined thresholds for missing or unobtainable items. In addition, consistency checks are employed for infants who transfer hospitals. We conducted a probabilistic record linkage of the CPQCC database to the OSHPD maternal and infant files for 2006–2012 on the basis of infant date of birth, maternal date of birth, infant sex, birth weight, birth location, infant disposition, infant discharge date, and birth order.

We included all inborn infants who were born in 2006–2012, met eligibility criteria at a CPQCC NICU, and who were discharged from the hospital or died. CPQCC eligibility criteria include all infants with a birth weight between 401 and 1500 g or gestational age between 22 + 0/7 and 29 + 6/7 weeks’ gestation, in addition to all infants admitted before 28 days of age and meeting any of the following criteria: death before discharge, acute transfer into an NICU, acute transfer out of an NICU, major surgery requiring anesthesia, assisted ventilation for >4 hours, nasal intermittent mandatory ventilation for >4 hours, early bacterial sepsis, readmission for total serum bilirubin >25 mg/dL, or exchange transfusion.

We extracted perinatal risk factor and outcomes data from this CPQCC record. Perinatal risk factors included birth weight, Cesarean delivery, multiple delivery, fetal distress, meconium aspiration, and patent ductus arteriosus (PDA), as well as maternal chorioamnionitis, hypertension (pregestational and gestational, including preeclampsia), and diabetes (pregestational and gestational). Infant outcomes included mechanical ventilation, pneumothorax, respiratory distress syndrome, chronic lung disease, necrotizing enterocolitis, intraventricular hemorrhage, hypoxic-ischemic encephalopathy, retinopathy of prematurity (ROP), extracorporeal life support, and in-hospital death. In addition, we evaluated combined grade 3 and grade 4 intraventricular hemorrhage and combined stages 3 through 5 ROP to align with common categorization of these more clinically important outcomes.

We then added relevant International Classification of Diseases, Ninth Revision, Clinical Modification (ICD-9-CM) diagnosis and International Classification of Diseases, Ninth Revision, Procedure Coding System (ICD-9-PCS) procedure codes from the OSHPD maternal and newborn inpatient discharge records, as outlined in Supplemental Table 3. For the primary analysis, International Classification of Diseases, Ninth Revision codes in either the maternal or infant record were considered to be coded in the administrative database. Using the CPQCC record as the gold standard, we then developed 2 × 2 tables and calculated several statistical measures for each risk factor and outcome.

Because of the highly variable group sizes, particularly for rare events, our primary measure of interest was the Matthews⁸ correlation coefficient (MCC) as a summary estimate of the quality of classification.⁹ MCC ranges from −1 (complete disagreement) to 1 (complete agreement), with a value of 0 representing predictive ability equal to random chance. Because MCC is a specific application of the Pearson correlation coefficient, we interpreted the strengths of associations accordingly, with 0.1 to 0.4 indicating weak positive correlation, 0.4 to 0.7 indicating moderate positive correlation, and 0.7 to 1 indicating strong positive correlation. Our secondary measures of interest included the positive predictive value (PPV) (proportion of positives that are true-positives), because this provides an intuitive measure of the precision of the test, or the ability to avoid false-positives. In a similar way, we calculated the negative predictive value (NPV) (proportion of negatives that are true-negatives) as a secondary measure of interest.

To evaluate the interhospital variability, we calculated summary statistics of MCC performance on all risk factors and outcomes, aggregated for each hospital in the sample. We then compared organizational factors between the highest-performing 10 hospitals to the lowest-performing 10 hospitals via Wilcoxon rank test for continuous variables and Fisher’s exact test for categorical variables. All tests were 2 sided with an α-level of .05.

Approval for this study was obtained through the Stanford University Institutional Review Board with a waiver of informed consent. Data analysis was performed using SAS (SAS Institute, Inc, Cary, NC), version 9.4, and Stata (Stata Corp, College Station, TX), version 15.0.

A total of 50 631 newborns were successfully linked out of 51 612 eligible records, representing a 98.1% linkage rate. Ranking of risk factors and outcomes by MCC value is shown in Fig 1, with graphical representation of PPV in relation to NPV shown in Fig 2.

MCC was >0 (positive correlation) for all risk factors, with an MCC >0.9 (very strong positive correlation) and PPV >90% for the risk factors of very low birth weight, cesarean delivery, and multiple-gestation delivery, as shown in Table 1. Similarly, an MCC >0.7 and PPV of 70% to 90% were observed for the risk factors of maternal hypertension, maternal diabetes, and PDA. MCC was <0.7 with a PPV <70% for the remainder of the risk factors evaluated. NPV was >90% for all risk factors. All variables other than in-hospital death revealed notable variation in PPV among hospitals as shown in Table 1. In particular, extracorporeal life support, pneumothorax, intraventricular hemorrhage, PDA surgery, and hypoxic-ischemic encephalopathy each exhibited a PPV of 100% in >25% of hospitals despite overall PPV values of <90% among the sample.

We also calculated 2 × 2 tables separately for risk factors coded in the maternal OSHPD record and the infant OSHPD record, as shown in Table 2. Cesarean delivery and multiple delivery revealed good precision, with a PPV of 98% or higher in both maternal and infant records. However, MCC and NPV were lower in the infant record compared with maternal record for all variables. Because a coded risk factor in either the maternal record or the infant record was considered to be a case as reported in Table 1, it was more common to have negative administrative items in the divided risk factors as reported in Table 2. For example, maternal diabetes was negative in 43 595 of maternal records and in 47 845 of infant records, but because of incomplete overlap, it was only negative in 43 244 of the combined records. This corresponds to a smaller number of true-negatives when compared with the 44 468 negative records from the CPQCC database. Similar trends were seen for hypertension and fetal distress. For cesarean delivery and multiple delivery, the maternal record captured all of the cases in the infant record, as well as a few additional cases, so no similar discrepancy existed for these items.

For patient outcomes, MCC was >0 for all, with MCC >0.9 only for in-hospital death. MCC >0.7 and a PPV of 70% to 90% were observed for PDA surgery, ROP surgery, extracorporeal life support, and intraventricular hemorrhage. MCC was <0.7 for the remainder of the outcomes evaluated, with MCC <0.4 for ROP and pneumothorax. NPV was >90% for all outcomes except chronic lung disease (86%), ROP (78%), respiratory distress syndrome (73%), and mechanical ventilation (70%).

Interhospital variability in performance as evaluated by mean MCC revealed similar performance among the majority of hospitals, with deviation at the extremes, as shown in Fig 3. There were no significant differences between the 10 highest-performing hospitals and the 10 lowest-performing hospitals, when comparing organizational factors encompassing patient volume, acuity, sociodemographics, and ownership (see Supplemental Table 4).

With this large, population-based study of NICU patients, we identified successes and opportunities for improvement in the use of administrative billing codes for perinatal risk factors and outcomes. Although many of the items evaluated performed well (nearly half of the MCCs were >0.7), we observed variation in performance across the items.

Risk factors and outcomes that are highly prevalent or easy to define revealed the most reliable coding in the OSHPD database. This finding is in line with those of Ford et al,¹⁰ who found particularly high accuracy of procedural coding in a similar study of 2432 infants but with more subjective diagnoses such as transient tachypnea performing poorly. With our findings, we suggest that usage of administrative data for very low birth weight designation, cesarean delivery, or multiple delivery determinations allows for similar results as usage of a clinical database. Similarly, mortality, mechanical ventilation, and ROP surgery remain highly congruent between the 2 types of databases. Conversely, diagnoses with subjective or complicated definitions, such as maternal chorioamnionitis, fetal distress, and chronic lung disease, performed less well. These entities that are more challenging to define are likely to represent high-yield targets of data improvement initiatives or incentive realignment.

Among infant risk factors, usage of a combination of the maternal and infant records reduced the false-negative rate over use of the infant record alone. This discrepancy was particularly pronounced for maternal factors that may not be as readily recognizable to neonatal providers, such as maternal chorioamnionitis, maternal hypertension, and maternal diabetes. It is suggested with these results that additional caution is needed when interpreting prevalence of maternal factors using infant records alone; with these results, the opportunity that arises when multiple records feature overlapping target areas is also highlighted. With increasingly ubiquitous electronic health records, it is conceivable that automated duplication of relevant codes between maternal and infant health records could be used to minimize discrepancies.

Of note, even the highest-performing metrics exhibited some degree of discrepancy between the 2 databases, indicating that all factors have some room for improvement. Very low birth weight, cesarean delivery, and multiple delivery all represent readily identifiable risk factors but featured hundreds of patients in each group with discrepant coding. Similarly, discrete and high-profile outcomes such as extracorporeal life support and mortality did not exhibit complete congruence. The reasons for these discrepancies are unclear, but their existence can be used to highlight the potential for improvement across the coding spectrum. This potential for improvement is particularly relevant for rare occurrences such as extracorporeal life support, because small numbers of discrepancies can represent a large proportion of rare cases. Additionally, although the degree of discrepancy among the more common high-performing factors is unlikely to materially affect analyses or conclusions at the population level, they can contribute to multiplicative effects when multiple factors are evaluated simultaneously.

Researchers of previous evaluations of administrative claims records have found them to exhibit use for identification of rare cases or clinical areas for improvement,^2,4,11 but less reliable for comparative evaluations or identification of absolute incidence.^3,12,13 In addition, ancillary financial data such as insurance coverage have been found to have poor reliability.¹⁴ However, unique characteristics of the NICU population lend themselves well to the usage of administrative claims data, including linkage between infant and maternal records and readily quantifiable risk factors such as birth weight and mode of delivery. This concept was recently illustrated by Howell et al¹⁵ in their analysis of maternal sociodemographic characteristics in relation to neonatal morbidity and mortality. Usage of linked maternal and infant data can also allow for increased power in the evaluation of population-level risk factors and expand horizons for health services research.

Similarly, comprehensive and accurate data are required for adequate performance metrics and quality improvement initiatives. Often, these goals are achieved through the development of clinical databases as part of quality care collaboratives. To be effective, these collaboratives must develop accurate methods of data collection, then seek to expand and become more comprehensive, as demonstrated by the CPQCC used in this study.⁷ However, many states and regulatory agencies lack the resources, regulatory mechanisms, or provider leadership to establish large clinical databases in this way, and even those with sufficient resources are unlikely to accomplish such a feat rapidly. With our findings, we suggest that an alternative approach is feasible, in which stakeholders instead seek to improve the accuracy of administrative data, leveraging the already-comprehensive nature of these databases. This alternative administrative-based approach could be used to support statewide quality improvement efforts such as the National Network of State Perinatal Quality Collaboratives supported currently by the Centers for Disease Control and Prevention.¹⁶ 

Compared with the creation of a novel quality care collaborative, improving the quality of data in existing administrative databases could be accomplished relatively efficiently through investigation, feedback, education, and incentive realignment. Data quality improvement should begin with a thorough investigation into the contributory factors to inaccurate coding, with the development of targeted improvement measures accordingly. Models already exist for feedback and education, including feeding back clinically relevant analyses of administrative data to clinicians and establishing regular data quality improvement seminars for medical coders.^7,17 Use of these data for performance metrics would also likely provide a strong incentive to accelerate data quality improvement efforts locally. If coupled with an implementation arm to effect change, the existing infrastructure may thus be leveraged to simultaneously improve outcomes for patients, increase efficiency for quality improvement programs, and reduce costs for public and private payers.

Although we observed interhospital variation among summary performance metrics, the magnitude of this variation was relatively small for the majority of the hospitals in this study. This suggests that current incentives are insufficient to promote an emphasis on high-quality administrative data collection at the majority of these sites. Implementing widespread data quality improvement efforts will thus be particularly important, because the results do not appear to be limited to any particular subset of hospitals.

This study must be interpreted in the context of its design. The probabilistic matching process carries the risk of false matches, which would be expected to bias the results toward lower PPVs. However, the large number of demographic variables shared between the 2 databases minimizes this possibility, and this matching process has shown good reliability in previous evaluations.¹⁸ Our usage of the CPQCC database as the gold standard for purposes of analysis reflects our objective to compare the administrative claims database to the currently available best alternative. Although CPQCC is likely not 100% accurate in coding, it employs extensive quality control measures and represents the most accurate currently available alternative to the usage of administrative claims records for this population, with definitions matching those used by the Vermont Oxford Network.^19,20 The ICD-9-CM and ICD-9-PCS codes used for case identification aimed for consistency with their usage in the current literature, but variations in case definitions may result in over- or underdiagnosis relative to our findings. Additionally, the transition to usage of International Classification of Diseases, 10th Revision may affect the accuracy of coding and merits additional evaluation once sufficient data exist with these codes. The performance of the hospitals in this sample did not associate with measured organizational factors, but other factors not measured in this analysis may affect coding performance, such as payer mix, coding policies, and training. Further evaluation will be needed to identify factors associated with high-accuracy coding.

Several important perinatal clinical risk factors and outcomes can reliably be identified in an administrative claims database, which may allow for the expansion of health services research investigations using these more readily available data. The successful use of administrative diagnosis and procedure codes needs to be based on a good understanding of what each code intends to capture and the consequences for the specific research question to be answered. Caution is needed when using administrative claims data for subjective or difficult-to-define diagnoses and when using infant records to identify maternal conditions. With these findings, we also highlight the opportunity for data quality improvement efforts, because the ability for accurate, comprehensive, and timely extraction of administrative inpatient data will be key to their usefulness as quality metrics.

CPQCC

California Perinatal Quality Care Collaborative

ICD-9-CM

International Classification of Diseases, Ninth Revision, Clinical Modification

ICD-9-PCS

International Classification of Diseases, Ninth Revision, Procedure Coding System

MCC

Matthews correlation coefficient

NPV

negative predictive value

OSHPD

Office of Statewide Health Planning and Development

PDA

patent ductus arteriosus

PPV

positive predictive value

ROP

retinopathy of prematurity

We acknowledge Beate Danielsen, PhD, for lending her expertise in data linkage.