Utility of Birth Certificate Data for Evaluating Hospital Variation in Admissions to NICUs

We obtained linked birth cohort files from the California Office of Statewide Health Planning and Development, which contained Vital Statistics birth (VSB) (ie, birth certificate) data elements for all live births in the state with linked information from hospital discharge records. Data from 2012 were used for this analysis because it was the latest year the linked birth cohort file was available. We used a standard birth certificate data element in VSB reporting “NICU admission,” which indicates whether an infant was admitted to a level III or IV NICU during the birth hospitalization.¹² 

All hospitals with an accredited NICU in California are required to report data on live births and NICU admissions to CCS. Hospitals self-report these data to the California Perinatal Quality Care Collaborative (CPQCC), which facilitates the data collection for CCS. We acquired hospital-specific summary data from the CPQCC regarding the number of inborn live births and NICU admissions among inborn live births in 2012. We assumed that the CCS data provided more accurate information on NICU admissions and used CCS data as the gold standard in analysis.

The VSB and CCS data were linked by hospital. There were a total of 103 hospitals in the CCS data with level III or IV NICUs that had at least 1 inborn live birth and confirmed information on birth volume. We successfully linked all 103 hospitals between the 2 data sources. This study was approved by the Yale University Human Investigation Committee, Stanford University Administrative Panel for the Protection of Human Subjects, and California Committee for the Protection of Human Subjects.

For each hospital, we calculated its NICU admission rate: (number of NICU admissions among inborn live births at this hospital/number of inborn live births at this hospital) × 100. Separate calculations were made for VSB data and CCS data. To evaluate the accuracy of the VSB-based admission rate, we calculated the difference between the VSB-based NICU admission rate and CCS-based rate and assessed its statistical significance using the Wilcoxon signed rank test. Additionally, to evaluate whether the VSB-based rate may reasonably inform a hospital’s performance on NICU admissions, we examined Lin’s concordance correlation coefficient and Kendall’s W concordance coefficient. Lin’s concordance correlation coefficient measures how well the VSB-based NICU admission rate agrees with the CCS-based NICU admission rate (ie, concordance between rates),¹³  whereas Kendall’s W concordance coefficient is a nonparametric statistic reflecting how well hospital ranks that are based on the 2 rates agree with each other (ie, concordance between ranks).

We considered Lin’s concordance correlation coefficient ≥0.90 as the threshold for good concordance (versus poor concordance) between the VSB- and CCS-based NICU admission rates¹⁴  and identified a subset of hospitals that demonstrated good concordance between the 2 rates. This was achieved by calculating and comparing Lin’s concordance correlation coefficients for subsets of hospitals when incrementally restricting to hospitals that had a smaller difference between VSB- and CCS-based rates (eg, ≤4 percentage points, ≤3 percentage points, etc). We compared institutional characteristics between good- and poor-concordance hospitals using χ² tests and Wilcoxon rank sum tests. Institutional characteristics assessed included teaching status, ownership type, multihospital system affiliation, NICU level, and delivery volume. NICU level was based on hospital self-designated information submitted to the CPQCC and reflected the highest-acuity nursery within each hospital. Delivery volume was based on CCS data. Other measures were obtained from the 2012 American Hospital Association Annual Survey Database.¹⁵ 

P <.05 was considered statistically significant. Analyses were performed by using SAS version 9.4 (SAS Institute, Inc, Cary, NC) and SPSS version 26 (IBM SPSS Statistics, IBM Corporation).

Among the 103 hospitals in the overall sample, NICU admission rates from the VSB data (median = 7.72%; interquartile range [IQR]: 5.27%–10.93%) were lower than those from the CCS data (median = 11.51%; IQR: 8.80%–14.64%; P < .001; Table 1). NICU admission rates based on these 2 data sources had a Lin’s concordance correlation coefficient of 0.33 (95% confidence interval [CI]: 0.19–0.46) and Kendall’s W concordance coefficient of 0.78 (P < .001).

When restricting to a subset of 35 hospitals where the difference in NICU admission rates between VSB and CCS data was no more than 2.1 percentage points, there was good concordance between the 2 data sources. Among these hospitals, Lin’s concordance correlation coefficient was 0.91 (95% CI: 0.84–0.95) and Kendall’s W concordance coefficient was 0.99 (P < .001; Table 1, Fig 1).

The 35 hospitals with good-concordance data are diverse in characteristics, with 51.4% being teaching hospitals, 77.1% having private nonprofit ownership, 77.1% being affiliated with a multihospital system, and 88.6% and 11.4% having a level III and IV NICU, respectively (Table 2). Their annual delivery volume ranged from 1277 to 6465. These hospitals did not differ significantly from poor-concordance hospitals in the institutional characteristics assessed.

Our data showed that birth certificates tended to underreport NICU admissions. This may be because of late-onset NICU admissions after birth certificates have been completed, poor recording of transient NICU care, or possible differences in the definition of NICU bed levels across hospitals. The large variation across hospitals in the accuracy of the NICU admission measure found in our study is consistent with other research showing variation in the sensitivity of this measure in identifying NICU admissions across states.¹⁶ 

However, in a selected subset of hospitals, we found good concordance between VSB-based and CCS-based NICU admission rates. Birth certificates from these hospitals may provide a reasonable means for studying NICU admissions. Although these good-concordance hospitals only reflect one-third of all hospitals assessed, they have diverse characteristics that are comparable to other hospitals. This finding is important because birth certificate data are often accessible across hospitals and linkable to diagnosis and procedure information in hospital discharge records. Linked birth certificate and hospital discharge data from these good-concordance hospitals can offer an important opportunity for studying infant diagnoses associated with NICU use and help identify possible overuse, underuse, or misuse.

Despite poor accuracy in documenting the exact rate of NICU admissions, hospital rank based on birth certificate data demonstrated reasonable concordance with CCS data. Kendall’s W concordance coefficient (measuring the extent of agreement between hospital ranks that are based on NICU admission rates from birth certificates and the gold standard CCS data) was 0.78 in the overall sample and 0.99 in the subset of good-concordance hospitals. This supports another potential use of birth certificate data: to reflect the relative performance of a hospital on NICU admissions across institutions. The data may be used as a performance metric for benchmarking hospitals or facilitating more targeted research (eg, in-depth evaluation of hospitals with high versus low NICU use).

Our analysis was based on data in 2012 from a single state. Because data quality may have improved in more recent years or differ across geographic regions, continued monitoring and evaluation of birth certificate data are warranted. We also recognize that our analysis was an ecological evaluation of NICU admission at the hospital level, not at the individual patient level. Although further efforts are needed to enhance accuracy, birth certificate data provide important information and opportunities for studying newborn care. Identifying hospitals with a good-quality birth certificate measure of NICU admissions and using their data to understand differences in hospital practices may offer a reasonable means to inform strategies for improving NICU services.