Using Administrative Billing Codes to Identify Acute Musculoskeletal Infections in Children

Data for this study were obtained from PHIS, an administrative database that contains inpatient, emergency department, ambulatory surgery, and observation encounter-level data from over 50 not- for-profit, tertiary care pediatric hospitals in the United States. These hospitals are affiliated with the Children’s Hospital Association (Lenexa, KS). Data quality and reliability are assured through a joint effort between the Children’s Hospital Association and participating hospitals. For the purposes of external benchmarking, participating hospitals provide discharge and encounter data, including demographics, diagnoses, and procedures. Nearly all of these hospitals also submit resource utilization data (eg, pharmaceuticals, imaging, and laboratory) into PHIS. Data are deidentified at the time of data submission and are subjected to a number of reliability and validity checks before being included in the database.¹⁸  The PHIS database was queried for inpatient and observation encounters for patients between 6 months and 18 years of age who were admitted between April 1, 2016, and March 31, 2020. Patient encounters were included in the primary cohort if they were assigned a primary or a secondary discharge diagnosis of osteomyelitis, septic arthritis, and/or pyomyositis based on International Classification of Diseases, Tenth Revision (ICD-10) diagnostic codes (Supplemental Table 4). All ICD-10 Clinical Modification codes within diagnostic categories A and B (Infectious and Parasitic Diseases), L (Skin and Subcutaneous Tissue), and M (Musculoskeletal and Connective Tissue) were reviewed for any possible codes associated with infectious osteomyelitis, septic arthritis, and pyomyositis to ensure the list of included patients was comprehensive.¹⁹  Six PHIS-participating hospitals were included in this study (Children’s Hospital Colorado, Aurora, Colorado; Seattle Children’s Hospital, Seattle, Washington; Monroe Carell Jr Children’s Hospital at Vanderbilt, Nashville, Tennessee; Children’s Hospital of Philadelphia, Philadelphia, Pennsylvania; Children’s Hospital at St. Louis, St. Louis, Missouri, and Cincinnati Children’s Hospital Medical Center, Cincinnati, Ohio). Sites were chosen to include a diverse mix of geographic regions in the United States. Any encounter in which the patient had been hospitalized for an MSKI within the previous 180 days was excluded from the PHIS query.

To ensure our list of included MSKI ICD-10 diagnostic codes was comprehensive and that no MSKI patients were missed because of coding errors, an additional, separate PHIS query (secondary cohort) was also performed to identify patient encounters with diagnostic and therapeutic characteristics similar to MSKI encounters but without any MSKI ICD-10 discharge diagnoses assigned (Supplemental Table 4).

For the purposes of this study, an acute hematogenous MSKI without contributing comorbidities (AH-MSKI) was defined as a child between 6 months and 18 years of age diagnosed and treated for an acute onset (<2 weeks symptoms) of bacterial osteomyelitis, septic arthritis, and/or pyomyositis without any of the following comorbidities that would contribute to disease pathogenesis: decubitus ulcers; neurologic comorbidity leading to significant decrease in mobility; sickle cell anemia; indwelling musculoskeletal hardware at site of infection; central venous catheter in place before onset of MSKI; surgery to affected extremity within 30 days of symptom onset; infection of the head, orbits, or central nervous system; immunocompromising conditions as defined by Infectious Diseases Society of America Immunocompromised Host guideline²⁰ ; open fracture; penetrating trauma to site of infection; or atypical infections caused by Borrelia burgdorferi, Neisseria gonorrheae, Brucella species, mycobacteria, endemic fungi, Coxiella burnetii, Actinomyces species, or Nocardia species. The above comorbidities and atypical pathogens routinely undergo notably different medical and surgical treatment plans because of differences in disease pathogenesis, complication rates, and expected antibiogram. Therefore, accurately labeling patients with these comorbidities is needed to exclude outliers and identify the population of interest for future MSKI research. Patients who did not meet the above criteria were classified as having a not acute hematogenous MSKI (NAH-MSKI).

Available PHIS data extracted included all primary and secondary discharge diagnoses, demographic data, hospital length of stay, surgical procedures performed, dates and classes of antimicrobial agents received, and time spent in an ICU.

We performed a sample size calculation to determine the number of encounters needed to review for algorithm validation. Assuming a 1:1 ratio of cases to controls (ie, AH-MSKI to NAH-MSKI) would be included via reference standard chart review, to estimate a sensitivity of 80% for the proposed algorithms with a margin of error of 5% and 95% confidence level, we determined we needed to review 492 total patient encounters. Each of the participating PHIS sites reviewed 82 randomly selected encounters from their center. In addition, each site reviewed all encounters from their site-specific “secondary cohort” to ensure there were no missed MSKI.

Each study site underwent a 1-hour virtual training session on chart review instructions for consistency. Each site was provided with a training document to reference the described criteria for what constituted an AH-MSKI versus a NAH-MSKI. Reviewers at each site evaluated provider documentation for encounters using their site-specific electronic medical record system and labeled each encounter as either an AH-MSKI or NAH-MSKI. Each site was instructed to review discharge summaries, history and physical documentation, and infectious diseases or orthopedic consultation notes to determine whether a patient was diagnosed and treated for a bacterial musculoskeletal infection. Reviewers used the diagnostic decisions of the original treating team as clinical truth and did not retrospectively alter discharge diagnoses. AH-MSKI encounters had data extracted via manual chart review for causative bacterial pathogens based on all microbiological data available in the electronic medical record.

If an encounter was ambiguous for determination of AH-MSKI versus NAH-MSKI, reviewers flagged the encounter for additional adjudication. All flagged charts underwent a collaborative discussion (without sharing protected health information) with the coordinating PHIS site to determine whether the encounter should be classified as AH-MSKI or NAH-MSKI. After manual chart reviews were completed, all 492 randomly reviewed encounters underwent additional review by the primary PHIS site leader to identify potential misclassifications, including unusual hospital lengths of stay, isolation of a pathogen rarely associated with AH-MSKI, or discharge diagnoses not typical of AH-MSKI. Any patient encounters flagged as possible misclassifications then underwent a second manual chart review by site leads to ensure accurate categorization for AH-MSKI versus NAH-MSKI for the reference standard. These manual chart review classifications for each encounter were used as the reference standard for algorithm comparison.

All collected information was stored in the coordinating PHIS site’s institutional Research Electronic Data Capture (REDCap) database.²¹  Data access groups were used within REDCap to ensure site privacy, and no decrypted protected health information was stored in the REDCap database. Each site received approval for this retrospective cohort study through their respective institutional review board.

Nineteen categories of medical conditions or procedures typically assigned to NAH-MSKI were developed. A list of relevant ICD-10 discharge diagnoses or Current Procedural Terminology codes was identified for each of these nineteen categories (Supplemental Table 5). A binary assignment (presence or absence) for each of these 19 indicator categories was given to each PHIS encounter. In addition to ICD-10 discharge diagnoses and Current Procedural Terminology codes assigned to each encounter, hospital length of stay, antimicrobial classes administered, complex chronic condition (CCC) classification flag,²²  immunomodulating medications received during admission, disposition location, and encounter type (inpatient versus observation) were used to develop each algorithm (Supplemental Table 6). Ten diagnostic algorithms were developed using clinical reasoning and were further informed by the preliminary relative performance (a basic ratio of false negatives to true negatives) of individual categories using a small subset of the sample. Individual algorithms were designed to maximize sensitivity, maximize specificity, or balance both, and 10 algorithms were developed to offer several comparable options for each of those 3 purposes.

Characteristics of the total study cohort queried from PHIS and the random subset sampled for chart review were described using frequencies and proportions for categorical variables and median and interquartile range (IQR) for continuous variables. Using manual chart review designation as the reference standard, the performance of each of the 10 algorithms was determined by calculating its sensitivity, specificity, and receiver operating characteristic (ROC) area under the curve (AUC), which for a binary predictor is equivalent to the average between sensitivity and specificity when calculated using a strategy accounting for ties. Variation across sites in chart review practices and performance of the algorithms was assessed by calculating their sensitivity, specificity, and ROC AUC for each site and comparing the 95% confidence intervals (CIs). Sensitivity and specificity for each algorithm were also compared across sites using χ² or Fisher’s exact tests of association. Analyses were performed using R (version 4.1.1, 2021).

A total of 3012 encounters (2757 inpatient admissions and 255 observation encounters) from the 6 participating sites were identified for the primary cohort (Fig 1A). Patient demographics, MSKI diagnoses assigned, and clinical characteristics are shown in Tables 1 and 2. Thirty eight percent of patients had osteomyelitis alone, 16% had septic arthritis alone, 26% had pyomyositis alone, and 20% of patients had more than one MKSI diagnosis assigned (Table 1). A total of 492 randomly selected hospital admissions (82 per site) were assigned reference standard of AH-MSKI or NAH-MSKI status via manual chart review. Of these, 244 (49.6%) were cases of acute hematogenous MSKI without any contributing comorbidities (AH-MSKI) and 248 (50.4%) were either nonbacterial MSKI, non-MSKI, subacute, chronic, or nonhematogenous MSKI, or were associated with significant contributing comorbidities (NAH-MSKI). Most AH-MSKI (84.4%) had osteomyelitis, septic arthritis, or pyomyositis as the primary discharge diagnosis (as opposed to a secondary diagnosis), whereas only 39.9% of NAH-MSKI encounters had MSKI as a primary discharge diagnosis (Table 2). Classification as either nonbacterial or non-MSKI (eg, viral myositis, superficial cellulitis only) was the most common reason to be classified as NAH-MSKI, followed by chronic infections, decubitus ulcers, and severe neurologic comorbidity (Fig 1A).

Of the 244 AH-MSKI encounters, a pathogen was identified in 146 (59.8%); institution-specific rates of pathogen identification ranged from 53.5% to 71.8%. Among patients with a pathogen identified, Staphylococcus aureus was the most common bacterium (72.6%) and rates of methicillin resistance among S. aureus isolates ranged from 6.7% to 41.2% at the 6 PHIS hospitals (Table 2). Median hospital length of stay was 4 days, and 4.9% of patients required intensive care. A majority of AH-MSKI patients (59%) underwent at least 1 surgical procedure during admission, though there was variability in rate of surgical procedures across PHIS sites (47.7% to 72.2%).

There was variable performance of the 10 algorithms with sensitivity ranging from 31.1% to 91% and 199 specificity ranging from 52.4% to 97.6% (Table 3 and Fig 2). Four algorithms demonstrated potential for future use with ROC AUC greater than 80% (Algorithms 1, 2, 3, and 10). Balancing sensitivity and specificity equally, the highest performing algorithm (Algorithm 3) had a sensitivity of 79.9% and specificity of 86.7% (Table 3). Algorithm 8 had the highest specificity (97.6%) but was associated with the lowest sensitivity (31.1%).

Individual algorithm ROC AUC was comparable across all PHIS sites with overlapping CIs. Sensitivity and specificity were also generally similar across sites. However, several differences in algorithm performance were noted with Site 5 demonstrating a lower sensitivity for Algorithms 1 (P < .001), 3 (P = .002), and 9 (P = .002) compared with the other sites and Site 1 having a lower specificity for Algorithm 8 (P = .04) compared with the other sites. There were no other significant differences in any algorithm sensitivity or specificity among the 6 PHIS sites (Supplemental Fig 3).

A total of 208 encounters (190 inpatient and 18 observation) were identified for the secondary cohort that had characteristics of an AH-MSKI (eg, fever, elevated C-reactive protein, imaging of an extremity, receipt of an antimicrobial) but without any MSKI ICD-10 discharge diagnoses assigned (Fig 1B, Supplemental Table 3). All 208 hospital admissions in the secondary cohort underwent manual chart review at their respective PHIS site. Nearly all secondary cohort encounters were classified as NAH-MSKI (97.6%). There were 5 encounters in the secondary cohort that were classified via manual chart review as AH-MSKI and were missed using our primary cohort MSKI discharge diagnostic codes that found 3012 MSKI encounters.

To date, harnessing administrative data to study acute hematogenous musculoskeletal infections in children has been challenging because of limited specificity of discharge diagnoses for this population. This study is the first to develop and evaluate the performance of several case-finding algorithms designed to reliably identify children hospitalized for acute hematogenous MSKI using administrative data alone. Several algorithms showed promise with sensitivities and specificities greater than 80% and consistent performance across multiple, geographically diverse PHIS hospitals. Nearly half (50.4%) of those patients assigned a discharge diagnosis of osteomyelitis, septic arthritis, and/or pyomyositis were found in manual chart review to have chronic infections, nonhematogenous sources, or significant contributing comorbidities. Therefore, when used in isolation, ICD-10 discharge diagnosis codes are an inaccurate method to identify children hospitalized with acute hematogenous MSKI, and case-finding algorithms like ours are needed to study acute hematogenous MSKI among hospitalized children in PHIS to prevent erroneous conclusions.

When equally balancing sensitivity and specificity, Algorithms 1, 2, 3, and 10 were the highest performing algorithms with Algorithm 3 demonstrating the highest ROC AUC (83.3%). These 4 algorithms shared several similar features that might offer insight into their performance. First, each algorithm excluded patients with short hospitalizations (1 day) and those with unusually long hospitalizations of more than 30 days (Algorithms 1–3) or more than 14 days (Algorithm 10). Second, each used a robust list of excludable discharge diagnoses likely to be assigned to patients without acute hematogenous MSKI, including chronic recurrent multifocal osteomyelitis and decubitus ulcers. Lastly, these algorithms excluded patients who never received an antimicrobial and who had a CCC flag in PHIS.

When comparing algorithm performance, we found use of this CCC flag was a high-yield addition to identify patients with likely excludable conditions. When using a narrowed list of acute MSKI discharge diagnoses (Algorithms 4 and 8), algorithm specificity was greatly increased (up to 97.6%), but at the cost of a reduced sensitivity (down to 31.1%). This suggests that PHIS-based studies limited to a narrow list of ICD-10 discharge diagnoses will fail to identify a significant proportion of inpatient acute hematogenous MSKI encounters, leading to biased results. Depending on the clinical outcomes and population of interest, administrative database studies of acute hematogenous MSKI could select an algorithm tailored to the population of interest. For instance, a study hoping to prioritize exclusion of patients with chronic or nonhematogenous infections would opt for a highly specific algorithm, and therefore Algorithm 8 may be ideal in such a scenario. Similarly, a study interested in only certain MSKI processes, such as osteomyelitis alone for instance, could easily tailor the included ICD-10 codes in these algorithms to capture only those patients of interest while still excluding nonacute or nonhematogenous infections.

When comparing algorithm performance across institutions, the ROC AUC for each algorithm was similar with overlapping CIs across all PHIS sites. Sensitivity and specificity for each algorithm were also mostly similar across sites. However, Site 5 had slightly lower sensitivity observed for Algorithms 1, 3, and 9 and Site 1 had slightly lower specificity for Algorithm 8. It is possible that inconsistency in manual chart review contributed to these findings, though efforts were extensive to ensure consistency at each site. There is also the potential for variable coding practices among institutions which would lead to variable algorithm performance. However, given the large number of intersite comparisons tested for both sensitivity and specificity of 10 algorithms, differences of at least 1 algorithm may be likely across sites because of random sampling error alone. In addition, the lower specificity for Algorithm 8 at Site 1 was because of 4 false positives compared with 0 or 1 at the other sites, a minor difference. In addition, these algorithms are based upon readily available administrative data that are used in many different healthcare databases (eg, discharge diagnoses, length of stay, and administered medications). Therefore, these algorithms are not limited to PHIS studies alone but can now be leveraged for future research to study acute musculoskeletal infections in children using many readily available pediatric administrative databases. These algorithms can now be harnessed to describe the national variability in pediatric MSKI treatment and address lingering questions relevant to this population, including the impact of early operative procedures on clinical outcomes and the burden of MSKI on hospital resources.

There are several limitations to this study. First, algorithms were developed using data from 6 of 52 participating centers in PHIS and may not be applicable to the entire PHIS dataset or other nonfreestanding pediatric hospitals. However, because we included a wide range of geographic regions across the United States, we feel our results are robust and representative. The overall consistent algorithm performance across sites supports the generalizability of these algorithms to other PHIS sites. Furthermore, it is possible that a patient treated for an acute hematogenous MSKI might be missed by these algorithms because of coding errors or misclassification of discharge diagnoses. To evaluate this, we performed manual chart reviews on a secondary cohort of inpatient encounters that met billing code criteria typical for patients with AH-MSKI, but that were not assigned a discharge diagnosis for MSKI. Among the 208 patients in this secondary cohort, only 5 additional AH-MSKI were identified via manual chart review. Compared with the validation subset of the primary cohort where 49.6% of identified encounters were AH-MSKI, only 2.5% of those in the secondary cohort were AH-MSKI. This suggests that the included ICD-10 diagnostic codes for MSKI in these PHIS algorithms are comprehensive and unlikely to miss a significant number of acute hematogenous MSKI encounters. Another potential limitation is that only a subset of patient charts was manually reviewed to develop our reference standard. It was not feasible to review all patient charts for children with MSKI diagnostic codes at each participating center; however, a large randomly selected sample of patients was used for algorithm validation.

This study successfully developed and validated several case-finding algorithms to accurately identify patients with acute hematogenous MSKI in PHIS while excluding those with complicating comorbidities. Future studies will now leverage these algorithms to identify acute hematogenous MSKI encounters based on administrative billing data to compare variable diagnostic and therapeutic strategies for children hospitalized with acute hematogenous MSKI across PHIS hospitals. In addition, our approach offers a methodologic example for how to validate other case-finding algorithms using administrative billing codes for specific pediatric diagnoses.