Identifying and Validating Pediatric Hospitalizations for MIS-C Through Administrative Data

We identified pediatric hospitalizations for MIS-C using multiple algorithms that relied on a combination of diagnostic codes and medication administration data in the Pediatric Health Information System (PHIS; Children’s Hospital Association, Lenexa, KS),²⁷  an administrative database that contains inpatient, emergency department, ambulatory surgery, and observation encounter-level data from 49 not-for-profit, tertiary care pediatric hospitals in the United States. Data quality and reliability are assured through a joint effort between the Children’s Hospital Association and participating hospitals.

We compared these potential cases of MIS-C identified by each algorithm in PHIS to participating hospitals’ internal list of reported MIS-C cases, as well as targeted chart review. We included patients aged 1 month to 20 years hospitalized at 7 geographically diverse children’s hospitals in the United States from April 2020 through August 2021.

Two main data sources were used: administrative data (PHIS) and participating hospitals’ MIS-C lists with medical chart review. Potential MIS-C cases were identified by using diagnosis and medication administration data present in PHIS. PHIS contains daily billing data from large tertiary care children’s hospitals. A joint process between the Children’s Hospital Association, which manages PHIS, and each participating site ensures data quality. Site investigators at each participating hospital performed a focused medical record review on select cases of discordance between local and PHIS algorithm-generated MIS-C case lists.

Investigators at the 7 hospitals used their own hospital-maintained MIS-C lists to identify MIS-C hospitalizations at their institutions. These lists were used for public reporting to local and state agencies and were considered the initial reference standard for MIS-C identification. Clinical cases at all institutions were reviewed by pediatric infectious disease experts or by a multidisciplinary team, which included infectious disease experts along with rheumatologists and hospitalists, and were congruent with the Centers for Disease Control and Prevention guidelines.

For cases identified by PHIS algorithms not on institutional lists, the investigator conducted a medical record review to determine if the patient was diagnosed and treated for MIS-C during the hospitalization. If clinician documentation indicated that the child was diagnosed and treated for MIS-C, the hospitalization was considered an MIS-C case. Thus, the institution list with the addition of the identified MIS-C cases through medical record review was considered the final reference standard for each site.

Before 2021, MIS-C did not have an ICD-10 diagnostic code available. Therefore, our multidisciplinary team of pediatric hospitalists, hematologists, and infectious disease specialists developed algorithms and applied them to PHIS data in 2020. We developed the algorithms iteratively; after evaluating algorithms A to C, we used data on the accuracy of these algorithms to hone and improve subsequent algorithms D to H.

In 2020, the American Academy of Pediatrics recommended billing MIS-C hospitalizations using a combination of 2 ICD-10 codes: COVID-19 diagnosis (U07.1) and other specified systemic involvement of connective tissue (M35.8).²⁸  This code combination was designated algorithm A.

We included all encounters with COVID-19 as any discharge diagnostic code and pharmacy codes for treatment consistent with MIS-C. Specifically, we first identified all COVID hospitalizations in which either intravenous immunoglobulin (IVIG) or a biologic agent (anakinra, infliximab, or tocilizumab) was administered on the same hospital day, the day before, or the day after the administration of anticoagulant medication, such as aspirin (see Supplemental Table 6). These therapies were chosen as the initial treatment of MIS-C, which was extrapolated from knowledge of the treatment of Kawasaki disease.¹⁵  We did not include the administration of steroids in the diagnostic algorithm because steroids were administered for acute COVID-19 infection, potentially making distinguishing the 2 entities more challenging, and the evidence recommending routine steroid administration for MIS-C emerged after 2020.^8,10 

We examined hospitalizations in which IVIG and an anticoagulant were administered irrespective of diagnostic code. We a priori excluded hospitalizations for births, as well as diseases that commonly receive IVIG in their treatment, including any malignancy or complex chronic condition (CCC),²⁹  which includes diffuse diseases of connective tissue, cardiac technology, heart and great vessel malformation, and hematologic or immunologic diseases. Specific diseases within each excluded category of CCC were reviewed by subject matter experts to include (not exclude) diagnoses for which IVIG and anticoagulation are indicated; as such, we included Kawasaki disease. We also included malformation of coronary vessels because this may be a potential sequela of MIS-C.

Next, we examined primary discharge diagnoses of all hospitalizations meeting the cohort definition for algorithm C from January 2016 to June 2020 to exclude diagnoses that were unlikely to be MIS-C or Kawasaki disease. We focused on the time period before MIS-C was present and recognized because we knew these encounters could not be MIS-C. Four members of the research team reviewed diagnoses to determine exclusion, including 2 hematologists and 2 hospitalists. Individual diagnoses were removed if either both hematologists or both hospitalists recommended its removal. Disagreements were discussed with the entire research team. Finally, any primary diagnosis with <5 encounters within the multiyear dataset was excluded. Thus, the final algorithm C consisted of children receiving treatment of MIS-C, excluding diagnoses listed in the Supplemental Information.

We conducted a medical record review of all cases identified by algorithms A to C that were not listed on the institution’s MIS-C list (algorithm false positives). We abstracted primary discharge diagnosis and whether MIS-C was considered a potential diagnosis for the patient. Given the limited sensitivity of the 3 original algorithms, we subsequently developed additional algorithms to attempt to increase sensitivity. We also used learnings from the medical record review to apply additional restrictions to existing algorithms to increase agreement.

We created algorithm D, which was a summation of both cases that met the conditions of either algorithms A or B with the intent of maximizing sensitivity while potentially sacrificing accuracy. Any hospitalization identified by either algorithms A or B was combined in algorithm D.

Algorithm E captured hospitalizations in which treatment was administered: (either [IVIG + (either aspirin or an anticoagulant)] OR [(any of these 3: anakinra, infliximab, tocilizumab) + (either aspirin or anticoagulant)]) with either a diagnosis of COVID-19 or systemic involvement of connective tissue (M35.8).

We created algorithm F as a combination of algorithms A and B while excluding hospitalizations for which one might expect a patient to receive MIS-C-like treatment (eg, idiopathic thrombocytopenia, Guillain-Barre; Supplemental Information).

Algorithm G is a combination of algorithms A and E with the goal to increase the sensitivity.

For algorithm H, we excluded transplant hospitalizations and cancer diagnoses from algorithm G using CCC categories (Supplemental Information).

We identified hospitalizations with the MIS-C ICD-10 code M35.81 at 5 sites from January 2021 (when the ICD-10 code was first available) through August 2021 as a potential MIS-C hospitalization. Two sites were excluded because of unavailable PHIS data in 2021. We used the same methods as outlined in the 2020 algorithms above for identifying reference standard cases (ie, individual hospital MIS-C list and medical record review).

As with the 2020 algorithms, we attempted to improve the performance of the 2021 diagnostic algorithm (ICD-10 code) to reduce the number of false positives. We conducted a limited medical record review of false positive cases at each of the sites to identify potential diagnostic exclusions. We reevaluated the performance of the MIS-C diagnosis code excluding hospitalizations with transplant or oncologic diagnoses.

We determined the sensitivity, false positive rate, and positive predictive value (PPV) across each of the algorithms. Estimates of specificity are presented in the Supplemental Information and Supplemental Table 7. We examined these statistics in 2020 and 2021, separately. For the best-performing algorithms, we examined the algorithm performance across each hospital.

Across the 7 different sites, we identified a total of 245 cases for MIS-C in 2020 on the hospital MIS-C lists. In 2021, we identified 358 hospitalizations from January to August on the 5 hospital MIS-C lists. The case totals included 8 hospitalizations in 2020 and 20 hospitalizations in 2021 identified by PHIS algorithms that were not on the hospital MIS-C lists and were added to the reference standard list after medical record review. Of the hospitalizations on the final reference standard list, most children with MIS-C were 5 to 14 years old, male, had government insurance, and did not have any CCCs. Most children had a stay in the ICU and were treated with IVIG (Table 1).

The sensitivity of algorithms applied to data from 2020 ranged from 41% to 82% (Table 2). False positives ranged from 12% to 59%. PPV ranged from 41% to 88%.

The Phase 2 algorithms (D to H), tailored after initial chart reviews to remove conditions frequently identified as false positives, generally performed better than the Phase 1 algorithms (A to C). Algorithm F, which included records identified by algorithm A or B with exclusions for chronic conditions, had the highest PPV (88%), with 57% sensitivity. Across the 7 hospitals, algorithm F had sensitivities ranging from 46% to 75% (Table 2). False positive rates ranged from 0% to 25% across the hospitals. (Table 3)

Algorithm H, which combined algorithms A and E with additional exclusions for oncology and transplant diagnoses, had one of the highest sensitivities of all algorithms (82%), with a 22% false positive rate and 78% PPV (Table 2). Across hospitals, algorithm H had point estimates of sensitivity ranging from 54% to 93%. PPV ranged from 71% to 97%. (Table 3)

The sensitivity of the MIS-C ICD-10 diagnosis code was 98%, and the PPV of the code was 84% (Table 4). Medical record review of the 16% of cases that were false positives indicated that, in 86%, MIS-C was initially considered as a potential diagnosis, and in 63% of cases, treatment of MIS-C was initiated. Excluding hospitalizations with diagnosis codes for transplant or malignancy slightly decreased sensitivity and minimally improved the rate of false positives (Table 4). The sensitivity of the ICD-10 code was excellent across each hospital (point estimate range from 91% to 100%; Table 5).

Pediatric hospitalizations for MIS-C can be identified in claims data with high sensitivity, even before the introduction of an MIS-C diagnosis code in 2021. These codes can be applied to the entire PHIS dataset and other administrative databases, allowing for the identification of a larger cohort of children across a broader, more diverse group of hospitals in the United States.

Reliable methods to identify MIS-C hospitalizations back to the beginning of the pandemic will allow for longitudinal epidemiologic studies of MIS-C hospitalizations in relation to the presence of different variants and the introduction of pediatric COVID-19 vaccines. These algorithms will also allow for the investigation of any changes in severity of illness from MIS-C over time. For epidemiologic research, researchers may want to prioritize algorithms with the highest sensitivity, such as algorithm H in 2020 and the MIS-C diagnosis code in 2021 and beyond. As administrative datasets like PHIS move to updating data monthly, this can facilitate near-real-time epidemiologic research as SARS-CoV-2 variants, perhaps some with a differential risk of MIS-C,^30,31  continue to emerge. These methods will also allow researchers to examine comparative effectiveness through propensity-adjusted analyses of treatment regimens for MIS-C extending back to the earliest patient cohort. For comparative effectiveness research, consideration may be given to prioritizing algorithms with the lowest number of false positives/highest PPV, such as algorithm F in 2020 and the MIS-C diagnosis code in 2021 and beyond.

Moving forward, hospitals may reasonably use the MIS-C diagnosis code for the near-real-time identification of patients. Reliable identification of patients allows for the tracking of outcomes of patients during acute hospitalization, as well as fostering tracking for long-term follow-up. Clinicians can identify patients to track treatment and assess the concordance of therapy with guidelines. Finally, for busy community hospitals that may care for children hospitalized with MIS-C with relatively less severe disease or without on-site subspecialist consultation, hospital staff can confidently use these approaches to ensure they review all eligible hospitalizations for public reporting standards, to guide clinical pathway design and treatment, and for close follow-up of patients after discharge.

Across our geographically diverse hospitals, we identified modest differences in algorithm performance, particularly in the 2020 MIS-C population. We propose algorithm F may be the most useful for comparative effectiveness research or other types of research in which a low false positive rate is the highest priority. Four of the 7 hospitals had 0 false positives. Although false positives were as high as 25% at 1 center, we found that most of these patients were, in fact, treated for MIS-C before the clinical team settled on an alternate diagnosis. In contrast, algorithm H may be more useful when high sensitivity is desired, such as conducting epidemiologic research. Finally, another reasonable approach for research using these algorithms for 2020 is to identify patients using 1 algorithm (F or H, depending on the type of research question) and verify the main findings with the alternative algorithm. As expected, given its outstanding sensitivity overall, the MIS-C ICD-10 code had an excellent sensitivity across each of the individual 5 centers. The MIS-C code also performed better than each of the algorithms, likely driven by the existence of the more precise code as well as a generally improved understanding of MIS-C in 2021.

Our study had several limitations. First, our referent population relied on institutional identification of MIS-C patient lists, which depended on human identification and reporting of cases that may have varied, especially early in the pandemic. Because MIS-C is a clinical diagnosis without a specific diagnostic test, the identification of MIS-C in clinical care can be challenging. We also added a small number of hospitalizations to these lists for our final reference standard definition after identification of these cases by an algorithm and discordance noted on institutional lists. Second, all the algorithms had a moderate number of false positives, such that research using these methodologies should consider this a limitation to be considered in the interpretation of their findings. Third, although we had a geographically diverse sample of children’s hospitals, only 5 had both PHIS and reference standard data for 2021, making the generalizability of these findings somewhat more limited. All research utilizing diagnostic algorithms in administrative data has found some degree of false negatives and false positives;^25,26  this includes diagnoses like urinary tract infection that have clear diagnostic criteria and gold-standard testing with a urine culture.²⁴  In light of this limitation, when false positives are a threat to the research question, one could consider a 2-step analysis when examining 2020 hospitalization data using a more sensitive algorithm, such as the application of algorithm H and then verifying the findings with an algorithm with lower false positives, such as algorithm F.

We have identified high-sensitivity algorithms for use in epidemiologic research and high-PPV algorithms for use in comparative effectiveness research. Accurate algorithms to identify MIS-C hospitalizations throughout the timespan of the pandemic will facilitate research critical to understanding this novel entity and the delivery of optimal care to patients.

CCC

complex chronic condition

COVID-19

coronavirus disease 2019

ICD-10

International Classification of Diseases, Version 10

IVIG

intravenous immunoglobulin

MIS-C

multisystem inflammatory syndrome in children

PHIS

Pediatric Health Information System

PPV

positive predictive value

SARS-CoV-2

severe acute respiratory syndrome coronavirus 2