Prediction Models for Intravenous Immunoglobulin Resistance in Kawasaki Disease: A Meta-analysis

This systematic review and meta-analysis followed the Preferred Reporting Item for Systematic Review and Meta-analysis (PRISMA) guidelines.¹⁴  We developed a detailed protocol and registered our study with the International Prospective Register of Systematic Reviews (PROSPERO: CRD42021283276).

We included all prospective and retrospective cohort studies. We did not apply language or country restrictions. We included all papers, including published articles, unpublished articles, conference abstracts, and letters. We did not include case-control studies (including nested case-control), case reports, and case series. We did not exclude studies based on the observation period or publication year.

We included studies reporting any patients diagnosed with KD and undergoing initial IVIG therapy. Studies including participants with initial IVIG doses other than the standard treatment of 2 g/kg^2–4  or those without reported IVIG doses and studies including patients using drugs other than aspirin concurrently with initial IVIG therapy were excluded.

We included studies aiming to develop, validate (internally or externally), or update prediction models for initial IVIG resistance in patients with KD and prediction models using available predictors in clinical practice before starting IVIG administration. We a priori decided to include models that were not originally developed to predict IVIG resistance but commonly used or validated to predict the outcome. We excluded studies aiming to establish independent prognostic factors associated with IVIG resistance and not to develop a prognostic model by combining 2 or more predictors.

IVIG resistance is generally defined as a recrudescent or persistent fever at least 36 h²  or 24 h^3,8  after IVIG administration completion. Other definitions, such as the attending physician’s judgment or a recrudescent or persistent fever for at least 48 h, were also accepted.

We searched the following databases: Medline (PubMed), the Cochrane Central Register of Controlled Trials (Cochrane Library) and Embase (Dialog) from inception to October 2021. For the assessment of reporting bias, we also searched the following databases for ongoing or unpublished trials: the World Health Organization International Clinical Trials Platform Search Portal and ClinicalTrials.gov. Supplemental Table 2 shows the details of search terms for each database. We checked the reference lists of studies, including international guidelines, such as American Heart Association,²  JCS/JSCS 2020,³  and SHARE recommendation,⁴  and the reference lists of eligible studies and articles that cite eligible studies. The authors of the original studies were contacted when a study was uncertain to be eligible.

Two of 6 independent reviewers (Y.K., M.B., S.T., T.A., N.T., and H.T.) screened titles and abstracts, followed by an eligibility assessment based on the full texts. We contacted the original authors if relevant data are missing. Disagreements between the 2 reviewers were resolved by discussion, with a third reviewer as an arbiter for failed discussion. Study selection was documented in a detailed flowchart based on the PRISMA guidelines.¹⁴ 

Two of 6 independent reviewers (Y.K., M.B., S.T., T.A., N.T., and H.T.) extracted data from the selected articles following the critical appraisal and data extraction for systematic reviews of prediction modeling studies checklist.¹⁵  We categorized the types of model analyses as follows: development model analysis with or without internal validation, validation analysis with a dataset that combines the original dataset for derivation and another dataset, internal-external validation, and external validation analysis.¹⁶  The internal-external validation analysis was defined as a validation using a dataset unused for model development in the same study. External validation analysis used datasets from a different study. Discrimination (C–statistic), calibration (observed–to–expected [OE] ratio, calibration–in–the–large and slope, calibration plot, and Hosmer–Lemeshow test), and classification (sensitivity, specificity, positive predictive value [PPV] and negative predictive value [NPV]) were separately extracted for each model if a study reported 2 or more models with different predictors or prediction rules. Cut-offs of 3,¹⁷  4,⁸  or 5^3,18  points were used for external validation of Kobayashi scores. Studies without the cutoff values were classified as 4 or 5 points based on cited articles.^8,18 

We used the Prediction model Risk of Bias Assessment Tool (PROBAST) to assess the risk of bias and applicability of each model analysis.¹⁹ 

Two of 6 independent reviewers (Y.K., M.B., S.T., T.A., N.T., and H.T.) independently assessed these individual studies. The questions assessed the risk of bias for all analyses reported in the selected articles following the 4 domains, namely, participants, predictors, outcome, and analysis. The applicability was assessed through the questions following the 3 domains, namely, participants, predictors, and outcome. We evaluated each domain for risk of bias as “high-risk of bias,” “unclear risk of bias,” “low-risk of bias,” and for applicability as “high concern,” “unclear concern,” and “low concern.” An overall judgment was evaluated as “high-risk of bias” or “high concern” if at least 1 domain of risk of bias or applicability was “high.” Any disagreements were resolved by discussion, with a third reviewer as an arbiter for failed discussion.

We applied a random-effects model that pools logit transformations of C–statistics. We adopted the restricted maximum likelihood estimation and the Hartung–Knapp–Sidik–Jonkman.²²  We used an inverse-variance-weighted random-effects model with the DerSimonian-Laird estimator to estimate the between-study variance and normal approximation intervals based on summary measures to calculate CIs for individual study results and estimate the pooled PPV and NPV.²³  To stabilize variances, we transformed the data with the Freeman–Tukey double arcsine transformation.

We computed the 95% prediction interval, which provided a range for the potential performance of a model in a new validation study. We also computed I² and τ² statistics to quantify the extent of the heterogeneity. All statistical analyses were performed using R software (version 4·0·3) including “meta,” “metamisc,” and “metafor” packages.

Study location (East Asia, Middle East Asia, Europe, or North America) or timing of outcome definition (recrudescent or persistent fever at least 24, 36, or 48 h after IVIG administration completion) subgroups were analyzed to explore the cause and extent of heterogeneity.

Figure 1 displays a PRISMA 2020 flowchart detailing the study selection process, including the reason for exclusion. We screened 2043 records after removing duplicates and assessed 193 full texts. The clinical trial registry system screened 27 records, and no records were included. We excluded 25 studies having participants with initial IVIG doses other than 2 g/kg or those without reported IVIG doses,^24–26  and 9 studies with patients using drugs other than aspirin concurrently with initial IVIG therapy (Supplemental Table 3). Finally, the analyses included 48 studies.^{8–13,17,18,27–66} 

Supplemental Table 4 shows the characteristics of the 48 included studies that evaluated 161 model analyses. Of which, 35, 6, 4, and 3 were reported from East Asia, Middle East Asia, North America, and Europe, respectively. Calibration performance was unavailable in most studies.

We applied the PROBAST tool to the 161 model analyses (Supplemental Fig 5, Supplemental Table 5). All development model analyses were classified as an overall “high-risk of bias.” All analyses were rated “high-risk of bias” in the analysis domain because of (1) insufficient number of outcome events relative to the parameters among candidate predictors (low events per variable), (2) inappropriate missing data handling, (3) inappropriate selection, and (4) no calibration. Most development model analyses were classified as “low concerns of applicability” (94%; 48 of 51).

All external validation analyses were classified as “high-risk of bias” because of (1) inappropriate missing data handling and (2) no calibration. Approximately two-thirds (74 of 110) of external validation analyses were classified as “low concern of applicability.”

Models with 3 or more external validation analyses in different studies are the Kobayashi, Egami, Sano, Formosa, and Harada scores (Supplemental Table 6). The Kobayashi,⁸  Egami,⁹  Sano,¹⁰  and Harada⁶⁷  scores were developed in Japan, whereas the Formosa³⁷  score was developed in Taiwan. The Harada Score⁶⁷  was originally developed in 1961, not to predict IVIG resistance, but to predict CAA development and determine whether to implement IVIG therapy. However, several studies have evaluated the external validity of the Harada score as a model for predicting IVIG resistance; thus, it was included in the evaluation in this review. Table 1 presents a result summary of the external validation analyses of the 5 prediction models. The included analyses in the meta-analysis revealed a 14% (12% to 20%) median (interquartile range) prevalence of IVIG resistance.

This meta-analysis included 35 studies and 33 external validation analyses (Supplemental Table 6). Figure 2A shows low discrimination (C–statistic: 0.65; 95% CI: 0.57–0.73), without differences in the C–statistic between studies from East and Middle East Asia (Supplemental Table 7). Europe has only 1 and North America has no report. Additionally, the C–statistic between studies with different IVIG response determination timing has no difference (Supplemental Table 7). PPV was higher at the cutoff score of 5 points (0.39; 95% CI: 0.32–0.46) (Fig 3A), whereas NPV was higher at the cutoff score of 4 points (0.89; 95% CI: 0.84–0.92) (Fig 4B).

This review included 31 studies and 30 external validation analyses (Supplemental Table 6). Figure 2B shows low discrimination (C–statistic: 0.63; 95% CI: 0.55–0.71), without differences in the C–statistic between studies from East and Middle East Asia (Supplemental Table 7). Europe has only 1 and North America has no report. Additionally, the C–statistic between studies with differentIVIG response determination timing has no difference (Supplemental Table 7). Figure 3C and Fig 4C shows low PPV (0.27; 95% CI: 0.18–0.37) and high NPV (0.88; 95% CI: 0.83–0.91).

This review included 21 studies and 20 external validation analyses (Supplemental Table 6). Figure 2C shows low discrimination (C–statistic: 0.58; 95% CI: 0.55–0.60), without differences in the C–statistic between studies from East Asia and Middle East Asia (Supplemental Table 7). Europe and North America have no reports. Additionally, the C–statistic between studies with different IVIG response determination timing has no difference (Supplemental Table 7). Figure 3D and Fig 4D shows low PPV (0.35; 95% CI: 0.26–0.45) and high NPV (0.85; 95% CI: 0.77–0.91).

This review included 9 studies and eight external validation analyses (Supplemental Table 6). Figure 2D shows low discrimination (C–statistic: 0.50; 95% CI: 0.36–0.63), without differences in the C–statistic between studies from East Asia and Middle East Asia (Supplemental Table 7). Europe has only 1 and North America has no reports. In all studies, IVIG response was determined at 36 h after the end of administration (Supplemental Table 7). Figure 3E and Fig 4E shows low PPV (0.14; 95% CI: 0.07–0.23) and high NPV (0.92; 95% CI: 0.88–0.95).

This review included 5 studies and 5 external validation analyses (Supplemental Table 6). Figure 2E shows low discrimination (C–statistic: 0.63; 95% CI: 0.44–0.78), without differences in the C–statistic between studies from East Asia and Middle East Asia (Supplemental Table 7). Europe and North America have no reports. Additionally, the C–statistic between studies with different IVIG response determination timing has no difference (Supplemental Table 7). Figure 3F and Fig 4F shows low PPV (0.16; 95% CI: 0.11–0.22) and high NPV (0.91; 95% CI: 0.79–0.99).

This systematic review of 161 model analyses in 48 studies summarized the currently available evidence on performing prediction models for initial IVIG resistance in patients with KD. Our meta-analyses suggested that none of the prediction models have properly distinguished between patients with and without IVIG resistance.

None of the 5 models that were externally validated could not discriminate well. The summary C–statistics, which integrates the external validation results, was <0.75,²²  although the Kobayashi score has the highest C–statistic of 0.65. The poor performance of the existing 5 predictive models may be attributed to the fact that KD is a heterogeneous syndrome. KD is speculated to occur as a result of immune responses to a variety of infectious etiologies, including viral, fungal, and bacterial agents, as well as environmental stimuli.^68,69  Genetic factors and racial differences may also lead to variation in clinical symptoms and laboratory parameters, thereby making these predictive models difficult to apply to populations other than the development cohort.

The existing prediction models showed NPV of 85% to 92% and PPV of 14% to 39%, indicating that approximately 10% to 20% of patients classified as low-risk are IVIG resistant, whereas only 15% to 40% of patients classified as high-risk do need adjuvant therapy with additional initial IVIG therapy. The clinically acceptable NPV and PPV on this research topic have no established consensus. Thus, the optimum cutoff point for the existing models should be based on the clinical context in each case. For instance, higher NPV (ie, missing fewer patients with IVIG resistance) would be desirable in patients with a severe condition. A lower cutoff point in the existing prediction models might be more appropriate for such patients. However, the model performance in the studies included in the present review was generally assessed only using a cutoff point that was reported (and arbitrarily determined) in the original developmental studies. Therefore, the model’s performance could not be evaluated with different cutoff points.

In clinical settings, collectively considering the prediction model score and other independent predictors might be practical to make an overall decision,³⁴  because a highly reliable prediction model remained unavailable. Each model had a higher rate of IVIG resistance in the high-risk group than in the low-risk group. The higher the point in the model, the higher the risk of IVIG resistance in the Kobayashi and Formosa scores. For example, a score of 8 on the Kobayashi score is associated with a higher-risk IVIG resistance than a score of 5. Rather than only focusing on the dichotomized judgment (ie, positive or negative) of the prediction models, it is more informative to refer to the score itself. Several predictors as a sole predictor have also been reported.^70,71  Furthermore, the development and validation of IVIG adjuvant therapy as the standard first-line therapy, which is more effective in preventing CAA formation than conventional IVIG therapy, may also be warranted if developing a prediction model with good performance is difficult.

Model performance for studies reported from East Asia and Middle East Asia was comparable. Contrastingly, model performance from East Asia and Europe or North America is not comparable because of the limited studies incorporated in this review. This was inconsistent with the previous indications that models developed in Japan might have performance differences between Japanese and non-Japanese patients. Recent studies from the United Kingdom,²⁴  Germany,²⁵  and Spain²⁶  demonstrated that the prediction models developed by the Japanese research group were not as effective for non-Japanese patients. However, these studies did not describe the IVIG dose and administration method. Therefore, further studies with well-defined IVIG doses and administration methods are needed to clarify differences in model performance by race or study area.

This was the first systematic review of prediction models’ performance for initial IVIG resistance in patients with KD. We included 48 studies and performed meta-analyses of the 5 different prediction models using only the external validation results in different studies. Referring to the critical appraisal and data extraction for systematic reviews of prediction modeling studies checklist and PROBAST tool, we could extract and assess detailed data on study characteristics and model performance metrics.

This study had some limitations. First, the effects of differences in inclusion criteria between studies, such as the timing of IVIG administration or complete or incomplete type, were not accounted for. Potential differences in the prediction model’s performance between different timings of initial treatment⁷  and between complete and incomplete KD have been suggested.³⁵  Second, the calibration, such as OE ratios, could not be assessed because they are not reported in external validation analyses.

The external validity remains unknown for most existing prediction models to predict IVIG resistance in patients with KD. Additionally, externally validated models have no sufficient performance to distinguish between patients with and without IVIG resistance. Therefore, further studies are required to develop and validate prediction models with reliable performance.

We are very grateful to Dr. Jun Watanabe, Department of Surgery, Division of Gastroenterological, General and Transplant Surgery, Jichi Medical University, for providing support to our research; and Enago (www.enago.jp) for their English language review.

CAA

coronary artery abnormality

CI

confidence interval

IVIG

intravenous immunoglobulin

KD

Kawasaki disease

NPV

negative predictive value

OE

observed–to–expected

PRISMA

Preferred Reporting Item for Systematic Review and Meta-analysis

PROBAST

Prediction model Risk of Bias Assessment Tool

PPV

positive predictive value