Failure of Risk Prediction Modeling for IVIG Resistance in Kawasaki Disease

Kawasaki disease (KD) is an acute pediatric vasculitis associated with coronary artery aneurysms (CAA) that require life-long care and, when large or giant, carry high morbidity and mortality.^1–3  Treatment with intravenous immunoglobulin (IVIG) within 10 days and ideally 7 days of illness onset dramatically reduces CAA risk.⁴  Nonetheless, 15% to 20% of children develop CAA in the first 6 weeks of illness despite early IVIG treatment.^5,6  IVIG resistance, most often defined as persistent or recrudescent fever 36 hours after IVIG completion, is a risk factor for CAA.¹  IVIG resistance is also an outcome for which multiple risk prediction models have been constructed. In this month’s issue of Pediatrics, Kuniyoshi et al present the results of their meta-analysis assessing the performance of clinical prediction models for IVIG resistance.⁷ 

Clinical prediction models are generally used to guide decision-making by predicting the probability of an outcome in a given patient. For a prediction model to perform well, it should have good discrimination (the model’s ability to distinguish patients who have the outcome of interest from those who do not, also known as the “C statistic”) and calibration (how well the probabilities predicted from the model agree with the observed probabilities of the outcome). Prediction models often perform well in the data used to develop the model, but external validation in independent data sets is important to test generalizability.

For their meta-analysis, Kuniyoshi et al identified 48 studies with 161 analyses using Preferred Reporting Items for Systematic Reviews and Meta-Analyses guidelines to assess the Kobayashi,⁸  Sano,⁹  Egami,¹⁰  Formosa,¹¹  and Harada¹²  prediction models for IVIG resistance. These models are composed of varying demographic, clinical, and laboratory characteristics; none include baseline coronary artery measurements. The models were chosen because each had performance metrics available from at least 3 external validation studies. On the basis of aggregated data from source studies for each risk score, all C statistics were ≤0.65 and calibration was generally not reported. All models demonstrated low positive predictive values (0.14–0.39) and high negative predictive values (0.85–0.92). Given these findings, the authors correctly conclude that their results do not support use of these published models for IVIG resistance.

The methodology employed by the authors is sound and thorough. However, meta-analyses rely on data from contributing studies that can be heterogeneous in study design, inclusion/exclusion criteria, case mix, and baseline risk for the outcome. Notably, heterogeneity is more common in meta-analyses involving prediction models than in those synthesizing evidence about the association between a single predictor and an outcome. If there is too much heterogeneity among source studies, the increased sample size may not be helpful. In this meta-analysis, the contributing studies used varying fever-based definitions for IVIG resistance (fever at 24 or 36 hours, or for 48 hours, after IVIG completion or per physician’s judgment), although the authors report that fever duration did not alter their findings. There was also heterogeneity for risk of IVIG resistance, because timing of initial IVIG administration was not accounted for. Lastly, previous studies have documented poor discrimination with these models^13–15 ; thus, the meta-analysis findings are not surprising.

Whereas risk scores fall short in predicting IVIG resistance in individual patients outside the population in which they were developed, they have proven successful in selecting clinical trial participants to constitute a sample at higher risk for CAA. In particular, the Kobayashi score⁸  has been used to identify high-risk cohorts for randomization in Japanese treatment trials of antiinflammatory agents adjunctive to IVIG. These trials were successful in achieving improved coronary artery outcomes in patients who received additional initial therapy with corticosteroids¹⁶  or cyclosporine.¹⁷  The advantages of using a prediction model to select a high-risk study population are evident from the difficulty of adequately powering coronary artery outcomes in trials of unselected KD patients.^18,19 

Although current scoring systems for IVIG resistance are inadequate in identifying individual high-risk KD patients, clinicians should nonetheless consider well-established risk factors as they tailor management decisions. When IVIG resistance is observed (rather than predicted), treatment with additional antiinflammatory therapy²⁰  and more frequent echocardiography¹  is indicated to mitigate risks of CAA. Other important risk factors for CAA include age <6 months,²¹  markers of greater systemic inflammation (higher C-reactive protein and lower hematocrit, platelet count, albumin, and sodium)¹  and coronary artery dilation on baseline echocardiography, as measured by z scores of the left anterior descending and right coronary arteries.^1,5,6,22,23  Indeed, all KD patients with z scores ≥2.5 in the left anterior descending and/or right coronary arteries at baseline or early in illness warrant primary intensification with corticosteroids,^16,24–27  the best-established adjunctive antiinflammatory medication, or other immunomodulatory agents^17,18,28  to decrease the risk of progressive coronary enlargement. Echocardiography at least twice weekly until coronary dimensions stabilize is also highly recommended.¹  These interventions remain the cornerstone of care for KD while researchers strive to develop accurate risk models that, in the future, may include not only sociodemographic and clinical variables, but also genetic variants and infectious/environmental factors.

CAA

coronary artery aneurysms

IVIG

intravenous immunoglobulin

KD

Kawasaki disease