Diagnostic Accuracy of a Real-Time Host-Protein Test for Infection

This retrospective clinical study enrolled patients at the ED or pediatric ward in 2 secondary medical centers (Hillel Yaffe Medical Center, Hadera, Israel and Bnai-Zion Medical Center, Haifa, Israel) between October 2014 and October 2017.

Inclusion criteria were age 3 months to 18 years, symptoms of acute infection with fever within a week prior to presentation, and MMBV ordered at the physician’s discretion as part of routine care to aid in differentiating between bacterial and viral infection.

Exclusion criteria were suspicion of gastrointestinal infection, patients with congenital or acquired immunodeficiency, a proven or suspected infection with hepatitis B virus or hepatitis C virus, another febrile illness within 3 weeks before sampling, hematologic malignancy, significant trauma or burns within the 7 days prior to sampling, and other illnesses that affect life expectancy and\or quality of life.

Upon ordering the MMBV test, physicians completed questionnaires intended for internal hospital use. Ethical clearance was granted by the Hillel Yaffe Medical Center and Bnai-Zion Medical Center Institutional Review Boards to retrospectively examine the questionnaires; informed consent was not required. This study was registered at ClinicalTrials.gov (NCT03075111).

Blood samples for MMBV were obtained during the ED stay or the initial evaluation in the pediatric ward. Upon ordering MMBV, physicians were requested to complete a questionnaire documenting the suspected clinical indication at the time of blood draw and their level of certainty in the infection etiology that they suspected (Supplementary Fig 1). The following clinical syndromes were listed in the questionnaire: lower respiratory tract infection (LRTI), upper respiratory tract infection (URTI), fever without a source (FWS), gastrointestinal infection, urinary tract infection (UTI), and other infection. The physician’s etiological certainty was scored as follows: bacterial +++, bacterial ++, bacterial +, undecided, viral +, viral ++, viral +++. The questionnaire was filled out after obtaining medical history and physical examination, and physicians documented the extent of available clinical information at the time of questionnaire completion. MMBV results were provided to the physicians within hours to days from blood draw (see Index Test section below). The patients' demographic data, medical history, physical examination findings, and all ancillary testing results, including laboratory, microbiology, and imaging assays performed as part of routine care, were recorded in electronic case report forms (eCRFs).

The primary objective of the authors of the SPIRIT study (NCT03075111) was to evaluate the diagnostic accuracy of MMBV ordered as part of routine care management of febrile pediatric patients.

The following cohorts were defined and analyzed in this study:

• The eligible cohort, comprising all eligible patients.

• The analysis cohort, comprising patients who received a bacterial or viral reference standard label.

• The low certainty cohort, comprising patients who received a bacterial or viral reference standard label and were documented by the physician as having uncertain etiology.

In the absence of a gold standard for determining bacterial versus viral etiology, the reference standard for infection diagnosis was generated based on expert panel adjudication in accordance with the National Health Service (NHS) Health Technology Assessment Guidelines for Evaluation of Diagnostic Tests.³⁰  The experts were 3 pediatricians who independently assessed the case report form for each case. The experts were provided with all laboratory biomarkers taken as part of routine care, including WBC, ANC, and CRP. Of note, procalcitonin was not in routine use for children in Israel at the time of this study. After reviewing the eCRF, the experts assigned one of the following labels: bacterial (including a mixed bacterial and viral coinfection), viral, noninfectious, and indeterminate. Experts were blinded to their peers’ labels and to MMBV results. A reference standard of “bacterial” or “viral” required all 3 experts to assign the same label (unanimous adjudication). Cases that did not meet this criterion were classified as “indeterminate” reference standard.

TRAIL and IP-10 were measured by using an enzyme-linked immunosorbent assay kit, ImmunoXpert (MeMed, Israel), and CRP was measured by using COBASc501. MMBV is a score ranging from 0 to 100 that computationally integrates TRAIL, IP-10, and CRP measurements using an algorithm derived previously²⁵  and employed in all previous studies;^{26,27,31–33}  ImmunoXpert software was used to calculate MMBV. Cutoffs were based on the manufacturer’s instructions for use, ie, MMBV <35 indicated viral infection or other nonbacterial condition, MMBV >65 indicated bacterial infection (including viral-bacterial coinfection), and 35 ≤ MMBV ≤65 was considered equivocal. Physicians were trained on the intended use and limitations of ImmunoXpert (see details from Instructions for Use in the Supplemental Information). Based on the training, physicians ordered MMBV at their discretion. The test was performed once daily during weekdays by an experienced technician within the medical center’s central laboratory, and so the result was often not received by the physician before antibiotic decision making. Of note, ImmunoXpert is no longer available, but MMBV can now be measured by an automated platform called MeMed Key (CE marked and Food and Drug Administration (FDA) cleared, i.e, can be legally commerialized in the European Union and in the US) that provides results from serum within 15 minutes; the scores generated by the 2 platforms have been established as comparable.³⁴ 

Sensitivity was defined as the number of cases with both a bacterial reference standard and a bacterial MMBV result (ie, true positive) divided by the number of cases with a bacterial reference standard (ie, true positive + false negative). Specificity was defined as the number of cases with both a viral reference standard and a viral MMBV result (ie, true negative) divided by the number of cases with a viral reference standard (ie, true negative + false positive). Positive predictive value (PPV) was defined as the number of cases with both a bacterial reference standard and a bacterial MMBV result (ie, true positive) divided by the number of cases with a bacterial MMBV result (ie, true positive + false positive). Negative predictive value (NPV) was defined as the number of cases with both a viral reference standard and a viral MMBV result (ie, true negative) divided by the number of cases with a viral result (ie, true negative + false negative). Equivocal MMBV scores (35 ≤ score ≤65) do not provide etiological information, and physicians are advised to employ other patient data in their decision making. Therefore, the percentage of cases with an equivocal MMBV score is reported, and these cases were not included in the calculation of diagnostic accuracy. The diagnostic accuracy of the physician's initial etiological suspicion as documented in a questionnaire was assessed in comparison with the reference standard; cases assigned as undecided in the questionnaire were removed from the calculation.

The McNemar’s test and the χ-squared test were used for comparing paired and unpaired proportions, respectively. Results were considered statistically significant when P < .05. P values < .001 are reported as P < .001 or marked *. P values < .0001 are reported as P < .0001 or marked **. Statistical analysis was performed by using Python version 3.9.4.

The medical records of 3003 patients for whom MMBV was ordered as part of routine management were reviewed. Of these, 1768 patients were enrolled at Hillel Yaffe Medical Center and 1235 at Bnai-Zion Medical Center. Of them, 2155 patients met the inclusion criteria for the study (Fig 1); 1277 were assigned viral and 284 bacterial reference standard etiologies.

The median age of the eligible cohort (n = 2155) was 1.8 years (interquartile range [IQR] 3.7), 54% were male, and the median time from symptoms onset until the MMBV test was ordered was 3 days (IQR 3; Table 1). The majority of MMBV tests (95.3%) were ordered in the ED as part of the initial evaluation. The hospitalization rate was 48.9% (1050 patients), with the median hospitalization duration being 3 days (IQR 2). The most frequent discharge diagnosis was unspecified viral infection (37.1%). LRTI and URTI were the discharge diagnoses for 41.4% of patients (15.7% and 25.7%, respectively). CRP, WBC, and blood cultures were routinely obtained for the majority of patients, 100% (2155/2155), 99.7% (2148/2155), and 93.1% (2006/2155), respectively (Supplemental Table 1). Chest radiographs were performed for 80.6% (174/216) of the suspected LRTI cohort. Ultrasounds were performed for 46.4% (51/110) of patients with suspected UTI.

In the analysis cohort (n = 1561), MMBV distinguished bacterial (including coinfection) from viral infection with a sensitivity of 84.9% (203/239; 95% confidence interval [CI]: 79.8–89.0), a specificity of 93.1% (1059/1137; 95% CI 91.5–94.5), a PPV of 72.2% (203/281; 95% CI 66.7–77.2), and an NPV of 96.7% (1059/1095; 95% CI 95.5–97.6). The equivocal rate was 11.9% (185/1561). Notably, among patients discharged with LRTI, MMBV attained a sensitivity of 91.3%, (63/69; 95% CI 82%–96.3%), a specificity of 94.4% (102/108; 95% CI 88.2–97.7), a PPV of 91.3% (63/69; 95% CI 82%–96.3%), and an NPV of 94.4% (102/108; 95% CI 88.2–97.7). The equivocal rate for this sub-cohort was 7.8% (15/192).

Physicians completed the questionnaire field about suspected clinical syndrome in 1991/2155 cases (92.4%), and the field about etiological certainty in 1926/2155 cases (89.4%). The most frequent clinical indications in which the physician ordered the test were fever without source (FWS, 39.4%), URTI (35.8%), and LRTI (10.8%; Fig 2A). Only 2.2% of patients for whom the physician initially suspected FWS according to the questionnaire were discharged with the diagnosis of FWS. Of the patients with suspected LRTI, 67.1% were discharged with the diagnosis of LRTI.

Across the 3 years of the study, 51% of MMBV tests were ordered during fall and winter seasons. The physicians documented suspected respiratory tract infections (LRTI or URTI) more commonly during winter months (December to February) as the suspected clinical indication, whereas suspected FWS was prevalent during summer months (June to August; Fig 2B). The rate of tests taken for UTI and other indications was constant throughout the year.

Of the 1926 cases for which the physician reported their degree of certainty in initial etiological suspicion, 277 (14.4%) were scored with a high level of certainty (bacterial+++/viral+++), and 646 (33.5%) were scored with a medium level of certainty (bacterial++/viral++; Fig 3). The remaining 1003 patients (52% of the 1926 questionnaires) were reported by the physician as cases with uncertain etiology (bacterial+/viral+/undecided). Comparing the subgroup of cases with uncertain etiology (n = 1003) to the subgroup of cases with medium/high certainty (n = 923), there was an enrichment of suspected FWS (426/1003 = 42.5% compared with 269/923 = 29.1%, respectively; P < .0001) and a diminished representation of URTI cases (290/1003 = 28.9% compared with 381/923 = 41.3%, respectively; P < .0001).

MMBV’s diagnostic accuracy and that of the physician’s initial etiological suspicion were assessed among the patients with a bacterial or viral reference standard for whom the physician had completed the relevant questionnaire field (n = 1410). MMBV outperformed in both sensitivity (178/210 = 84.8% vs 125/198 = 63.1%, respectively; P < .0001) and specificity (956/1027 = 93.1% vs 743/918 = 80.9%, respectively; P < .0001; Table 2, Supplementary Figure 2). In a sub-analysis of age groups (3m ≤ age <9m, 9m ≤ age <18m, 18m ≤ age <3y, 3y ≤ age <7y, 7y ≤ age <12y, and 12y ≤ age <18y), MMBV outperformed in sensitivity and/or specificity (Supplementary Figure 3).

For 76/1410 patients, physicians already had WBC and CRP results available at the time of ordering MMBV. In this sub-cohort, MMBV demonstrated significantly higher specificity as compared with the physician’s initial etiological suspicion (43/47 = 91.5% vs 34/45 = 75.6%, respectively; P = .04; Table 2, Supplementary Figure 4).

Among the 1003 cases documented by the physician as having uncertain etiology (Fig 3), 736 were assigned a bacterial or viral reference standard. In this low certainty cohort (n = 736), MMBV yielded a sensitivity of 89.7% (96/107; 95% CI 82.4–94.3), a specificity of 92.6% (498/538; 95% CI 90.0–94.5), a PPV of 70.6% (96/136; 95% CI 62.4–77.6), and an NPV of 97.8% (498/509; 95% CI 96.1–98.8). The equivocal rate was 12.4% (91/736). Supplementary Table 2 details the 11 cases with a bacterial reference standard and a viral MMBV score (false negatives). The physicians’ initial etiological suspicion attained a sensitivity of 66.2% (49/74; 95% CI 54.8–76) and a specificity of 72.0% (265/368; 95% CI 67.2–76.4; Table 2, Supplementary Figure 5). MMBV significantly outperformed the physicians’ initial etiological suspicion in both sensitivity and specificity (P < .001 and P < .0001, respectively). Significant differences were maintained when diagnostic accuracy was assessed on the basis of a paired cohort (ie, in which MMBV equivocal and physician undecided cases were removed from both calculations, n = 393; Supplementary Figure 6).

This study evaluated MMBV’s diagnostic accuracy in a population of febrile children for whom the test was ordered at the physicians’ discretion during the initial examination. Additionally, MMBV’s diagnostic accuracy was compared with the physician's initial etiological suspicion. MMBV significantly outperformed the physician in the determination of infection etiology, even in cases in which the physician had laboratory tests available at the initial examination. These findings support that early availability of the MMBV test result during patient evaluation in the ED adds value to the physician's initial suspicion and has the potential to support medical decision making, thereby potentially expediting triage and reducing additional workup.

The most common clinical indications for which physicians ordered the test were reflective of the indications in which MMBV is anticipated to have the highest clinical value because of etiology determination being confounded by colonizers (eg, respiratory tract infection) and unknown infection source (eg, FWS). It is noteworthy that MMBV’s diagnostic accuracy for patients discharged with LRTI was observed to be even higher than MMBV’s overall performance. Indeed, multiple clinical studies have validated MMBV’s diagnostic accuracy in these subpopulations.^25–28 

Because MMBV was ordered at the physician's discretion, it is assumed that the patients most easily diagnosed and managed were left out of the current analysis. The assumption that the study population represents more diagnostically or medically challenging cases is supported by the high hospitalization rate of nearly 50% of the patients, compared with ∼25% for a general pediatric ED visit.²⁵  In addition, physicians documented diagnostic uncertainty in >50% of patients for whom MMBV tests were ordered.

In medical centers in which MMBV is being deployed as part of routine care, it is anticipated that the MMBV test will be ordered at the same time as blood is drawn for conventional tests, particularly for children, because there is a strong desire to reduce invasive sampling. This is precisely the time point at which the questionnaire was completed in the current study to uncover the physician’s etiological suspicion and certainty. Accordingly, the findings of this study support that rapid provision of the MMBV result has the potential to confirm or change the physician’s initial suspicion. In the future, after data on MMBV’s real-world performance have accumulated, it is anticipated that workflows may be revised so that the MMBV result impacts decisions relating to work-up.

A major strength of this diagnostic accuracy study is that the study population comprises patients selected at the physician’s discretion who fall within the test’s indication for use. In this way, the study population reflects the real-world use of the test. An additional strength is the use of questionnaires to reveal the physician's initial suspicion of the presenting syndrome and etiology with the scoring of uncertainty, the latter a known methodology developed for examining physician perception.³⁵  Lastly, the long duration of the study enabled a large and clinically diverse cohort to be enrolled.

A study limitation is that MMBV was measured by using a time-consuming enzyme-linked immunosorbent assay (ELISA)-based platform that was performed only once daily, during weekdays, in the medical center’s central laboratory. Thus, although MMBV results were made available to physicians, the time to results varied from hours to days and necessitated taking management decisions before receiving the results. Accordingly, the study could not evaluate the real-time use of the test and its impact on decision making. Of note, MMBV can now be performed rapidly by using a new user-friendly and compact platform, enabling its application within the ED workflow. Real-world studies are required to evaluate its impact on patient management.

In summary, MMBV demonstrated high performance, specifically in cases considered difficult to diagnose after initial assessment, regardless of the availability of laboratory tests. Future studies using a rapid MMBV measurement platform are warranted to assess clinical utility.

We thank our colleagues for their inputs: Yaly Orr, MD, Michal Stein, MD, Hagai Hamami, MD, Roy Navon, MSc, Amir Nakar, PhD, Efrat Flashner-Abramson, PhD, Naama Sitry, MD, Moran Barak, MD, Tanya M. Gottlieb, PhD, and Eran Eden, PhD. These colleagues are/were employees of MeMed. We thank Daniel Glikman, MD, Renata Yakobov, MD, Sharona Paz, MD, and Yael Shahor, MD, for their adjudication.

ANC

absolute neutrophil count

CI

confidence interval

CRP

C-reactive protein

ED

emergency department

FWS

fever without source

IP-10

interferon γ-induced protein-10

IQR

interquartile range

LRTI

lower respiratory tract infection

MMBV

MeMed BV

NPV

negative predictive value

PCR

polymerase chain reaction

PPV

positive predictive value

TNF

tumor necrosis factor

TRAIL

TNF-related apoptosis-induced ligand

URTI

upper respiratory tract infection

UTI

urinary tract infection

WBC

white blood cell count