Reliably distinguishing bacterial from viral infections is often challenging, leading to antibiotic misuse. A novel assay that integrates measurements of blood-borne host-proteins (tumor necrosis factor-related apoptosis-inducing ligand, interferon γ-induced protein-10, and C-reactive protein [CRP]) was developed to assist in differentiation between bacterial and viral disease.
We performed double-blind, multicenter assay evaluation using serum remnants collected at 5 pediatric emergency departments and 2 wards from children ≥3 months to ≤18 years without (n = 68) and with (n = 529) suspicion of acute infection. Infectious cohort inclusion criteria were fever ≥38°C and symptom duration ≤7 days. The reference standard diagnosis was based on predetermined criteria plus adjudication by experts blinded to assay results. Assay performers were blinded to the reference standard. Assay cutoffs were predefined.
Of 529 potentially eligible patients with suspected acute infection, 100 did not fulfill infectious inclusion criteria and 68 had insufficient serum. The resulting cohort included 361 patients, with 239 viral, 68 bacterial, and 54 indeterminate reference standard diagnoses. The assay distinguished between bacterial and viral patients with 93.8% sensitivity (95% confidence interval: 87.8%–99.8%) and 89.8% specificity (85.6%–94.0%); 11.7% had an equivocal assay outcome. The assay outperformed CRP (cutoff 40 mg/L; sensitivity 88.2% [80.4%–96.1%], specificity 73.2% [67.6%–78.9%]) and procalcitonin testing (cutoff 0.5 ng/mL; sensitivity 63.1% [51.0%–75.1%], specificity 82.3% [77.1%–87.5%]).
Double-blinded evaluation confirmed high assay performance in febrile children. Assay was significantly more accurate than CRP, procalcitonin, and routine laboratory parameters. Additional studies are warranted to support its potential to improve antimicrobial treatment decisions.
Pediatricians are often unsure if a febrile child has bacterial or viral disease. Clinical judgment and laboratory tests are sometimes insufficient for diagnosis, leading to antibiotic misuse. Recently, a host-protein assay for distinguishing bacterial versus viral infections exhibited promising accuracy.
This double-blind study evaluated the host-protein assay's performance for differentiating between bacterial and viral infections in pediatric inpatients and emergency department arrivals. The assay revealed accurate results and significantly outperformed routine biomarkers, including C-reactive protein, procalcitonin, and other laboratory parameters.
It is estimated that medical attention is sought for 250 million episodes of acute infectious disease annually in the United States, with roughly 40 million managed in the emergency department (ED) setting.1 A major challenge to effective management of febrile patients, especially children, is the clinical difficulty of distinguishing between bacterial and viral (or other) infections.2 This clinical uncertainty drives antibiotic misuse, both underuse2 and overuse,3 with detrimental ramifications for the patient, the health care system, and society, including the emergence of antibiotic resistance.4,–6
Although routine diagnostic tests for pathogen detection may aid in determining infection etiology, they often suffer from 1 or more of the following key limitations: (1) lengthy time to result; (2) the need for obtaining a sample from the site of infection, which is often not readily accessible; (3) uncertainty regarding the clinical interpretation of viral identification, which does not preclude the possibility of a bacterial coinfection7; and (4) false alarms caused by the carriage of potentially pathogenic microbes that are also part of the natural flora (eg, Streptococcus pneumoniae).8,9
Host-based biomarker approaches represent a promising complement to pathogen-based diagnostics.10 In recent studies, authors examining host-RNA signatures in response to different infections show promising results.11,–16 However, quantitative and rapid measurement (within minutes) of multiple host RNAs remains a technical challenge, particularly at the point of need. In contrast, circulating (particularly soluble) host proteins are a promising basis for a diagnostic test, as they are readily amenable to rapid and quantitative measurement by using well established technologies. Proteins routinely used today to support diagnosis of infection include procalcitonin (PCT) and C-reactive protein (CRP), but these markers are subject to interpatient variability.17,–22
Recently, a 3-protein host-assay was developed, comprising a novel viral-induced host protein called tumor necrosis factor-related apoptosis-inducing ligand (TRAIL), as well as interferon γ-induced protein-10 (IP-10) and the traditional bacterial-induced protein CRP.23,24 This host-signature assay was used to accurately differentiate bacterial from viral infection etiologies in a study of 1002 febrile children and adults with a wide range of clinical syndromes, pathogens, time from symptom onset, and comorbidities: sensitivity of 92% (95% confidence interval [CI]: 88%–96%) and specificity of 89% (95% CI: 86%–93%). However, a validation study initiated by investigators external to the sponsor, conducted in a double-blind manner and focusing on children, has been lacking. Of note, while this study was under review, an independent validation study was published that reported high assay performance (sensitivity of 88% [95% CI: 75%–95%] and specificity of 93% [95% CI: 90%–95%]) and superiority to PCT, CRP, and other routine tests in preschool children with lower respiratory tract infections or fever without source.25
The aim of the current study is to validate the assay performance in a double-blind, multicenter, investigator-driven clinical study focusing on pediatric inpatients and ED arrivals aged 3 months to 18 years old.
Methods
Study Population
This study used banked, frozen serum sample remnants from the following sources: (1) pediatric patients prospectively recruited between 2008 and 2010 from the pediatric EDs (PEDs) of 2 tertiary and 1 secondary hospital in Switzerland, where the target populations were patients with clinical and radiological pneumonia for infectious patients and patients with minor elective surgery for healthy (noninfectious) patients26; (2) pediatric patients prospectively recruited between 2010 and 2013 from the PED of a tertiary hospital in Switzerland, where the target population for infectious patients was patients who had fever without identified source27; and (3) pediatric patients presenting to the PED and pediatric wards of 2 secondary hospitals in Israel between 2011 and 2013, from whom blood was drawn and sent to the microbiology laboratories for serological analysis. For more details about the study population see the Supplemental Information.
On the basis of the discharge diagnosis of each accompanying anonymized medical case record, the banked serum samples were divided into potentially eligible patients with no suspicion of acute infection (ie, healthy and/or noninfectious patients) and potentially eligible patients with suspicion of acute infection. Next, the medical case records were screened according to the inclusion/exclusion criteria. Inclusion criteria for potentially eligible patients with no suspicion of acute infection (ie, healthy and/or noninfectious) were as follows: age ≥3 months and ≤18 years and a fever <38°C. Inclusion criteria for potentially eligible patients with suspicion of acute infection were as follows: age ≥3 months and ≤18 years and a fever ≥38°C documented at clinical encounter (Swiss patients), or a peak fever ≥38°C at some point after symptom onset was documented in the medical case record (Israeli patients), and symptom duration ≤7 days. The exclusion criteria for all participants were as follows: antibiotic treatment duration at the time of serum sampling >48 hours; another infection episode during the last 3 weeks before sampling; congenital immune deficiency; proven or suspected HIV, hepatitis B virus, or hepatitis C virus infection; active hematologic malignancy; current treatment with immune-suppressive or immune-modulating therapies; or other illnesses that affect life expectancy and/or quality of life.
This study was registered at www.clinicaltrials.gov (identifier NCT01911143) and was approved by the local ethics committees in the participating countries. Informed consent was obtained in both Swiss prospective studies. For the testing of banked, anonymized sample remnants from the Israeli microbiological laboratories, the requirement for informed consent was waived by the local ethics committee.
Index Test
The index test is a host-signature assay (ImmunoXpert; MeMed Diagnostics, Ltd, Tirat Carmel, Israel) that is used to measure and computationally integrate the blood levels of TRAIL, IP-10, and CRP into a bacterial/viral likelihood score.23 On the basis of predefined cutoffs, the index test is used to generate 3 possible outcomes: (1) viral infection (or other nonbacterial etiology): ImmunoXpert score <35; (2) equivocal: 35 ≤ ImmunoXpert score ≤65; and (3) bacterial infection (including mixed bacterial and viral coinfection): ImmunoXpert score >65.23,24
Reference Standard
There is no single gold standard for broadly determining the underlying etiology for acute infections. Therefore, in line with the Standards for Reporting of Diagnostic Accuracy and National Institute for Health Research Health Technology Assessment guidelines for evaluation of diagnostic tests, predetermined criteria and a panel of experts were employed to adjudicate the reference standard for each patient (for details, see the Supplemental Information).28,–31 The experts were blinded to the diagnoses of their peers to prevent group pressure or influential personality bias and were blinded to the host-signature assay result.
For the primary analysis, a reference standard outcome of bacterial or viral etiology was assigned when the majority of the panel, that is, at least 2 of the 3 panel members, gave the same diagnosis, and predetermined criteria were fulfilled. A reference standard outcome of indeterminate was assigned when at least 2 of the 3 panel members diagnosed the patient as indeterminate or when there was no majority.
For the secondary analysis, the diagnostic performance of the host-signature assay was evaluated by employing 2 alternative, and increasingly stringent, approaches to assigning reference standard outcomes: (1) unanimous: a bacterial/viral outcome required that all 3 panel members assign the same diagnosis plus fulfillment of predetermined criteria; and (2) microbiologically confirmed: in addition to a unanimous diagnosis, a bacterial outcome required proven (according to the predetermined criteria) bacteremia, bacterial meningitis, lower/upper urinary tract infection, acute tonsillitis or peritonsillar abscess, and a viral outcome required detection of 1 or more viruses.
Blinding Procedure
The study was conducted in a double-blind manner. Specifically, index test performers were blinded to reference standard outcomes, and the assay was performed on anonymized samples. Additionally, all personnel involved in generating the clinical database and reference standard outcomes were blinded to the index test outcomes. The index test and reference standard outcomes were locked before unblinding.
Statistical Analysis
A statistical analysis plan was written in accordance with the International Conference on Harmonization topic E9 and the Standards for Reporting of Diagnostic Accuracy recommendations30,31 and was signed off on before unblinding.
Primary analysis of diagnostic performance was based on sensitivity and specificity (Supplemental Information). By definition, the cases with indeterminate reference standard outcomes could not be included in the diagnostic performance analysis. However, the distribution of index test scores among these patients was examined as a subgroup analysis.
Standard cutoffs of routine laboratory parameters and biomarkers were predefined for evaluation of their diagnostic performance; for details, see the Supplemental Information.
Results
Characteristics of Study Population
In total, 597 banked samples were potentially eligible for evaluation: 529 from patients with suspicion of acute infection noted on their medical case record and 68 from patients without suspicion of an acute infectious disease (Fig 1A). Since the host-signature assay is intended to aid the clinician in discriminating between bacterial and viral infections when there is a clear suspicion of acute infection, the cohort for primary analysis included the potentially eligible infectious patients (Fig 1B); an exploratory analysis that also includes the patients without suspicion of acute infection (ie, healthy and/or noninfectious patients) is described in the Supplemental Information (Supplemental Fig 9, Supplemental Table 9). Review of the 529 medical case records in which the clinician noted suspicion of acute infection led to exclusion of 100 patients who failed to meet 1 or more of the infectious inclusion criteria (Fig 1, Supplemental Table 6): 46 had infection with fever below the 38°C threshold; 41 had symptoms for more than 7 days; and 21 were younger than 3 months. A further 68 patients were excluded because their samples contained insufficient volume to run the host-signature assay. The resulting infectious cohort on which the index test was performed totaled 361 participants. Application of the majority panel reference standard resulted in bacterial or viral reference standard diagnoses for 307 of the study population (85%). Specifically, there were 239 viral (66%), 68 bacterial (19%), and 54 (15%) indeterminate reference standard outcomes (Fig 1).
A, Banked serum samples from potentially eligible participants were classified as without or with suspicion of acute infection. B, Flow through of participants with suspicion of acute infection. C, Host-signature assay outcomes. A–B, Flow diagram is structured according to Standards for Reporting of Diagnostic Accuracy.31 The study cohort (n = 361) included the subjects on which the index test was performed (this includes the cases with an indeterminate reference standard). a Participant may satisfy >1 exclusion criteria. b Reassigned as reference indeterminate because expert panel did not employ predetermined criteria (see Supplemental Information, Methods). C, Schematic showing outcomes of host-signature assay according to manufacturer’s instructions. An equivocal outcome is a nonmissing, nonerroneous result that does not provide diagnostic information. FN, false-negative; FP, false-positive; TN, true-negative; TP, true-positive.
A, Banked serum samples from potentially eligible participants were classified as without or with suspicion of acute infection. B, Flow through of participants with suspicion of acute infection. C, Host-signature assay outcomes. A–B, Flow diagram is structured according to Standards for Reporting of Diagnostic Accuracy.31 The study cohort (n = 361) included the subjects on which the index test was performed (this includes the cases with an indeterminate reference standard). a Participant may satisfy >1 exclusion criteria. b Reassigned as reference indeterminate because expert panel did not employ predetermined criteria (see Supplemental Information, Methods). C, Schematic showing outcomes of host-signature assay according to manufacturer’s instructions. An equivocal outcome is a nonmissing, nonerroneous result that does not provide diagnostic information. FN, false-negative; FP, false-positive; TN, true-negative; TP, true-positive.
The infectious cohort was sex-balanced, with an average age of 4.1 years and a wide range of temperatures, times from symptom onset, and clinical syndromes (Table 1).
Baseline Characteristics of Study Population (for Patients With Suspicion of Acute Infection)
| Study Population n = 361 | |
|---|---|
| Age, y (mean, SD) | 4.1 (4.2) |
| Sex, female, n (%) | 170 (47) |
| Maximal temperature, °C (mean, SD) | 39.3 (0.8) |
| Received antibiotics, n (%) | 188 (52) |
| Time from symptom onset, d (median, IQR) | 4 (3) |
| Hospital admission, n (%) | 139 (47) |
| Hospitalization duration, d (median, IQR) | 0 (3) |
| Clinical syndrome, n (%) | |
| CNS infection | 3 (1) |
| FWS | 125 (35) |
| GE | 11 (3) |
| LRTI | 56 (16) |
| URTI | 114 (32) |
| UTI | 17 (5) |
| Other | 35 (10) |
| Study Population n = 361 | |
|---|---|
| Age, y (mean, SD) | 4.1 (4.2) |
| Sex, female, n (%) | 170 (47) |
| Maximal temperature, °C (mean, SD) | 39.3 (0.8) |
| Received antibiotics, n (%) | 188 (52) |
| Time from symptom onset, d (median, IQR) | 4 (3) |
| Hospital admission, n (%) | 139 (47) |
| Hospitalization duration, d (median, IQR) | 0 (3) |
| Clinical syndrome, n (%) | |
| CNS infection | 3 (1) |
| FWS | 125 (35) |
| GE | 11 (3) |
| LRTI | 56 (16) |
| URTI | 114 (32) |
| UTI | 17 (5) |
| Other | 35 (10) |
Clinical syndrome was the diagnosis recorded by the attending clinician at discharge (and was not modified in light of data accrued after discharge). CNS infections included meningitis. A diagnosis of FWS required a urine analysis with negative results. LRTIs included pneumonia and bronchiolitis. URTIs included acute tonsillitis, pharyngitis, sinusitis, acute otitis media, aphthous stomatitis, herpengina, retropharyngeal abscess, and scarlet fever. UTIs included cystitis and pyelonephritis. The “other” category included febrile convulsions, hepatitis, lymphadenitis, myositis, parotitis, sialoadenitis, and lymphadenopathy. CNS, central nervous system and encephalitis; FWS, fever without source; GE, gastroenteritis; IQR, interquartile range; LRTI, lower respiratory infection; URTI, upper respiratory infection; UTI, urinary tract infection.
Performance of the Host-Signature Assay
The levels of TRAIL and IP-10 were significantly higher in children with viral as compared with bacterial infections: mean (SD) levels of TRAIL were 139 (122) versus 52 (27) pg/mL (P < .001); and IP-10 levels were 1011 (626) versus 845 (677) pg/mL (P < .03), in agreement with previous studies.23,24 CRP exhibited the opposite pattern, with mean levels significantly lower in children with a viral infection as compared with a bacterial infection: 31 (32) versus 165 (108) mg/L (P < .001). The host-signature assay exhibited the most pronounced differential (P < .001), with a mean viral patient score of 20 (24) versus 85 (23) in bacterial patients (Fig 2).
Differential expression of the host proteins TRAIL, IP-10, and CRP and the host-signature assay score in bacterial and viral infections. Box plots for TRAIL (A), IP-10 (B), CRP concentrations (C), and host-signature assay score (D) measured in patients with bacterial/viral reference standard outcomes assigned according to majority expert panel (n = 307). The red line and circle correspond to group median and average, respectively.
Differential expression of the host proteins TRAIL, IP-10, and CRP and the host-signature assay score in bacterial and viral infections. Box plots for TRAIL (A), IP-10 (B), CRP concentrations (C), and host-signature assay score (D) measured in patients with bacterial/viral reference standard outcomes assigned according to majority expert panel (n = 307). The red line and circle correspond to group median and average, respectively.
Of the 361 patients in the infectious cohort, the host-signature assay classified 209 patients (58%) with a viral outcome (0 ≤ score < 35), 99 patients (27%) with a bacterial outcome (65 < score ≤ 100), and the remaining 53 patients (15%) with an equivocal outcome (35 ≤ score ≤ 65) (Fig 1). Primary analysis of the 307 patients with a majority panel reference standard of bacterial or viral diagnosis yielded a sensitivity of 93.8% (95% CI: 87.8%–99.8%) and specificity of 89.8% (85.6%–94.0%), with positive and negative likelihood ratios of 9.2 (6.1–13.9) and 0.07 (0.03–0.18), respectively (for details of other accuracy measures, see Table 2, Supplemental Fig 6, and Supplemental Table 7). The host-signature assay generated 4 false-negative (patients with bacterial reference standard and viral assay outcomes; Table 3) and 21 false-positive outcomes (patients with viral reference standard and bacterial assay outcomes). The cohort included 6 patients with serious bacterial infection and bacteremia, all of which were correctly classified by the host-signature assay.
Diagnostic Performance of the Host Signature Assay (for Patients With Suspicion of Acute Infection)
| Accuracy Measure | Assay Performance (95% CI) |
|---|---|
| Sensitivity | 93.8% (87.8%–99.8%) |
| Specificity | 89.8% (85.6%–94.0%) |
| PPV | 74.4% (63.6%–83.4%) |
| NPV | 97.9% (94.7%–99.4%) |
| LR+ | 9.2 (6.1–13.9) |
| LR− | 0.07 (0.03–0.18) |
| DOR | 134.3 (44.4–406.7) |
| Equivocal | 11.7% |
| Accuracy Measure | Assay Performance (95% CI) |
|---|---|
| Sensitivity | 93.8% (87.8%–99.8%) |
| Specificity | 89.8% (85.6%–94.0%) |
| PPV | 74.4% (63.6%–83.4%) |
| NPV | 97.9% (94.7%–99.4%) |
| LR+ | 9.2 (6.1–13.9) |
| LR− | 0.07 (0.03–0.18) |
| DOR | 134.3 (44.4–406.7) |
| Equivocal | 11.7% |
Diagnostic performance was evaluated by comparing the reference standard outcome with the host-signature assay outcome; “positive” refers to a bacterial outcome. As predefined in the statistical analysis plan, the reference standard outcomes were assigned according to expert panel majority (n = 307). DOR, diagnostic odds ratio; LR-, negative likelihood ratio; LR+, positive likelihood ratio; NPV, negative predictive value; PPV, positive predictive value.
Details of 4 Patients With Bacterial Reference Standard Outcomes That Were Classified as Viral by Host-Signature Assay (False-Negatives)
| Assay Score | Sex | Age, y | Discharge Diagnosis | Country | Max Temperature | Hospitalization Duration, d | ABX | Microbiology | Clinical Details | Unanimous Adjudication |
|---|---|---|---|---|---|---|---|---|---|---|
| 5 | F | 2.9 | UTI | IL | 39° | 2 | Cephalexin | Escherichia coli | Presented with 5-d history of fever without dysuria. On physical examination, no focus of infection. Two serial midstream urine specimens obtained. In first urinalysis, pyuria (8–10 leukocytes), no nitrites. In second, large number of leukocytes, no nitrites. Urine culture with E coli, 100 000 CFU/mL. Kidney ultrasound within normal limits. | Yes |
| 7 | F | 7 | FWS | IL | 39° | 0 | None | Bartonella henselae | Presented with a 5-d history of fever and unilateral axillary pain. Lymphadenopathy observed on same side as axillary pain. Was scratched by the family cat (no record of location of scratches). On physical examination, erythematous tonsils with exudate. Throat culture negative for GAS. A respiratory viral PCR analysis was not performed. Positive serology for BH (IgM+, IgG−). Negative serology for EBV and CMV. | Yes |
| 33 | F | 8 | LRTI | CH | 38° | 0 | β lactam antibiotics | Mycoplasma pneumoniae | Presented with a 6-d history of fever and cough. On physical examination, no focus of infection. Chest radiograph: RLL infiltrate with pleural effusion. Nasopharyngeal PCR positive for MP. No growth in blood culture. | No |
| 10 | F | 1.5 | FWS | CH | 40.2° | Unknown | Cefuroxime | None | Presented with a 1-d history of fever and apathy. Initial physical examination revealed no focus of infection. The patient had a previous abnormal otoscopy and a diagnosis of AOM 2 wk before the current ED visit, but in the current ED visit, the otoscopy was reported normal. Tympanocentesis was not performed. Laboratories with leukocytosis and elevated PCT (>6). | No |
| Note: Because of the previous AOM history, a technical error was made whereby AOM was written as discharge diagnosis on eCRF given to expert panel.a |
| Assay Score | Sex | Age, y | Discharge Diagnosis | Country | Max Temperature | Hospitalization Duration, d | ABX | Microbiology | Clinical Details | Unanimous Adjudication |
|---|---|---|---|---|---|---|---|---|---|---|
| 5 | F | 2.9 | UTI | IL | 39° | 2 | Cephalexin | Escherichia coli | Presented with 5-d history of fever without dysuria. On physical examination, no focus of infection. Two serial midstream urine specimens obtained. In first urinalysis, pyuria (8–10 leukocytes), no nitrites. In second, large number of leukocytes, no nitrites. Urine culture with E coli, 100 000 CFU/mL. Kidney ultrasound within normal limits. | Yes |
| 7 | F | 7 | FWS | IL | 39° | 0 | None | Bartonella henselae | Presented with a 5-d history of fever and unilateral axillary pain. Lymphadenopathy observed on same side as axillary pain. Was scratched by the family cat (no record of location of scratches). On physical examination, erythematous tonsils with exudate. Throat culture negative for GAS. A respiratory viral PCR analysis was not performed. Positive serology for BH (IgM+, IgG−). Negative serology for EBV and CMV. | Yes |
| 33 | F | 8 | LRTI | CH | 38° | 0 | β lactam antibiotics | Mycoplasma pneumoniae | Presented with a 6-d history of fever and cough. On physical examination, no focus of infection. Chest radiograph: RLL infiltrate with pleural effusion. Nasopharyngeal PCR positive for MP. No growth in blood culture. | No |
| 10 | F | 1.5 | FWS | CH | 40.2° | Unknown | Cefuroxime | None | Presented with a 1-d history of fever and apathy. Initial physical examination revealed no focus of infection. The patient had a previous abnormal otoscopy and a diagnosis of AOM 2 wk before the current ED visit, but in the current ED visit, the otoscopy was reported normal. Tympanocentesis was not performed. Laboratories with leukocytosis and elevated PCT (>6). | No |
| Note: Because of the previous AOM history, a technical error was made whereby AOM was written as discharge diagnosis on eCRF given to expert panel.a |
The discharge diagnosis was the diagnosis recorded by the attending clinician at discharge (and was not modified in light of data accrued after discharge), ABX, antibiotics; AOM, acute otitis media; BH, Bartonella henselae; CFU, colony forming units; CH, Switzerland; CMV, cytomegalovirus; eCRF, electronic case report form; EBV, Epstein Bar virus; FWS, fever without source; GAS, group A streptococcus; Ig, immunoglobulin; IL, Israel; LRTI, lower respiratory tract infection; MP, Mycoplasma pneumoniae; PCR, polymerase chain reaction; RLL, right lower lobe; UTI, urinary tract infection.
To ensure conservative estimation of assay performance, this patient was retained in the performance analysis.
Comparison of Assay to Routinely Used Laboratory Parameters and Biomarkers
The assay was compared with laboratory parameters routinely used to assist in distinguishing bacterial from viral infection; it outperformed white blood cell count (predefined cutoff of 15 000 cells/µL; sensitivity: 72.7% [61.7%–83.8%], P < .002; specificity: 83.2% [78.3%–88.1%], P < .05) and demonstrated significantly improved sensitivity and comparable specificity to absolute neutrophil count (predefined cutoff of 10 000 cells/µL; sensitivity: 68.2% [56.6%–79.7%], P < .001; specificity: 92.9% [89.6%–96.3%], P < .30) (Fig 3). The assay was also compared with 2 biomarkers, CRP32,–35 and PCT,35,–39 which are routinely used in clinical practice, by using standard predefined cutoffs of 40 mg/L and 0.5 µg/mL, respectively. The assay demonstrated comparable sensitivity and significantly improved specificity compared with CRP (sensitivity: 88.2% [80.4%–96.1%], P < .37 and specificity: 73.2% [67.6%–78.9%], P < .001) and significantly improved sensitivity and specificity compared with PCT (sensitivity: 63.1% [51.0%–75.1%], P < .001 and specificity: 82.3% [77.1%–87.5%], P < .03) (Fig 3). Additional comparisons to CRP and PCT are presented in Supplemental Figs 6–8.
Diagnostic performance of host-signature assay compared with routine laboratory parameters and other biomarkers (for patients with suspicion of acute infection). Diagnostic performance was evaluated by comparing the reference standard outcome (according to expert panel majority) with the outcome classified by the host-signature assay or by other parameters or biomarkers. Predefined cutoffs were applied. The assay was run on 307 patients (68 bacterial and 239 viral; 11.7% had an equivocal result). ANC and WBC testing were performed on 292 patients (66 bacterial and 226 viral), PCT testing was performed on 274 patients (65 bacterial and 209 viral); and CRP testing was performed on 307 patients (68 bacterial and 239 viral). ANC, absolute neutrophil count; WBC, white blood cell count.
Diagnostic performance of host-signature assay compared with routine laboratory parameters and other biomarkers (for patients with suspicion of acute infection). Diagnostic performance was evaluated by comparing the reference standard outcome (according to expert panel majority) with the outcome classified by the host-signature assay or by other parameters or biomarkers. Predefined cutoffs were applied. The assay was run on 307 patients (68 bacterial and 239 viral; 11.7% had an equivocal result). ANC and WBC testing were performed on 292 patients (66 bacterial and 226 viral), PCT testing was performed on 274 patients (65 bacterial and 209 viral); and CRP testing was performed on 307 patients (68 bacterial and 239 viral). ANC, absolute neutrophil count; WBC, white blood cell count.
Subgroup Performance Analysis
The host-signature assay yielded robust sensitivity and specificity across a wide range of patient subgroups, including age and clinical syndrome (Table 4, Supplemental Table 8). It performed comparably irrespective of time from symptom onset (sensitivity: P = .61; specificity: P = .63). The Swiss and Israeli subcohorts yielded comparable sensitivity, with higher specificity demonstrated for the Swiss cohort (P < .001). There was no significant difference in sensitivity and specificity between the subgroup that received antibiotics (≤48 hours of treatment) and the entire cohort (P = 0.99 and 0.39, respectively).
Subgroup Analysis of Host-Signature Assay’s Diagnostic Performance (for Patients With Suspicion of Acute Infection)
| Subgroup | No. of Patients (Ref. Bacterial, Ref. Viral) | Equivocal (%) | Sensitivity (95% CI) | Specificity (95% CI) |
|---|---|---|---|---|
| Age | ||||
| 3 mo ≤ age < 3 y | 184 (30, 154) | 10.3 | 93.1 (83.2–100) | 91.9 (87.3–96.6) |
| 3 y ≤ age ≤ 5 y | 41 (12, 29) | 12.2 | 100 (100–100) | 83.3 (67.3–99.4) |
| 5 y ≤ age ≤ 18 y | 82 (26, 56) | 14.6 | 91.7 (79.7–100) | 87.0 (76.8–97.1) |
| Time from symptom onset | ||||
| 0 ≤ day < 2 | 68 (17, 51) | 5.9 | 93.3 (79.0–100) | 87.8 (78.2–97.3) |
| 2 ≤ day < 4 | 80 (17, 63) | 10.0 | 100 (100–100) | 92.9 (85.9–99.8) |
| 4 ≤ day < 6 | 94 (18, 76) | 17.0 | 88.9 (72.8–100) | 86.7 (77.8–95.5) |
| 6 ≤ day ≤ 7 | 63 (16, 47) | 11.1 | 93.8 (80.4–100) | 92.5 (84.0–100) |
| Clinical syndrome | ||||
| CNS infection | 3 (0, 3) | 33.3 | N/A | 100 (100–100) |
| FWS | 111 (8, 103) | 11.7 | 87.5 (57.9–100) | 87.8 (80.9–94.7) |
| GE | 11 (1, 10) | 9.1 | 100 (100–100) | 100 (100–100) |
| LRTI | 47 (20, 27) | 8.5 | 95.0 (84.5–100) | 100 (100–100) |
| URTI | 92 (17, 75) | 15.2 | 92.9 (77.4–100) | 84.4 (75.2–93.5) |
| UTI | 16 (16, 0) | 0 | 93.8 (80.4–100) | N/A |
| Other | 21 (6, 15) | 4.8 | 100 (100–100) | 100 (100–100) |
| Country | ||||
| Israel | 212 (32, 180) | 16 | 93 (84–100) | 86 (80–92) |
| Switzerland | 95 (36, 59) | 3 | 94 (86–100) | 100 (100–100) |
| Subgroup | No. of Patients (Ref. Bacterial, Ref. Viral) | Equivocal (%) | Sensitivity (95% CI) | Specificity (95% CI) |
|---|---|---|---|---|
| Age | ||||
| 3 mo ≤ age < 3 y | 184 (30, 154) | 10.3 | 93.1 (83.2–100) | 91.9 (87.3–96.6) |
| 3 y ≤ age ≤ 5 y | 41 (12, 29) | 12.2 | 100 (100–100) | 83.3 (67.3–99.4) |
| 5 y ≤ age ≤ 18 y | 82 (26, 56) | 14.6 | 91.7 (79.7–100) | 87.0 (76.8–97.1) |
| Time from symptom onset | ||||
| 0 ≤ day < 2 | 68 (17, 51) | 5.9 | 93.3 (79.0–100) | 87.8 (78.2–97.3) |
| 2 ≤ day < 4 | 80 (17, 63) | 10.0 | 100 (100–100) | 92.9 (85.9–99.8) |
| 4 ≤ day < 6 | 94 (18, 76) | 17.0 | 88.9 (72.8–100) | 86.7 (77.8–95.5) |
| 6 ≤ day ≤ 7 | 63 (16, 47) | 11.1 | 93.8 (80.4–100) | 92.5 (84.0–100) |
| Clinical syndrome | ||||
| CNS infection | 3 (0, 3) | 33.3 | N/A | 100 (100–100) |
| FWS | 111 (8, 103) | 11.7 | 87.5 (57.9–100) | 87.8 (80.9–94.7) |
| GE | 11 (1, 10) | 9.1 | 100 (100–100) | 100 (100–100) |
| LRTI | 47 (20, 27) | 8.5 | 95.0 (84.5–100) | 100 (100–100) |
| URTI | 92 (17, 75) | 15.2 | 92.9 (77.4–100) | 84.4 (75.2–93.5) |
| UTI | 16 (16, 0) | 0 | 93.8 (80.4–100) | N/A |
| Other | 21 (6, 15) | 4.8 | 100 (100–100) | 100 (100–100) |
| Country | ||||
| Israel | 212 (32, 180) | 16 | 93 (84–100) | 86 (80–92) |
| Switzerland | 95 (36, 59) | 3 | 94 (86–100) | 100 (100–100) |
Diagnostic performance was evaluated by comparing the reference standard outcome (according to the expert panel majority) with the host-signature assay outcome. The number of patients refers to the subgroup analyzed for diagnostic performance; that is, it does not include patients with indeterminate reference standard outcomes. The clinical syndrome was the diagnosis recorded by the attending clinician at discharge (and was not modified in light of data accrued after discharge) as described in the Table 1 legend. CNS, central nervous system and encephalitis; FWS, fever without source; GE, gastroenteritis; LRTI, lower respiratory infection; URTI, upper respiratory infection; UTI, urinary tract infection
In the secondary analysis, 2 alternative approaches to assigning reference standard outcomes were applied: unanimous expert panel and microbiological confirmation (see Methods). A unanimous expert diagnosis (3 out of 3) of a bacterial or viral infection was observed in 68% of the cases, and microbiological confirmation was observed in 29% of cases. As anticipated, the increase in reference standard stringency led to higher assay accuracy: sensitivity of 96.2% (90.9%–100%) and specificity of 93.3% (89.4%–97.2%) in the unanimous subcohort and sensitivity of 96.0% (87.7%–100%) and specificity of 98.6% (96.0%–100%) in the microbiologically confirmed subcohort (Fig 4).
Diagnostic performance of host-signature assay when alternative approaches are employed to assigning reference standard outcome (for patients with suspicion of acute infection). The performance of the host-signature assay was evaluated when reference standard outcomes were assigned by using increasingly stringent approaches (see Methods for definitions): majority adjudication (n = 307; 11.7% equivocal; primary analysis); unanimous adjudication (n = 244; 11.1% equivocal; secondary analysis); and microbiologically confirmed (n = 106; 6.6% equivocal; secondary analysis).
Diagnostic performance of host-signature assay when alternative approaches are employed to assigning reference standard outcome (for patients with suspicion of acute infection). The performance of the host-signature assay was evaluated when reference standard outcomes were assigned by using increasingly stringent approaches (see Methods for definitions): majority adjudication (n = 307; 11.7% equivocal; primary analysis); unanimous adjudication (n = 244; 11.1% equivocal; secondary analysis); and microbiologically confirmed (n = 106; 6.6% equivocal; secondary analysis).
Assay Results in Children With Indeterminate Reference Standard
Of the 361 children in the infectious cohort, there were 54 children (15%) with indeterminate etiology as defined by the application of the majority panel reference standard. Of the 54 children, there was no expert panel majority for 38, that is, each panel member assigned a different diagnosis; and 14 patients were given an indeterminate outcome, that is, at least 2 of 3 panel members assigned the diagnosis as indeterminate. Two of the cases are those reassigned as reference indeterminate (Fig 1). No significant differences were found in the age, sex, maximal temperature, time from symptom onset, and hospital admission or hospitalization duration for this indeterminate subgroup in comparison with the patients with bacterial/viral reference standard outcomes (Table 5). The proportion of patients who received antibiotics was significantly larger in the indeterminate subgroup (P = .04).
Baseline Characteristics of the Subgroup of Patients With an Indeterminate Reference Standard
| Patients With Reference Bacterial/Viral Outcomes, n = 307 | Patients With Reference Indeterminate Outcomes, n = 54 | P | |
|---|---|---|---|
| Age, y (mean, SD) | 4.0 (4.3) | 4.6 (4.0) | .34 |
| Sex, female, n (%) | 148 (48) | 22 (41) | .38 |
| Maximal temperature, °C (mean, SD) | 39.3 (0.8) | 39.4 (0.9) | .59 |
| Received antibiotics, n (%) | 153 (49) | 35 (65) | .04 |
| Time from symptom onset, d (mean, SD) | 3.5 (2.1) | 3.8 (2.1) | .34 |
| Hospital admission, n (%) | 113 (45) | 26 (54) | .13 |
| Hospitalization duration, d (median, IQR) | 0 (3) | 1 (3) | .43 |
| Patients With Reference Bacterial/Viral Outcomes, n = 307 | Patients With Reference Indeterminate Outcomes, n = 54 | P | |
|---|---|---|---|
| Age, y (mean, SD) | 4.0 (4.3) | 4.6 (4.0) | .34 |
| Sex, female, n (%) | 148 (48) | 22 (41) | .38 |
| Maximal temperature, °C (mean, SD) | 39.3 (0.8) | 39.4 (0.9) | .59 |
| Received antibiotics, n (%) | 153 (49) | 35 (65) | .04 |
| Time from symptom onset, d (mean, SD) | 3.5 (2.1) | 3.8 (2.1) | .34 |
| Hospital admission, n (%) | 113 (45) | 26 (54) | .13 |
| Hospitalization duration, d (median, IQR) | 0 (3) | 1 (3) | .43 |
Baseline characteristics of patients with bacterial or viral reference standard outcomes (left column) and patients with indeterminate reference standard outcome (right column). Reference standard outcomes were assigned according to expert panel majority. IQR, interquartile range.
In the absence of reference standard outcomes, it is not possible to directly evaluate the accuracy of the host-signature assay for the indeterminate subgroup. However, it is of interest to examine the distribution of assay scores for children with indeterminate reference standard outcomes (Fig 5). In 31% of those children, the assay outcome classification was bacterial; in 38% of children, it was viral; and in 31% of children, it was equivocal (Fig 1). This proportion of equivocal assay outcome classifications was higher than that observed in patients with bacterial/viral reference standard outcomes (P < .001). Of note, in 33.3% of the children with an indeterminate reference standard outcome, the assay outcome was classified with a high degree of confidence,23 that is, the bacterial/viral likelihood score was <10 (high likelihood viral) or >90 (high likelihood bacterial).
Host-signature assay scores for children with suspicion of acute infection per reference standard outcome. Box plots of host-signature assay scores for children per reference standard outcome show the following: unanimous reference standard outcome bacterial (3 of 3 experts diagnose bacterial); majority reference standard outcome bacterial (2 of 3 experts diagnose as bacterial); indeterminate reference standard outcome (used in cases in which ≥2 of 3 experts diagnose as indeterminate or in cases in which there was no majority diagnosis); majority reference standard outcome viral (2 of 3 experts diagnose as viral); unanimous reference standard outcome viral (3 of 3 experts diagnose as viral). The red line and circle correspond to group median and average, respectively.
Host-signature assay scores for children with suspicion of acute infection per reference standard outcome. Box plots of host-signature assay scores for children per reference standard outcome show the following: unanimous reference standard outcome bacterial (3 of 3 experts diagnose bacterial); majority reference standard outcome bacterial (2 of 3 experts diagnose as bacterial); indeterminate reference standard outcome (used in cases in which ≥2 of 3 experts diagnose as indeterminate or in cases in which there was no majority diagnosis); majority reference standard outcome viral (2 of 3 experts diagnose as viral); unanimous reference standard outcome viral (3 of 3 experts diagnose as viral). The red line and circle correspond to group median and average, respectively.
Discussion
The current study is the first double-blind, investigator-driven validation study in febrile children ≥3 months and ≤18 years old of a novel assay comprising the host-proteins TRAIL, IP-10, and CRP for distinguishing between bacterial and viral etiologies. In line with the authors of previous reports,23,–25 we independently confirm that the assay is highly accurate (sensitivity of 93.8% [95% CI: 87.8%–99.8%] and specificity 89.8% [85.6%–94.0%]) and that it outperforms routine laboratory parameters and biomarkers (white blood cell count, absolute neutrophil count, CRP, PCT, and their combinations), significantly reducing both false-negatives (potentially reducing missed bacterial infections) and false-positives (potentially reducing antibiotic overuse), with an equivocal assay outcome in under 12% of cases. Moreover, subgroup analysis supports that the assay exhibits consistent performance across a wide range of ages, time from symptom onset, and clinical syndromes.
The robustness of the host-signature is likely explained by the distinctive and complementary viral- and bacterial-induced expression dynamics of the 3 constituent host proteins, as observed previously,23,–25 which may reflect their nonredundant participation in the network of immune responses to infection.40,–42 Moreover, from these data, we can confirm the previously observed unique dynamics of TRAIL, whereby it is induced in response to a wide range of viral infections and significantly reduced in bacterial infections.23,24,43
Notably, among the indeterminate-diagnosis patients without a reference standard, the assay gave a bacterial or viral outcome for 69% of the cases (the rest were equivocal), with half of these yielding a score associated with a particularly high degree of assay diagnostic confidence. This finding suggests that the assay may be applicable to “harder to diagnose” cases in real-life clinical settings.
A key strength of this study is the double-blinded design, as this ensured that the principal investigator and the expert panel were not influenced by the assay result and that the assay performer was not biased by the reference standard diagnosis. Another strength is the diversity of recruitment settings (secondary and tertiary) and geography (Israel and Switzerland) of the study population. It is noteworthy that the majority of Swiss patients (∼70%) presenting to the ED are not referrals and, therefore, are representative of what might be seen in a community setting, whereas the majority of Israeli patients (∼70%) are referred to the ED by a community clinician. This demographic difference between the 2 countries, and the broader spectrum of microbiological tests performed on the Swiss patients, may explain the observed higher specificity of the assay in the Swiss subcohort.
Typically, studies evaluating such diagnostic tests exclude patients who have received antibiotics before patient recruitment. Therefore, another strength of the current study design is that receipt of antibiotics for up to 48 hours was not an exclusion criterion, as these cases are routinely observed in the real clinical setting and their inclusion enhances the generalizability of the study results.
An inherent limitation of this study, and all studies evaluating tests for distinguishing between bacterial and viral infections, is the general lack of objective clinical criteria that can be used to reliably discriminate between these etiologies, which constrains the ability to ascertain “true” and “false” index test outcomes. Often, a microbiologically confirmed approach is employed for determining the reference standard,14,–16,44 which has the advantage of creating a well-established and reproducible diagnosis. Disadvantages of this approach, however, are that microbiological confirmation of etiology is only obtained in a small proportion of cases, and there is an enrichment of the study cohort with “easier to diagnose” cases, resulting in overly optimistic diagnostic performance.28,45 To address this challenge, we applied 3 approaches for assigning the reference standard with increasing levels of stringency: majority adjudication, unanimous adjudication, and microbiologically confirmed. The assay performance in the real-life clinical setting is probably higher than the values we obtained by using the majority adjudication approach but lower than the microbiologically confirmed approach. We therefore performed the primary analysis on the former to ensure more conservative estimates.
Two further limitations also relate to the reference standard. First, microbiological polymerase chain reaction analysis of nasal swabs to detect common respiratory viral and bacterial pathogens, which can improve the reference standard, was only available for some patients, and this should be addressed in the design of future studies. Secondly, although the panel was blinded to the host-signature score, the experts were provided with CRP values, because CRP is routinely used to help differentiate between bacterial and viral infections46 and masking of this information may reduce the quality of the reference standard. Provision of the CRP value to the expert panel might introduce a potential incorporation bias, but it is noteworthy that the signature significantly outperformed CRP.
The population in this study was constrained by the availability of patient samples. Additionally, future studies should seek to replicate the results in other pediatric populations, including the developing world, where there is a need for reliable tests to aid in the diagnosis of pediatric nonmalarial febrile illness, particularly tests that take into account different comorbidities, such as malnutrition.47
The assay evaluated in the study has a 2-hour analytical time and requires a trained laboratory technician, which precludes its wide use in the outpatient setting, where much antibiotic overuse occurs. Development of more rapid (within minutes) and easy-to-use formats of the assay for the ED and point of care would be valuable and is underway.
Conclusions
The diagnostic performance of a novel assay for distinguishing bacterial from viral infections was validated in febrile children and found to significantly outperform routine laboratory parameters and biomarkers. These promising results merit evaluation in a clinical utility study, to measure the impact of the host-signature assay on actual clinical practice. Taken together with the high performance observed in other independent studies, these data support a new effective tool to help clinicians avoid missing bacterial infections or overusing antibiotics.
Drs Srugo, Klein, and Gevaix substantially contributed to the conception and design of the study, acquisition of data, and analysis and interpretation of data, and drafted the manuscript; Drs Stein, Kerem, Chistyakov, Genizi, Glazer, Yaniv, German, Miron, Shachor-Meyouhas, Paz, Etshtein, Boico, Cohen, Chappuy, Angoulvant, and Lacroix and Ms Kronenfeld substantially contributed to the acquisition of data and analysis and interpretation of data, and revised the manuscript for important intellectual content; Dr Golan-Shany substantially contributed to the acquisition of data and revised the manuscript for important intellectual content; Dr Bamberger substantially contributed to the analysis and interpretation of data and revised the manuscript for important intellectual content; Dr Oved substantially contributed to the conception and design of the study and the analysis and interpretation of data, and revised manuscript for important intellectual content; Dr Gottlieb and Mr Navon substantially contributed to the analysis and interpretation of data, and drafted the manuscript; Dr Eden substantially contributed to the conception and design of the study and the analysis and interpretation of data, and drafted the manuscript; and all of the above-named authors approved the final manuscript as submitted and agree to be accountable for all aspects of the work in ensuring that questions related to the accuracy or integrity of any part of the work are appropriately investigated and resolved.
This trial has been registered at www.clinicaltrials.gov (identifier NCT01911143).
FUNDING: This was an investigator-driven study. MeMed funded and conducted the ImmunoXpert assays.
COMPANION PAPER: A companion to this article can be found online at www.pediatrics.org/cgi/doi/10.1542/peds.2017-1210.
Acknowledgments
We thank Profs Danny Glikman, Shai Ashkenazi, and Itamar Shalit and Dr Liran Shani for scientific comments and for reviewing the manuscript. We thank Dr Sharona Paz for being an expert panel member.
