Development and Validation of a Cellulitis Risk Score: The Melbourne ASSET Score

This was a prospective cohort study in a convenience sample of children presenting with cellulitis to the ED at the Royal Children’s Hospital in Melbourne. Recruitment occurred over a 15-month period from January 2014 to May 2015. Children aged 6 months to 18 years were eligible if an ED clinician made a diagnosis of cellulitis. The decision to treat with oral or IV antibiotics was made by an experienced ED physician (at least at the registrar and/or fellow level). Per routine institutional practice, patients who were treated with oral antibiotics were discharged from the ED, whereas those who were started on IV antibiotics were admitted to the hospital. Exclusion criteria were children with complicated cellulitis or associated toxicity. Complicated cellulitis was defined as cellulitis associated with the following conditions: orbital cellulitis, undrained abscess, penetrating injury, immunosuppression, fasciitis or foreign body, or cellulitis caused by large animal or human bite. Toxicity was defined as those with signs or symptoms of hypotension, resting tachycardia, or poor central perfusion.

We correlated clinical features and outcomes to derive a cellulitis scoring system and, when appropriate, take into account clinicians’ opinions and practice on the basis of a survey. This study was approved by the institutional human research ethics committee (HREC34018).

Data collection occurred in real time as patients were being assessed in the ED after written consent was obtained. This included demographics, whether previous oral antibiotics were taken before presenting to the ED, and clinical features at presentation. Clinical features were collected from ED clinicians on a standardized proforma and consisted of the size of the area affected (longest diameters in length and width measured in centimeters), the absence or presence (0 or 1, respectively) of functional impairment, lymphangitis and/or tracking, systemic features, and periorbital cellulitis. Additionally, clinicians were asked to rate 3 features of cellulitis on a 3-point scale (absent = 0, mild = 1, and moderate to severe = 2) for the following features: erythema, tenderness, and swelling. Patients were recorded as either starting IV or oral antibiotics; the duration administered was also recorded.

To devise and validate the clinical scoring system, there needs to be a standard against which the decision to prescribe IV or oral antibiotics is judged. Because a gold standard for the appropriateness of this decision does not exist, a consensus was reached among the study investigators at our institution who had the relevant expertise and represented pediatric infectious diseases, general pediatrics, pediatric emergency medicine, and biostatistics. We determined that the route of ongoing antibiotic treatment after review at 24 hours was likely to be the correct one on the basis of ongoing symptom progression even if the patients had been started via the other route at presentation. Patients were therefore divided into 2 groups based on their ongoing route of antibiotic administration at 24 hours and having needed that route from the start regardless of their initial management route: IV or oral at 24 hours.

We conducted follow-up with patients within 48 hours to ascertain their route of ongoing antibiotic treatment (either oral or IV at 24 hours) by checking the hospital attendance record electronically. In addition, within 14 days after the initial ED presentation, we contacted all patients by telephone to follow-up and ascertain their outcomes.

This survey was performed at The Royal Children’s Hospital in Melbourne over a period of 4 weeks in December 2014. Participants (selected on the basis of exposure to cellulitis cases in their practices) were acute-care pediatricians from the following departments: ED, general medicine, infectious diseases, and adolescent and developmental medicine. Excluded participants were pediatric clinicians in subspecialties in which cellulitis cases would not be managed and clinicians with a predominantly academic role. Participants were contacted via their respective hospital-based e-mail address, whereby every participant was given a link to the Web-based survey via Research Electronic Data Capture (hosted at the Murdoch Children’s Research Institute).²⁶ 

All data were entered into a Research Electronic Data Capture database. A univariate analysis was performed in which we compared the demographic and clinical features of the 2 groups: IV versus oral antibiotics at 24 hours. A χ² test was used for categorical variables, and a t test was used for continuous data. By using features that were significantly different, receiver operating characteristic (ROC) curves were then calculated for various combinations of the different cellulitis features. The size affected, measured in centimeters for length and width, was converted to a percentage of the body surface area affected on the basis of each child’s height. Proportions of survey responses were calculated. All statistical analysis was performed by using Stata/IC version 15.0 (Stata Corp, College Station, TX). A sample size calculation was not performed at the outset because of the exploratory nature of this study. We aimed to recruit at least 100 patients in each group for each part of the study.

There were 285 children in the derivation cohort. Of these, 171 (60%) received oral antibiotics, and 114 (40%) received IV antibiotics at initial presentation. Of those who were started on oral antibiotics, 10 (6%) re-presented within 24 hours and were deemed to require IV antibiotics. Of those who were started on IV antibiotics, 14 (8%) were switched to oral antibiotics within 24 hours. Therefore, 175 of the 285 (61%) were receiving oral antibiotics at 24 hours, and 110 of the 285 (39%) were receiving IV antibiotics at 24 hours. Clinical features at presentation were compared between the 2 groups (Table 1). Only age and sex did not differ between the IV and oral groups at 24 hours. There were 9 features that differed significantly between the groups. These were converted either to a binary score of 0 or 1 (absent or present, respectively; previous oral antibiotics, systemic features, area affected ≥1% of body surface area, functional impairment, lymphangitis, and periorbital cellulitis) or a score on a 3-point scale (0 = absent, 1 = mild, or 2 = moderate to severe; erythema, swelling, and tenderness).

There were 106 of 138 (77%) clinicians who returned the cellulitis questionnaire. Of these, 61% were consultants or fellows, whereas 39% were trainee doctors. For the purpose of deriving a clinical score, we considered only responses from senior clinicians with consultant or fellow experience. The features clinicians usually used when deciding the route of antibiotic administration were lymphangitis and/or tracking (86%), functional impairment (76%), systemic features (78%), whether the patient had received previous oral antibiotics (70%), the size of the affected area (63%), whether the site affected was periorbital (52%), swelling (52%), and tenderness (48%). Features that were not commonly used by clinicians were erythema (25%) and family preference (2%).

Various combinations and weighting of the 9 features that differed between the 2 groups were used to plot ROC curves and calculate the area under the curve (AUC), sensitivity, and specificity for each combination. The AUC for all 9 features was 0.89 (95% confidence interval [CI] 0.85–0.93), which is reassuringly high (Fig 1). However, having 9 features to score is impractical and inconvenient for clinicians. We therefore iteratively reduced the features by 1 to determine the minimum number of features required to maintain a high AUC. We found that the AUC remained high until the combination was reduced to <5 features (Fig 1). With only 4 features, the lower limit of the 95% CI of the AUC dropped to 0.77, so the minimum number of features for this clinical score was determined to be 5. We tested multiple combinations of 5 features because each significantly differed between the groups, and the AUC results (and therefore sensitivities and specificities) were similar (data not shown). Because there was no mathematical difference, the next stage was to use clinical reasons to determine the 5 most useful features. The 4 features that were removed were erythema, lymphangitis, functional impairment, and previous oral antibiotics. Erythema is difficult to assess in darker skin, therefore limiting the wide applicability of the score, and was only used by 25% of senior clinicians. Lymphangitis is an uncommon sign at presentation and is specific to limb cellulitis.⁸ Additionally, although 86% of survey respondents said they would use this feature when considering IV antibiotics, 12 of 29 (41%) patients who had lymphangitis were on oral antibiotics at 24 hours. Functional impairment of the limb occurs because of significant swelling or tenderness, which are both already represented in the score; in addition, this feature is specific to limb cellulitis. Previous oral antibiotic administration was documented, but in 54 of 116 (47%) patients, the type of antibiotic, dose, frequency, and duration were unknown, and parent recall of dosage is not necessarily reliable.²⁸ The validity of this as a marker of needing IV treatment was therefore uncertain, potentially relating more to the perceived need by physicians to change something at presentation to ED. The 5 remaining features were used to address specific factors relating to the potential need for IV antibiotics: risk of sepsis (systemic features), severity and/or extent of infection (size, swelling, and tenderness), and risk of orbital cellulitis (eye involvement).

The optimal score therefore consisted of the following features: size of the affected area, systemic features, severity of swelling, periorbital and/or eye involvement, and severity of tenderness, which can be recalled with the acronym ASSET. The Melbourne ASSET Score has a maximum possible score of 7. The AUC was 0.86 (95% CI 0.83–0.91; Fig 2). Using a cutoff score of 4 (patients with a score of ≥4 receive IV antibiotics and those with a score <4 receive oral antibiotics) yields the highest proportion of patients who are correctly classified at 80% (sensitivity 60%; specificity 93%; Supplemental Table 2, Fig 3). If the cutoff score were reduced to a more conservative 3, the sensitivity would increase to 85%, but the specificity would decrease to 76%. By using a frequency distribution graph (Fig 4), a score of 4 would result in 11 (10%) patients receiving unnecessary IV treatment, and 44 (25%) patients on oral antibiotics may represent needing IV antibiotics. Lowering the cutoff to 3 would result in 44 (40%) patients receiving unnecessary IV treatment and 17 (10%) representing needing IV antibiotics.

A subsequent cohort of patients presenting to the ED with cellulitis with the same inclusion and exclusion criteria was used to validate the score. Data collection methods and outcome rules were the same as for the derivation cohort. There were 251 children in the validation cohort. When using the same gold standard to define the appropriate route of antibiotic administration, 107 of the 251 (43%) children were receiving oral antibiotics at 24 hours, and 144 of the 251 (57%) children were receiving IV antibiotics at 24 hours.

When the Melbourne ASSET Score was retrospectively applied to the validation cohort, the AUC remained high at 0.83 (95% CI 0.78–0.89; Fig 2). A cutoff score of 4 as the threshold to start IV antibiotics has a sensitivity of 85% and a specificity of 63%, with which 76% of patients were correctly classified. A cutoff of 3 has a sensitivity of 98% and a specificity of 32%, with which 70% of patients were correctly classified.

The results of this study reveal that children who were started on IV treatment have more severe features of cellulitis than those who were started on oral antibiotics, which is consistent with previous literature.⁸ Using these features, we were able to derive a score and accurately classify 80% of patients. Importantly for pragmatic use, it is simple to use and widely applicable. Crucially, when the Melbourne ASSET Score was applied to a subsequent unrelated cohort of children, it still could be used to correctly classify a high proportion of patients. Interestingly, this was despite the sensitivity and specificity of the score cutoff of 4 being different between the 2 cohorts. This is useful information because the trade-off between sensitivity and specificity will be different in different populations and settings.

The score comprises clinical features that clinicians routinely assess for and document in cellulitis without needing any investigations or other measures. This is the first study of cellulitis in which the authors recommend assessing the area involved as >1% or <1% by using the size of the patient’s own hand,²⁷ and we support this with our data. The site affected was measured by using a tape measure during the study, which is an onerous and time-consuming task for busy physicians, with traditional conversion to percentage of total body surface area also requiring a child’s height and weight and a calculator. Using this novel hand-size method to assess the contribution of the area of cellulitis to the score is simple and convenient. A minimum of 5 features maintained a high AUC, sensitivity, and specificity and is also correlated with the number of fingers on 1 hand, making it easy for clinicians to check off the number of features (Fig 5). All of these features ensure that a child with cellulitis can be examined rapidly in the ED or primary care office with a score that is easy to calculate; both are imperative for it to be useful.²⁹ 

The 5 features of the Melbourne ASSET Score are each important in the consideration of risk of more severe infection. First, the size of the area involved reflects the burden of infection. Second, the presence of systemic features (fever and lethargy) potentially reflects sepsis. It is 1 of the reasons that the authors of the Infectious Diseases Society of America guidelines for adults recommend that systemic features are used to guide the need for IV antibiotics. Third, swelling reflects the severity of inflammation and can also represent induration in the early formation of abscess. Fourth, the eye is a vulnerable location, and it is not always straightforward to differentiate orbital infection, which has more severe morbidity. Fifth, tenderness also reflects the severity of inflammation, and as a bonus, it can be used to differentiate cellulitis from other inflammatory conditions, such as allergic reactions, which are typically nontender. Several features considered to be important in some contexts were excluded from the score as detailed, and not necessarily because they have no value in individual patients. In addition to those mentioned above, the age of the child is an important consideration in any pediatric condition. However, whether analyzed on a continuum or stratified by different age categories, age was not different between the groups.

A hurdle for us in this study, like for many others attempting to devise a clinical score, is the absence of a gold standard for cellulitis against which to assess the true requirement for IV antibiotics. The rationale behind using the status at 24 hours is that if a change in route has already been made within 24 hours, the features at presentation could likely be used to predict this. It is possible that clinicians who treat children with IV antibiotics may unnecessarily treat beyond 24 hours simply because there is IV access in situ. This would result in a slightly more conservative gold standard than true need, which is similar to other scoring systems.¹⁸ If found to be too conservative when validated in a different setting, the score cutoff can be refined.^30,31 Similarly, if using this score in settings in which patients live farther from the hospital, minimizing ED reattendance may be more important, so the cutoff could be lowered.

Our study has some limitations. Firstly, it was conducted in a tertiary pediatric hospital with experienced pediatric ED clinicians, and a more conservative approach may be required in a different setting. However, we have been able to use the expertise of these clinicians in the absence of a gold standard to derive and validate a clinical score for cellulitis in children for the first time. If a more conservative approach is desired, the cutoff threshold for IV treatment can be lowered. Second, our region has a low prevalence of methicillin-resistant Staphylococcus aureus, which is similar to many other pediatric populations. However, although we would recommend external validation of the score in high-prevalence areas, there is no reason to suspect that this score would not be applicable because it is the defining route, not choice, of antibiotic. Lastly, with this clinical score, we do not aim to replace clinical assessment for each individual patient but to aid in decision-making.²² 

The Melbourne ASSET Score is the first risk assessment scoring system for pediatric cellulitis that is proposed to aid clinicians in deciding whether to treat with IV or oral antibiotics. It is simple, easy to use, applicable, and reliable. Although intended for widespread use, if limitations exist in other settings, it is designed to allow for refinement and is amenable to local impact analysis. We propose an impact analysis of this score, ideally in a different setting and population.

ASSET

Area, Systemic features, Swelling, Eye, Tenderness

AUC

area under the curve

CI

confidence interval

ED

emergency department

IV

intravenous

ROC

receiver operating characteristic

We acknowledge the participation of the patients and families.