Basic Statistics for Comparing Categorical Data From 2 or More Groups

One common way of summarizing categorical data are to use proportions (or percent if we multiply the proportion by 100). Suppose we want to compare the proportion of patients in our hospital’s ICU that experience an adverse drug event (ADE) with that of another hospital’s ICU. In our hospital, 20 of 180 patients experienced an ADE (20/180 = 11.1%), whereas in another hospital, 25 of 155 patients (16.1%) experienced an ADE. It looks like these percentages are different (11.1% vs 16.1%), but do we have enough evidence to conclude that they are really different and not due to random chance? To get a better sense for whether the observed differences are due to random chance, we will perform a hypothesis test.

To facilitate our hypothesis test, we need to organize the data into a contingency table (ie, an r × c table where r is the number of rows corresponding to the number of levels for the categorical variable and c is the number of columns corresponding to the number of comparison groups). In our example, the data can be organized into a 2 × 2 table with 2 rows (experienced an ADE: yes vs no) and 2 columns (hospital A and hospital B) as in Table 1.

We would like to know whether the proportion of ICU patients experiencing ADEs is the same at our hospital compared with the other hospital. In statistical terminology, we are testing the following hypotheses:

where π represents the proportion of ICU patients experiencing an ADE. In statistical testing, we always assume that the null hypotheses is true and then determine if we have enough evidence from the data to reject the null in favor of the alternative hypothesis. In other words, we assume the proportion of ICU patients experiencing an ADE at the 2 hospitals is the same until we have sufficient evidence from the data to conclude otherwise.

Generally, 2 main tests are used for comparing categorical data across ≥2 groups: χ² test¹  (sometimes referred to as Pearson’s χ² test of independence) and Fisher’s exact test.²  The development of the χ² test is fairly intuitive. At a high level, we decide how the data would look in our table if the null hypothesis was true (ie, the 2 proportions were equal) and then measure how far off the actual data are from these expected counts. By default, most statisticians use the χ² test because it performs well under most circumstances. However, the validity of the test can come into question when you have small observed numbers in the cells of your table. If this is the case, then you should consider using Fisher’s exact test.

If the null hypothesis is true and the proportions are equal, then we can calculate the counts that we would expect to see in the 4 cells of our table. We can do this by using the total count of cases that we are trying to redistribute (335 in our example), and the marginal proportions displayed in Table 2. To determine the expected number of patients who experienced an ADE in Hospital A (ie, the upper left cell of the table), we simply take the overall population size (335) and multiply it by the proportion of patients who we expect in Hospital A (0.5373) and the proportion of all patients who we would expect to experience an ADE (0.1343). That is, of the 335 patients, we expect 53.73% to be in Hospital A and 13.43% of patients to experience an ADE (335 × 0.5373 × 0.1343 = 27.18 patients). Similar calculations are performed for each of the cells until the expected table is completed, as in Table 3. Notice that the marginal totals are exactly as they were in the original Table 1 but that the percentage of patients in each hospital experiencing an ADE is now equal (24.18/180 = 13.4% and 20.82/155 = 13.4%).

We now know what the table would have looked like if the null hypothesis was true and the proportions were equal (ie, the expected in Table 3), and we know what the data actually looked like (ie, the observed in Table 1). Next, we measure how far the observed was from what was expected and decide if we have enough evidence to reject the null hypothesis. To do this, we calculate the following measure for each cell:

and then sum them to form the χ² statistic:

Finally, we need to determine if the Χ² we calculated is “large enough” to reject the null hypothesis. This is why we need to compute a P value. It tells us the probability of getting a test statistic (Χ²) at least this extreme, assuming that the null hypothesis is true. The test statistic has a χ² distribution, so we compare the value we calculated for Χ² to a χ² distribution. One of the nuances of the χ² distribution is that you need to know the degrees of freedom (df; ie the number of data points available to estimate a parameter of the population) for the test statistic. For the χ² test, the df is (number of rows – 1) × (number of columns – 1). For a 2 × 2 table, the df = 1. Comparing our Χ² = 1.803 to the χ² distribution with df = 1 [using an online calculator like Social Science Statistics: http://www.socscistatistics.com/tests/chisquare/default2.aspx or the EXCEL function = 1-CHISQ.DIST(1.803,1,TRUE)], we get P = .179. Here, the probability of getting a test statistic at least as extreme as the one we calculated is rather high (P > .05), so we fail to reject the null hypothesis and conclude that the proportion of patients experiencing ADEs between hospital A (11.1%) and hospital B (15.6%) are not significantly different.

When you have cells in your table where the expected count is <5, you should be cautious of using the χ² test and instead use Fisher’s exact test to be more conservative. Most statistical packages will give you a warning that the χ² test may be invalid and also compute Fisher’s exact test for you. Although it is valid for all sample sizes, computing Fisher’s exact test is substantially more mathematically involved than doing a χ² test and computationally challenging with tables beyond 2 × 2. Although programming is a bit tricky in Excel, there are online calculators such as Social Science Statistics (http://www.socscistatistics.com/tests/fisher/default2.aspx) that can be used to calculate Fisher’s exact test.

Fisher developed his test as part of the classic “lady tasting tea” experiment that he proposed with Muriel Bristol, who claimed that she could tell if milk was added to a cup before or after her tea. He proposed a randomized experiment with 8 cups of tea (4 with the milk added first, and 4 with the tea added first) given to her in random order. He described his exact test using probability calculations to determine the likelihood that she was simply guessing. She supposedly got them all correct, and the likelihood of getting them all correct if she was guessing was 1.4%. He describes the experiment in his classic book on the design of experiments.³ 

The χ² test can easily be generalized to tables with >2 rows and/or 2 columns. The steps involved and the computations are nearly identical with the appropriate changes made to the df for the number of rows and columns. Table 4 shows a comparison of 30-day asthma readmission rates across 3 hospitals. In this case, the χ² test is evaluating whether the distribution across the 3 hospitals is the same or not. The P = .009 indicates that there are significant differences across the hospitals, but it does not necessarily tell us where these differences occur. To determine where these differences exist, we can break the table into three 2 × 2 tables and do pairwise comparisons. The first 2 × 2 table would compare Hospital A versus Hospital B (P = .002), the second would compare Hospital B versus Hospital C (P = .488), and the last would compare Hospital A versus C (P = .023). A little caution is necessary when performing pairwise comparisons. By doing 3 statistical tests on the 2 × 2 tables, you are increasing your likelihood of coming to an erroneous conclusion by committing a type I error (ie, erroneously rejecting a true null hypothesis of equal proportions). One easy way to protect from this is by doing a Bonferroni correction,⁴  which reduces the significance level for each individual test from the traditional P = .05 level, and shares this across the multiple tests. Because we are performing 3 tests, we would reduce the significance level to 0.05/3 = 0.017 for each test. With this new level of significance, we would conclude that Hospitals A and B are statistically different, but none of the other pairs are. Although the Bonferroni correction can be overly conservative, it is intuitive and easy to perform. Another common approach is to control for the false discovery rate (ie, the expected proportion of tests that reject the null hypothesis erroneously) using the Benjamini-Hochberg procedure.⁵ 

It is important to understand how populations differ on baseline characteristics before assessing outcomes so that these differences can be accounted for appropriately in the analysis. The χ² test is an intuitive test that is easy to calculate, and it is useful for comparing proportions across groups for categorical variables. With this test, you can decide if there are important differences that may confound your results and take appropriate steps to avoid this.

ADE

adverse drug event

df

degrees of freedom