Fisher's Exact Test - Biostatistics

SECTION 1

Historical Context & Motivation

In the early decades of the twentieth century, statisticians increasingly confronted the problem of analyzing categorical data arranged in contingency tables. The chi-squared test, introduced by Karl Pearson in 1900, provided an asymptotic framework for evaluating the independence of two categorical variables. However, the chi-squared approximation becomes unreliable when expected cell frequencies are small — a situation that arises routinely in clinical trials, genetics studies, and epidemiological investigations where sample sizes are limited. This shortcoming motivated the search for a method that would yield exact probabilities without relying on large-sample approximations.

The solution arrived through the work of Sir Ronald A. Fisher, one of the most influential statisticians of the twentieth century. Fisher devised an elegant test that computes the probability of obtaining any particular arrangement of cell counts in a 2×2 table, conditional on the fixed marginal totals. The famous illustration of this method involved a colleague, Dr. Muriel Bristol, who claimed she could distinguish whether milk or tea had been poured into the cup first — an anecdote that has become one of the most celebrated thought experiments in the history of statistics.

1900 • Pearson's Chi-Squared Test

Karl Pearson introduces the chi-squared goodness-of-fit test, laying the foundation for asymptotic categorical data analysis. The test relies on large-sample theory for its distributional approximation.

1922 • Fisher's Foundational Papers

R.A. Fisher publishes seminal works on the mathematical foundations of statistics, including maximum likelihood estimation, which frame the theoretical landscape for exact inference methods.

1935 • The Lady Tasting Tea

Fisher publishes The Design of Experiments, presenting the exact test through the famous tea-tasting experiment. This work formalizes the hypergeometric probability model for 2×2 tables with fixed margins.

1970s • Computational Feasibility

With the advent of digital computing, Fisher's exact test becomes practical for routine use. Software implementations eliminate the need for manual combinatorial calculations, democratizing the method for applied researchers.

2000s–Present • Modern Extensions

Generalizations to r×c tables and network-based algorithms extend exact testing beyond the 2×2 case. Fisher's exact test remains a standard tool in genomics, clinical trials, and any setting with sparse categorical data.

The central question Fisher addressed is deceptively simple: given the observed marginal totals of a 2×2 contingency table, how likely is it to observe the specific arrangement of cell counts — or one more extreme — if the row and column variables are truly statistically independent? By computing this probability exactly rather than relying on an asymptotic χ² distribution, Fisher's test provides valid inference even when cells contain very few observations, making it indispensable in modern biostatistical practice.

SECTION 2

Core Principles & Definitions

Fisher's exact test rests upon a set of interlocking principles that distinguish it from approximate methods like the chi-squared test. Understanding these principles is essential before engaging with the mathematical machinery, because the test's validity hinges on the specific probabilistic model it invokes — namely, the hypergeometric distribution applied to the cell counts of a 2×2 contingency table with fixed marginal totals.

Conditioning on Margins

Fisher's test treats the row and column marginal totals as fixed. Under this conditioning, only one cell is free to vary in a 2×2 table, and the probability of each possible table configuration is computed exactly via the hypergeometric distribution.

Exact Probability Calculation

Unlike the chi-squared test, which approximates the sampling distribution, Fisher's test calculates the exact probability of observing the data (or more extreme configurations) under the null hypothesis of independence. No minimum expected cell count is required.

Null Hypothesis of Independence

The null hypothesis (H₀) posits that the two categorical variables are statistically independent — knowing one variable provides no information about the other. The alternative hypothesis (H₁) asserts that an association exists.

The Hypergeometric Model

When margins are fixed, the distribution of any single cell count follows the hypergeometric distribution. This is analogous to drawing colored balls from an urn without replacement — the exact model underlying the test's combinatorial logic.

One-Tailed vs. Two-Tailed

A one-tailed test evaluates association in a specific direction, while a two-tailed test assesses association in either direction by summing probabilities of all tables at least as extreme as the observed one, regardless of the direction of deviation.

✦

KEY TAKEAWAY

Think of Fisher's exact test like counting all possible ways to deal cards into a specific layout. Imagine you have a fixed number of red and black cards (row totals) and two piles of fixed size (column totals). Under the null hypothesis, the cards are randomly assigned to piles. Fisher's test enumerates every possible deal and computes the exact fraction of deals that are as extreme as — or more extreme than — the one you actually observed. If that fraction is very small, the arrangement is unlikely to have arisen by chance alone, and you reject independence.

SECTION 3

Visual Explanation

The following diagram illustrates the structure of a 2×2 contingency table and the relationship between the cell counts, marginal totals, and the grand total. Because the marginals are treated as fixed, only one cell value (here, cell a) is free to vary; the remaining three cells are determined by subtraction. The diagram also shows how the hypergeometric probability is constructed from the factorial terms associated with each margin and cell.

A 2×2 contingency table with cell counts a, b, c, d. The row totals (a + b and c + d) and column totals (a + c and b + d) are treated as fixed. The grand total is n. Only one cell (marked FREE) is free to vary under the constraint.

As the diagram makes clear, once the four marginal totals are fixed, specifying any single cell — say, cell a — automatically determines the other three by subtraction: b = (a + b) − a, c = (a + c) − a, and d = n − a − b − c. This means there is only one degree of freedom in the table, and the test evaluates all possible values of a under H₀ to compute the p-value.

SECTION 4

Mathematical Framework

The mathematical foundation of Fisher's exact test is the hypergeometric probability of observing a particular cell count a given fixed marginal totals. We denote the row totals as R₁ = a + b and R₂ = c + d, the column totals as C₁ = a + c and C₂ = b + d, and the grand total as n. The probability of the observed table — or equivalently, the probability that cell a takes a particular value — is given by the following expression.

HYPERGEOMETRIC PROBABILITY

P(a) = C(R₁, a) × C(R₂, C₁ − a) / C(n, C₁)

where C(n, k) = n! / (k!(n − k)!) is the binomial coefficient, R₁ = a + b (row 1 total), R₂ = c + d (row 2 total), C₁ = a + c (column 1 total), and n is the grand total.

An equivalent formulation, often seen in textbooks, expresses the probability directly in terms of factorials of the margins and cells.

FACTORIAL FORM

P = (R₁! × R₂! × C₁! × C₂!) / (n! × a! × b! × c! × d!)

This is algebraically equivalent to the binomial-coefficient form and is often more convenient for manual computation. The numerator contains the product of the four marginal factorials, and the denominator contains the grand total factorial multiplied by the four cell factorials.

To compute the p-value, one sums the hypergeometric probabilities of all table configurations that are as extreme as or more extreme than the observed table. For a one-tailed test in the direction of a positive association, this means summing over all values of a from the observed value up to min(R₁, C₁). For a two-tailed test, one also includes tables in the opposite tail whose individual probabilities are less than or equal to the probability of the observed table.

ONE-TAILED P-VALUE (UPPER TAIL)

p = Σ P(k) for k = a_obs, a_obs + 1, ..., min(R₁, C₁)

The summation runs over all values of k from the observed cell count a_obs to the maximum possible value of a given the marginal constraints.

RANGE OF CELL a

max(0, C₁ − R₂) ≤ a ≤ min(R₁, C₁)

The lower bound ensures non-negative counts in all cells; the upper bound ensures that a does not exceed either the row 1 total or the column 1 total. Enumerating all values of a in this range produces every possible table under the given marginals.

SECTION 5

Probability Landscape of All Possible Tables

To build intuition for how the p-value is computed, consider a concrete scenario. Suppose we observe a 2×2 table with marginals R₁ = 5, R₂ = 7, C₁ = 4, and C₂ = 8 (n = 12). The free cell a can range from max(0, 4 − 7) = 0 to min(5, 4) = 4, giving five possible table configurations. The diagram below plots the hypergeometric probability of each configuration and shows which tables are included in one-tailed and two-tailed p-value calculations when the observed value is a = 4.

Hypergeometric probability distribution for a table with marginals R₁ = 5, R₂ = 7, C₁ = 4, C₂ = 8. Each bar represents the probability of that value of cell a under H₀. If a_obs = 4, the one-tailed p-value sums P(3) + P(4) = 0.088 + 0.008 = 0.096.

The bar chart reveals that under independence, the most probable table configurations are a = 1 and a = 2, which together account for over 76% of the probability mass. The observed value a = 4 lies in the extreme right tail with a probability of only 0.008. The one-tailed p-value (testing whether a is unusually large) sums P(a ≥ 3) = 0.088 + 0.008 = 0.096. Since this exceeds the conventional α = 0.05 threshold, we would fail to reject H₀ at the 5% level in this example, though the result would be significant at α = 0.10.

SECTION 6

Worked Example: Drug Efficacy Trial

Consider a small clinical trial in which 12 patients are randomized to either a new drug (n = 5) or placebo (n = 7). The primary outcome is binary: improvement (yes/no). Suppose 3 of the 5 drug patients and 1 of the 7 placebo patients improve. We wish to test whether the drug is associated with improvement using Fisher's exact test at the α = 0.05 significance level (one-tailed, since we are testing whether the drug improves outcomes).

	Improved	Not Improved	Total
Drug	3 (a)	2 (b)	5
Placebo	1 (c)	6 (d)	7
Total	4	8	12

Observed 2×2 contingency table for drug efficacy trial

Fisher's Exact Test: Drug vs. Placebo

1
Step 1 — Identify Margins and Cell Range

The marginals are R₁ = 5, R₂ = 7, C₁ = 4, C₂ = 8, and n = 12. The free cell a (Drug × Improved) ranges from max(0, 4 − 7) = 0 to min(5, 4) = 4. The observed value is a = 3.

a ∈ {0, 1, 2, 3, 4}, a_obs = 3
2
Step 2 — Compute P(a = 3) Using the Factorial Formula

P(a = 3) = (R₁! × R₂! × C₁! × C₂!) / (n! × a! × b! × c! × d!) = (5! × 7! × 4! × 8!) / (12! × 3! × 2! × 1! × 6!). Computing the numerator: 120 × 5040 × 24 × 40320 = 585,252,864,000. Computing the denominator: 479,001,600 × 6 × 2 × 1 × 720 = 4,137,533,952,000. Therefore P(3) = 585,252,864,000 / 4,137,533,952,000.

P(a = 3) ≈ 0.1414
3
Step 3 — Compute P(a = 4) for the More Extreme Table

P(a = 4) = (5! × 7! × 4! × 8!) / (12! × 4! × 1! × 0! × 7!) = 585,252,864,000 / (479,001,600 × 24 × 1 × 1 × 5040). The denominator = 57,915,973,017,600. Wait — let us recompute carefully. Denominator = 479,001,600 × 24 × 1 × 1 × 5040 = 57,915,973,017,600. This is incorrect; let's use the ratio approach. P(4)/P(3) = [b × c] / [(a+1)(d+1)] = [2 × 1] / [4 × 7] = 2/28 = 1/14 ≈ 0.0714. So P(4) = 0.1414 × 0.0714 ≈ 0.0101.

P(a = 4) ≈ 0.0101
4
Step 4 — Sum Probabilities for One-Tailed P-Value

The one-tailed p-value (testing for a positive association between drug and improvement) is the sum of probabilities for tables at least as extreme as the observed table in the direction of interest: p = P(a = 3) + P(a = 4) = 0.1414 + 0.0101.

p = 0.1515
5
Step 5 — Decision and Interpretation

Since p = 0.1515 > α = 0.05, we fail to reject the null hypothesis of independence. Although a higher proportion of drug patients improved (60% vs. 14%), the sample size is too small to conclude that this difference is statistically significant at the 5% level. The result suggests the trial may be underpowered, and a larger sample size could be warranted.

Fail to reject H₀ at α = 0.05

💡

Computational Tip

For manual calculations, the recurrence relation P(a+1) = P(a) × [b × c] / [(a+1)(d+1)] allows you to compute successive probabilities from any starting value without recomputing all factorials. Most statistical software (R's fisher.test(), Python's scipy.stats.fisher_exact()) handles this automatically.

SECTION 7

Fisher's Exact Test vs. Chi-Squared Test

A natural question arises: when should one use Fisher's exact test rather than the more familiar Pearson's chi-squared test? The answer depends on sample size, expected cell counts, and the specific inferential context. The table below summarizes the key distinctions between these two approaches to testing independence in 2×2 contingency tables.

Feature	Fisher's Exact Test	Chi-Squared Test
Probability model	Hypergeometric (exact)	χ² approximation (asymptotic)
Sample size requirement	None — valid for any sample size	All expected counts ≥ 5 (rule of thumb)
Computational cost	Higher for large n (combinatorial enumeration)	Low — single formula
Type I error control	Conservative (actual α ≤ nominal α)	Liberal when expected counts are small
Marginals	Conditions on fixed margins	Does not require fixed margins
Extensions	Generalizable to r×c tables (computationally expensive)	Naturally extends to r×c and multi-way tables
When to prefer	Small samples, sparse cells, regulatory submissions	Large samples, exploratory analysis

Comparison of Fisher's exact test and Pearson's chi-squared test for 2×2 tables

✦

KEY TAKEAWAY

Fisher's exact test is analogous to counting every possible arrangement of items in labeled boxes, while the chi-squared test is like estimating the distribution of those arrangements using a smooth continuous curve. When there are enough items (large n), the smooth approximation works well and is computationally efficient. But when items are few, the smooth curve oversimplifies the discrete, lumpy reality, and only the exact enumeration can be trusted. As a general rule, favor Fisher's exact test whenever any expected cell count falls below 5, or whenever precise p-values are required (e.g., regulatory submissions to the FDA).

SECTION 8

Connections to Advanced Methods

Fisher's exact test for 2×2 tables is the simplest member of a broader family of exact conditional tests. Understanding its relationship to more advanced methods helps situate the test within the larger statistical ecosystem and prepares the student for extensions encountered in advanced biostatistics coursework.

Feature	Fisher's Exact Test (2×2)	Advanced Extensions
Table dimension	2×2 only	Fisher–Freeman–Halton test extends to r×c tables
Stratification	Single 2×2 table	Cochran–Mantel–Haenszel test pools evidence across K strata
Effect estimation	Provides p-value only (no direct effect estimate)	Exact logistic regression provides conditional maximum likelihood estimates and exact confidence intervals
Multiple testing	Single comparison	Permutation tests and exact methods in genomics handle thousands of simultaneous comparisons
Bayesian analog	Frequentist exact inference	Bayesian analysis of contingency tables with Dirichlet priors provides posterior probability of association

Fisher's exact test in the landscape of categorical data methods

A particularly important connection is to exact logistic regression, which can be understood as a multivariate generalization of Fisher's exact test. Where Fisher's test conditions on the marginals of a single 2×2 table, exact logistic regression conditions on the sufficient statistics of the logistic model to obtain exact conditional distributions for the regression coefficients. This method is invaluable when modeling binary outcomes with small samples and multiple predictors — a scenario where standard maximum likelihood logistic regression may fail to converge.

Additionally, the Barnard's test offers an unconditional alternative to Fisher's test. Rather than fixing both sets of margins, Barnard's test fixes only one margin and maximizes the p-value over nuisance parameters. Although Barnard's test can be more powerful than Fisher's in certain configurations, it is computationally more demanding and less widely implemented in standard software, which explains Fisher's test's continued dominance in practice.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why Fisher's exact test conditions on the marginal totals being fixed. What probabilistic model does this conditioning invoke, and why does it matter for the test's validity with small samples?

Show answer

Conditioning on fixed marginals transforms the problem into one governed by the hypergeometric distribution, which counts the number of ways to distribute items into categories given fixed group sizes. This conditioning eliminates nuisance parameters (the marginal probabilities) and ensures that the probability of any table configuration can be computed exactly without relying on asymptotic approximations. For small samples, asymptotic methods like the chi-squared test may produce inaccurate p-values because the discrete distribution of the test statistic does not closely approximate the continuous χ² distribution. By computing exact combinatorial probabilities, Fisher's test maintains valid Type I error control regardless of sample size.

Question 2: BASIC CALCULATION

Given a 2×2 table with a = 4, b = 1, c = 2, d = 8, compute the exact probability of observing this particular table configuration using the factorial formula P = (R₁! × R₂! × C₁! × C₂!) / (n! × a! × b! × c! × d!).

Show answer

First, compute the marginals: R₁ = 4 + 1 = 5, R₂ = 2 + 8 = 10, C₁ = 4 + 2 = 6, C₂ = 1 + 8 = 9, and n = 15. Now apply the formula: P = (5! × 10! × 6! × 9!) / (15! × 4! × 1! × 2! × 8!). Numerator: 120 × 3,628,800 × 720 × 362,880 = 1.137 × 10¹⁴. Denominator: 1,307,674,368,000 × 24 × 1 × 2 × 40,320 = 2.530 × 10¹⁵. Therefore P ≈ 1.137 × 10¹⁴ / 2.530 × 10¹⁵ ≈ 0.0449. This single-table probability tells us the likelihood of observing exactly this arrangement under H₀, but note that the p-value would require summing over all tables at least as extreme.

Question 3: INTERMEDIATE

A researcher studies the association between a genetic variant (present/absent) and disease status (case/control) in 20 participants. The observed table is: variant-present cases = 6, variant-present controls = 2, variant-absent cases = 4, variant-absent controls = 8. Enumerate all possible tables under the observed marginals and compute the two-tailed p-value for Fisher's exact test.

Show answer

Marginals: R₁ = 8 (variant present), R₂ = 12 (variant absent), C₁ = 10 (cases), C₂ = 10 (controls), n = 20. Cell a ranges from max(0, 10 − 12) = 0 to min(8, 10) = 8. Using the hypergeometric formula, we compute P(a) for a = 0 through 8. Key probabilities: P(0) = 0.000108, P(1) = 0.00389, P(2) = 0.0390, P(3) = 0.154, P(4) = 0.294, P(5) = 0.282, P(6) = 0.162, P(7) = 0.0540, P(8) = 0.00810. The observed table has a = 6 with P(6) = 0.162. For the two-tailed p-value, we sum probabilities of all tables with P ≤ P(6): these include a = 0 (0.000108), a = 1 (0.00389), a = 2 (0.0390), a = 7 (0.0540), a = 8 (0.00810), and a = 6 (0.162) itself, but a = 3 (0.154) also has P ≤ 0.162. The two-tailed p = 0.162 + 0.154 + 0.0540 + 0.0390 + 0.00810 + 0.00389 + 0.000108 ≈ 0.421. We fail to reject H₀ at α = 0.05.

Question 4: APPLIED

In a pilot study for a new antibiotic, 8 patients with antibiotic-resistant infections receive the new drug and 6 receive the standard treatment. Bacterial clearance is observed in 5 of the new drug patients and 1 of the standard treatment patients. Perform a one-tailed Fisher's exact test (testing whether the new drug has a higher clearance rate) at α = 0.05. Should the researchers proceed to a Phase II trial based on this result?

Show answer

The observed table: a = 5 (drug + cleared), b = 3 (drug + not cleared), c = 1 (standard + cleared), d = 5 (standard + not cleared). Marginals: R₁ = 8, R₂ = 6, C₁ = 6, C₂ = 8, n = 14. Cell a ranges from max(0, 6 − 6) = 0 to min(8, 6) = 6. We compute P(a) for a = 5 and a = 6 (tables as or more extreme than observed in the direction of interest). P(5) = (8! × 6! × 6! × 8!) / (14! × 5! × 3! × 1! × 5!) ≈ 0.1119. P(6) = (8! × 6! × 6! × 8!) / (14! × 6! × 2! × 0! × 6!) ≈ 0.00933. One-tailed p = 0.1119 + 0.00933 ≈ 0.1212. Since p = 0.121 > 0.05, we fail to reject H₀ at the 5% level. However, the clinical effect size is substantial (62.5% vs. 16.7% clearance), and the lack of significance likely reflects low statistical power due to the small sample size. The researchers could reasonably proceed to Phase II with a larger sample to confirm the promising signal.

Question 5: CRITICAL THINKING

Fisher's exact test is known to be conservative — its actual Type I error rate is often below the nominal α. Explain the source of this conservatism mathematically, and discuss how the mid-p correction addresses this issue. Under what circumstances might a biostatistician prefer the standard (conservative) Fisher's test despite the availability of the mid-p approach?

Show answer

The conservatism arises because the hypergeometric distribution is discrete, so the actual achievable significance level (the largest p-value not exceeding α) is often strictly less than α. In essence, the cumulative probability 'jumps' at each possible table, and rarely does a jump land exactly at α. The mid-p correction mitigates this by counting only half the probability of the observed table: mid-p = P(more extreme) + ½ × P(observed). This yields actual Type I error rates closer to the nominal α, improving power without dramatically inflating error. However, a biostatistician might prefer the standard Fisher's test in regulatory contexts (e.g., FDA submissions) where conservative inference is viewed as a safeguard against false positives, or in confirmatory studies where the consequences of a Type I error are severe. The conservative approach guarantees that the probability of rejecting a true null never exceeds α, which is especially important in drug approval processes where patient safety is paramount.

SUMMARY

Summary & Review

Summary

Fisher's exact test is a non-parametric statistical test that evaluates the independence of two categorical variables in a 2×2 contingency table by computing exact probabilities under the hypergeometric distribution. Unlike the chi-squared test, which relies on large-sample approximations, Fisher's test is valid for any sample size and does not require a minimum expected cell count. The test conditions on the fixed marginal totals of the table, reducing the problem to a single free cell whose distribution under the null hypothesis is fully specified.

The p-value is computed by summing the probabilities of all tables as extreme as or more extreme than the observed table. The test may be conducted as one-tailed or two-tailed depending on the research hypothesis. Although Fisher's test is conservative due to the discreteness of the hypergeometric distribution, the mid-p correction can bring the actual Type I error rate closer to the nominal level. The test remains a cornerstone of biostatistical practice, indispensable in clinical trials, genetics, and epidemiology whenever sample sizes are small or data are sparse.