False Discovery Rate (FDR) - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The rise of high-throughput technologies in the late twentieth century — from gene expression microarrays to neuroimaging — presented statisticians with a new and urgent challenge: how to draw reliable inferences when thousands or even millions of hypotheses are tested simultaneously. Classical approaches to multiple testing correction, such as the Bonferroni method, controlled the probability of making even one false rejection (the familywise error rate, or FWER), but this stringent control came at a steep cost: as the number of tests grew, the power to detect genuinely significant effects shrank to practically zero. Researchers needed a framework that acknowledged some false positives as an acceptable trade-off for greater statistical power.

The intellectual groundwork stretched back decades, through early concerns about multiplicity in experimental design and the limitations of the Bonferroni inequality in large-scale testing. By the 1990s, genomics laboratories were routinely running experiments that tested tens of thousands of genes for differential expression, making existing FWER methods untenable. The field was primed for a paradigm shift — one that would redefine what it means to control error in a principled yet practical way.

1961 • Simes and Early Multiplicity Concerns

R. J. Simes and other statisticians began formalizing concerns about conducting many simultaneous tests. Early work on ordered p-values laid the mathematical groundwork for step-up procedures that would later be central to FDR control.

1979 • Holm's Sequential Bonferroni

Sture Holm introduced a step-down procedure improving on the Bonferroni correction, providing uniformly more powerful FWER control. This highlighted the potential for procedures based on ordered p-values to gain power while maintaining error guarantees.

1995 • Benjamini–Hochberg Procedure

Yoav Benjamini and Yosef Hochberg published their landmark paper introducing the False Discovery Rate and a simple step-up procedure (BH procedure) that controls FDR at a user-specified level q under independence of test statistics. This paper fundamentally changed how large-scale inference is conducted.

2001 • Storey's q-value and Positive FDR

John Storey introduced the positive false discovery rate (pFDR) and the q-value framework, offering a Bayesian interpretation and adaptive estimation of the proportion of true nulls π₀, which further boosted power in practice.

2002–present • Extensions and Modern Developments

Benjamini and Yekutieli extended FDR control to arbitrary dependency structures. Efron developed empirical Bayes approaches to large-scale inference. FDR methodology now permeates genomics, neuroscience, finance, and machine learning.

The central question that FDR addresses is deceptively simple: Among all the hypotheses we declare significant, what proportion are actually false positives? By shifting focus from avoiding any single error to controlling the expected fraction of errors among discoveries, Benjamini and Hochberg opened the door to powerful, scalable inference in the era of big data.

SECTION 2

Core Principles & Definitions

To understand FDR, one must first situate it within the broader taxonomy of error rates in multiple testing. When we conduct m simultaneous hypothesis tests, we can organize the outcomes into a two-by-two table of counts: the number of true nulls that we correctly retain (U), the number of true nulls we incorrectly reject (V, false discoveries), the number of true alternatives we fail to reject (T), and the number of true alternatives we correctly reject (S). The total number of rejections is R = V + S. The FDR is defined as the expected value of the ratio V/R, with the convention that V/R = 0 when R = 0.

FDR vs. FWER

The FWER is P(V ≥ 1) — the probability of at least one false positive. The FDR is E[V/R] — the expected proportion of false positives among all rejections. FDR is always ≤ FWER, making it a less stringent but more powerful criterion.

The Role of π₀

The proportion of true null hypotheses, denoted π₀ = m₀/m, critically influences FDR. When most hypotheses are truly null (π₀ close to 1), false discoveries are more likely. Adaptive methods estimate π₀ to sharpen FDR control and increase power.

Step-Up Procedure

The BH procedure orders all m p-values from smallest to largest and compares each p_(i) to the threshold (i/m) × q. It rejects all hypotheses up to and including the largest i for which p_(i) ≤ (i/m) × q.

Independence Assumption

The original BH procedure guarantees FDR ≤ q when test statistics are independent or satisfy a condition called positive regression dependency on subsets (PRDS). Under arbitrary dependence, the Benjamini–Yekutieli (BY) correction replaces q with q / Σ(1/i).

q-Values

A q-value is the minimum FDR level at which a given hypothesis would be called significant. It is the FDR analogue of the p-value: while a p-value measures evidence against a single null, a q-value measures the cost — in false discoveries — of including that hypothesis among the rejections.

✦

KEY TAKEAWAY

Think of FDR control like quality control on a factory line. If you inspect items and flag some as defective, FWER is like demanding that you never wrongly flag a good item — which means you barely inspect anything. FDR is like saying: among all items you flag, at most 5% should be wrongly flagged. This lets you inspect aggressively while keeping the defect rate in your flagged pile under control. In research, the 'flagged items' are your discoveries, and FDR ensures the proportion of false ones stays at an acceptable level.

SECTION 3

Visual Explanation

The BH Procedure on Ordered p-Values

The following diagram illustrates the geometric intuition behind the Benjamini–Hochberg procedure. When we sort our m p-values from smallest to largest and plot them against their rank, the BH threshold line y = (i/m) × q forms a straight line from the origin to the point (m, q). We scan from right to left (from the largest rank to the smallest) and find the largest index k where the ordered p-value p_(k) falls on or below the BH line. All hypotheses with rank ≤ k are then rejected.

The pink dots represent the five smallest p-values, all of which lie below the cyan BH threshold line at their respective ranks. The largest rank k for which p_(k) ≤ (k/m) × q is k = 5, so all hypotheses with ranks 1 through 5 are rejected. The purple dots above the line are retained.

Notice that the BH procedure does not simply compare each p-value to a fixed threshold; rather, the threshold adapts to the rank of the p-value. This is what makes it a step-up procedure: p-values with smaller ranks (stronger evidence) are held to a stricter threshold, while those at larger ranks have a more lenient cutoff. The geometric elegance is that we are looking for the rightmost point where the ordered p-values still lie below the linear BH line. Everything to the left of that crossing is declared a discovery.

SECTION 4

Mathematical Framework

Formal Definition of FDR

Let us test m null hypotheses H₁, H₂, …, H_m simultaneously. Denote by m₀ the (unknown) number of true null hypotheses. After applying a testing procedure, let V denote the number of false rejections (Type I errors) and R the total number of rejections. We define the false discovery proportion (FDP) as the random variable Q = V/R when R > 0, and Q = 0 when R = 0.

FALSE DISCOVERY RATE

FDR = E[Q] = E[V / max(R, 1)]

V = number of false discoveries (true nulls incorrectly rejected); R = total number of rejections; Q = false discovery proportion. The max(R, 1) convention avoids division by zero.

The BH Procedure

Given ordered p-values p₍₁₎ ≤ p₍₂₎ ≤ … ≤ p_(m) and a desired FDR level q ∈ (0, 1), the Benjamini–Hochberg procedure determines a data-driven cutoff as follows.

BH CRITICAL VALUE

k = max{ i : p₍ᵢ₎ ≤ (i / m) × q }

Reject all H₍₁₎, H₍₂₎, …, H_(k). If no such i exists, reject nothing.

Proof Sketch: FDR ≤ q Under Independence

Benjamini and Hochberg's original proof proceeds by conditioning on the p-values corresponding to the true alternative hypotheses. Under the assumption that the m₀ true null p-values are independent and uniformly distributed on [0, 1], one can show that each true null hypothesis H_i contributes at most q/m to the FDR. Summing over the m₀ true nulls gives FDR ≤ m₀ × (q/m) = (m₀/m) × q ≤ q, since m₀ ≤ m. This shows that the BH procedure is actually conservative when π₀ = m₀/m < 1, which motivates adaptive procedures that estimate π₀.

FDR BOUND UNDER INDEPENDENCE

FDR = (m₀ / m) × q = π₀ × q ≤ q

π₀ = m₀/m is the proportion of true nulls. Since π₀ ≤ 1, the BH procedure guarantees FDR ≤ q. When π₀ < 1, the actual FDR is strictly less than q, leaving room for adaptive improvement.

Storey's q-Value

Q-VALUE DEFINITION

q(p) = min{ FDR level q* : hypothesis with p-value p is rejected at level q* }

Storey's q-value provides a per-hypothesis measure of significance analogous to p-values but in the FDR framework. A hypothesis with q-value 0.05 means that including it among the rejections incurs an FDR of at most 5%.

SECTION 5

FDR Procedures & Classification

Taxonomy of FDR-Controlling Methods

Since the original BH procedure, a rich family of FDR-controlling methods has emerged, each tailored to different assumptions about the dependence structure among test statistics, the proportion of true nulls, and computational constraints. Understanding the relationships among these methods is essential for selecting the right tool in practice. The diagram below maps the major procedures by their dependency assumptions and whether they incorporate adaptive estimation of π₀.

The left column shows non-adaptive methods that use the nominal FDR level q directly. The right column shows adaptive methods that estimate π₀ to gain power. Rows correspond to dependency assumptions, from independence/PRDS at the top to arbitrary dependence in the middle. The knockoff filter at the bottom represents a newer model-free approach that constructs synthetic null variables.

Method	Dependency Assumption	Adaptive?	Key Feature
BH (1995)	Independence or PRDS	No	Simple, widely used, guarantees FDR ≤ π₀q
BY (2001)	Arbitrary	No	Valid under any dependence; threshold divided by Σ(1/j)
Storey q-value	Independence or weak	Yes (π̂₀)	Per-hypothesis significance measure; gains power by estimating π₀
Efron Local FDR	Mixture model	Yes (f₀, f₁)	Bayesian posterior probability; estimates full null/alternative distributions
Knockoff Filter	Model-free	N/A	Constructs synthetic nulls; finite-sample FDR guarantee

Comparison of Major FDR-Controlling Procedures

SECTION 6

Worked Example: Applying the BH Procedure

Suppose a genomics researcher tests m = 10 genes for differential expression between a treatment and control group. The researcher wishes to control the FDR at q = 0.10. The raw p-values from the 10 tests are: 0.005, 0.042, 0.012, 0.310, 0.001, 0.085, 0.720, 0.460, 0.030, 0.150.

BH Procedure with m = 10 Tests and q = 0.10

1
Step 1 — Order the p-values

Sort the p-values from smallest to largest: p₍₁₎ = 0.001, p₍₂₎ = 0.005, p₍₃₎ = 0.012, p₍₄₎ = 0.030, p₍₅₎ = 0.042, p₍₆₎ = 0.085, p₍₇₎ = 0.150, p₍₈₎ = 0.310, p₍₉₎ = 0.460, p₍₁₀₎ = 0.720.

Ordered p-values established.
2
Step 2 — Compute BH critical values

For each rank i = 1, 2, …, 10, compute the BH threshold (i/m) × q = (i/10) × 0.10 = i/100. This gives thresholds: 0.010, 0.020, 0.030, 0.040, 0.050, 0.060, 0.070, 0.080, 0.090, 0.100.

BH thresholds: 0.010, 0.020, 0.030, 0.040, 0.050, 0.060, 0.070, 0.080, 0.090, 0.100.
3
Step 3 — Compare each p-value to its threshold

Rank 1: 0.001 ≤ 0.010 ✓. Rank 2: 0.005 ≤ 0.020 ✓. Rank 3: 0.012 ≤ 0.030 ✓. Rank 4: 0.030 ≤ 0.040 ✓. Rank 5: 0.042 ≤ 0.050 ✓. Rank 6: 0.085 > 0.060 ✗. Rank 7: 0.150 > 0.070 ✗. Ranks 8–10 also fail.

p-values at ranks 1 through 5 satisfy p_(i) ≤ (i/m) × q.
4
Step 4 — Identify the cutoff k

The largest rank i for which p_(i) ≤ (i/m) × q is k = 5. This is because rank 5 is the last one satisfying the inequality; rank 6 is the first to exceed its threshold.

k = 5
5
Step 5 — Reject and interpret

Reject all hypotheses corresponding to ranks 1 through 5 (i.e., genes with p-values 0.001, 0.005, 0.012, 0.030, and 0.042). We declare 5 genes as differentially expressed. The BH procedure guarantees that the expected proportion of false discoveries among these 5 rejections is at most 10%. Since we do not know the true π₀, the actual FDR may be less.

5 genes rejected, FDR controlled at q = 0.10.

💡

Comparison to Bonferroni

Under the Bonferroni correction, the adjusted significance level would be α/m = 0.10/10 = 0.01. Only ranks 1 and 2 (p-values 0.001 and 0.005) would be rejected — missing three additional discoveries. The BH procedure's advantage is clear: by tolerating a controlled fraction of false discoveries, it recovers substantially more power.

SECTION 7

Strengths, Limitations & Comparisons

FDR vs. FWER: A Detailed Comparison

Criterion	FDR Control (e.g., BH)	FWER Control (e.g., Bonferroni)
Error metric	E[V/R]: expected fraction of false rejections	P(V ≥ 1): probability of any false rejection
Stringency	Less stringent; tolerates some false positives proportionally	Very stringent; aims to eliminate all false positives
Power	Higher power, especially as m grows	Power decreases rapidly with m
Ideal setting	Exploratory analysis, large-scale screening, genomics	Confirmatory studies, safety-critical decisions, clinical trials
Dependency handling	BH valid under independence/PRDS; BY under arbitrary	Bonferroni valid under arbitrary dependence; Holm improves it
Interpretation	Among discoveries, at most q fraction are expected false	Probability of at least one false discovery ≤ α

FDR vs. FWER — Side-by-Side Comparison

Strengths and Limitations of FDR

Strength — Scalability: FDR methods scale gracefully to millions of tests, as seen in genome-wide association studies (GWAS), where Bonferroni thresholds become absurdly conservative.
Strength — Interpretability: The statement 'among our reported discoveries, at most 5% are expected to be false' is intuitive and directly relevant to scientific practice.
Strength — Adaptivity: Adaptive FDR methods (Storey, two-stage BH) exploit the data to estimate π₀, gaining further power without inflating error.
Limitation — Expected value: FDR is an expectation, not a guarantee for any single experiment. The realized FDP can exceed q, especially when R is small.
Limitation — Dependence sensitivity: The BH procedure can lose FDR control under strong negative dependence among test statistics, necessitating the more conservative BY correction.
Limitation — Not suitable for all contexts: In safety-critical settings (e.g., drug approval), even one false positive may be unacceptable, making FWER control more appropriate than FDR.

✦

KEY TAKEAWAY

Choosing between FDR and FWER is analogous to choosing between a screening test and a diagnostic test in medicine. A screening test (FDR) casts a wide net — it aims to catch most true cases while accepting that some flagged results will turn out to be false alarms. A diagnostic test (FWER) demands high certainty for each positive result. In exploratory research where the goal is to identify promising leads for follow-up, FDR is the natural choice; in confirmatory settings where each declared positive carries immediate consequences, FWER is appropriate.

SECTION 8

Connection to Advanced Theory

Local FDR and the Bayesian Connection

The local false discovery rate (local FDR or fdr, lowercase) provides a finer-grained measure than the tail-area FDR. Introduced by Bradley Efron, the local FDR measures the posterior probability that a specific hypothesis is null given its observed test statistic z: fdr(z) = π₀ f₀(z) / f(z), where f₀ is the null density, f is the mixture density, and π₀ is the prior probability of the null. This formulation connects FDR to empirical Bayes methodology: one estimates the mixture model from the data to obtain per-hypothesis posterior probabilities. The tail-area FDR can then be recovered by averaging the local FDR over the rejection region, bridging frequentist FDR control with Bayesian interpretations.

Concept	Tail-Area FDR (Benjamini–Hochberg)	Local FDR (Efron)
Scope	Average over all rejected hypotheses	Specific to each individual hypothesis
Framework	Frequentist (controls an expected proportion)	Empirical Bayes (posterior probability)
Input	p-values only	Full test statistics and estimated densities
Output	Binary reject/retain decision set	Continuous probability per hypothesis
Relationship	FDR(t) = E[fdr(Z) \| Z ∈ rejection region]	fdr(z) is the integrand whose average gives FDR

Tail-Area FDR vs. Local FDR

Beyond local FDR, several important extensions push the boundaries of FDR methodology. The model-X knockoffs framework (Candès and Barber, 2015) achieves finite-sample FDR control for variable selection in regression by constructing synthetic 'knockoff' covariates that mimic the correlation structure of the originals but are conditionally independent of the response. This approach sidesteps the need for p-values entirely. Meanwhile, online FDR control (Javanmard and Montanari, Foster and Stine) extends FDR to sequential settings where hypotheses arrive over time, maintaining FDR guarantees as a running average — essential for applications in A/B testing and real-time signal detection.

ℹ

Looking Ahead

Modern machine learning increasingly relies on FDR-type reasoning. Feature selection with FDR guarantees, conformal prediction with multiple-testing corrections, and selective inference after model selection all draw on the principles established by Benjamini and Hochberg. As data science problems grow in dimension and complexity, the intellectual lineage from the 1995 BH paper continues to generate new methodology.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain, in your own words, why FDR control is preferable to FWER control when conducting a genome-wide association study with 500,000 SNPs. What specific problem does FWER control create in this context, and how does FDR address it?

Show answer

With 500,000 simultaneous tests, a Bonferroni correction at α = 0.05 would require each individual p-value to fall below 0.05/500,000 = 1 × 10⁻⁷. This is extremely stringent and makes it nearly impossible to detect SNPs with small-to-moderate effect sizes, even if they are genuinely associated with the phenotype. The power of the study plummets. FDR control, by contrast, accepts that among the discoveries we declare significant, a controlled fraction (say 5%) may be false. Because the BH threshold at rank i is (i/500,000) × 0.05, the procedure adapts to the evidence in the data: if many SNPs show strong signals, more will pass the threshold. This preserves substantially more power to detect true associations while keeping the expected proportion of false leads manageable for downstream follow-up.

Question 2: BASIC CALCULATION

A researcher conducts m = 8 hypothesis tests and obtains the following p-values: 0.003, 0.009, 0.025, 0.048, 0.062, 0.110, 0.350, 0.880. Apply the BH procedure at q = 0.05 and determine which hypotheses are rejected.

Show answer

First, the p-values are already ordered. Compute BH thresholds: rank 1: 1/8 × 0.05 = 0.00625; rank 2: 2/8 × 0.05 = 0.0125; rank 3: 3/8 × 0.05 = 0.01875; rank 4: 4/8 × 0.05 = 0.025; rank 5: 5/8 × 0.05 = 0.03125; rank 6: 6/8 × 0.05 = 0.0375; rank 7: 7/8 × 0.05 = 0.04375; rank 8: 8/8 × 0.05 = 0.05. Now compare: rank 1 (0.003 ≤ 0.00625 ✓), rank 2 (0.009 ≤ 0.0125 ✓), rank 3 (0.025 > 0.01875 ✗), rank 4 (0.048 > 0.025 ✗), and so on. The largest i satisfying the condition is k = 2. Therefore, reject the hypotheses at ranks 1 and 2 (p-values 0.003 and 0.009). Two hypotheses are declared significant at FDR ≤ 0.05.

Question 3: INTERMEDIATE

Suppose m = 1000 hypotheses are tested, and 900 are truly null (π₀ = 0.9). The BH procedure is applied at q = 0.05. Using the theoretical result that FDR = π₀ × q under independence, (a) compute the actual expected FDR, and (b) explain how an adaptive procedure (e.g., Storey's method) could use this information to gain power.

Show answer

(a) Under independence, the BH procedure yields FDR = π₀ × q = 0.9 × 0.05 = 0.045. This means the procedure is slightly conservative — the actual FDR is 4.5%, not the full 5% the user requested. (b) An adaptive procedure estimates π̂₀ from the data (for instance, Storey's estimator counts the proportion of p-values above a tuning parameter λ and scales appropriately). If the estimate is π̂₀ ≈ 0.9, the adaptive procedure replaces q with q/π̂₀ = 0.05/0.9 ≈ 0.0556 as the effective FDR level in the BH step-up comparison. This raises each BH threshold, allowing more hypotheses to pass, thus increasing the number of true discoveries while still controlling the FDR at the target level of 0.05. The power gain can be substantial when π₀ is well below 1.

Question 4: APPLIED

A neuroscience lab performs an fMRI study and tests 50,000 voxels for activation. Due to spatial correlation in brain imaging data, the test statistics exhibit positive dependence. The researcher uses the BH procedure at q = 0.05 and identifies 1,200 significant voxels. (a) Is the BH procedure valid here? Justify your answer. (b) If the dependence were arbitrary (not PRDS), which procedure should be used instead, and approximately how many voxels would you expect to survive correction? (Assume the harmonic sum Σ₁⁵⁰⁰⁰⁰(1/j) ≈ 11.4.)

Show answer

(a) Yes, the BH procedure is valid here. Brain voxel test statistics under the null typically satisfy the PRDS (positive regression dependency on subsets) condition because spatial correlation in fMRI data induces positive dependence among nearby voxels. Benjamini and Yekutieli (2001) proved that the BH procedure controls FDR under PRDS. (b) Under arbitrary dependence, one must use the BY procedure, which replaces the BH threshold (i/m) × q with (i/m) × q / Σ(1/j). With the harmonic sum ≈ 11.4, the BY threshold is approximately 11.4 times stricter than BH. If BH yielded 1,200 rejections, BY would yield far fewer — a rough estimate is on the order of 1,200 / 11.4 ≈ 105 voxels, though the exact number depends on the p-value distribution. This dramatic loss of power illustrates why establishing PRDS (and thus using the standard BH procedure) is important in imaging applications.

Question 5: CRITICAL THINKING

The FDR is defined as E[V/max(R,1)], an expected value. A critic argues that this is inadequate because in a single experiment, the realized false discovery proportion V/R could be much higher than q. (a) Is this criticism valid? (b) Propose and briefly describe a criterion that addresses this concern. (c) Discuss the trade-offs of using your proposed criterion versus standard FDR.

Show answer

(a) The criticism has merit. Because FDR is an expected value, it guarantees that the false discovery proportion averages out to ≤ q across many experiments, but in any single realization, the actual FDP = V/R can exceed q, particularly when R is small. For instance, if R = 1 and that single rejection is a false positive, FDP = 1, even though E[FDP] ≤ q. (b) A natural alternative is the false discovery exceedance (FDX), which controls P(V/R > γ) ≤ α, ensuring that the probability of the FDP exceeding a threshold γ is at most α. This is sometimes called 'FDP tail probability control' or is related to simultaneous control of the FDP. (c) FDX provides a probabilistic guarantee for each experiment rather than an on-average guarantee, which is more reassuring when individual experiments carry high stakes. However, FDX control requires stronger assumptions or yields fewer rejections than BH at the same nominal level, since it must guard against tail events. In large-scale screening where individual experiments are cheap and results will be validated in follow-up studies, standard FDR is more practical and powerful. FDX is preferable when each declared discovery has immediate costly consequences and the experimenter cannot afford occasional large FDP realizations.

SUMMARY

Summary & Review

Summary

The False Discovery Rate (FDR) is defined as E[V/max(R, 1)], the expected proportion of false positives among all rejected hypotheses. Introduced by Benjamini and Hochberg in 1995, FDR provides a less stringent alternative to the familywise error rate (FWER), enabling substantially greater statistical power when the number of simultaneous tests is large. The BH procedure ranks p-values and compares each to the threshold (i/m) × q in a step-up fashion, rejecting all hypotheses up to and including the largest rank that satisfies the inequality. Under independence or positive regression dependency (PRDS), this guarantees FDR ≤ π₀ × q ≤ q.

Key extensions include Storey's q-value, which adaptively estimates the proportion of true nulls π₀ to sharpen power; the Benjamini–Yekutieli (BY) procedure for arbitrary dependence; and Efron's local FDR, which connects FDR to empirical Bayes through the posterior probability of each hypothesis being null. FDR is the standard error criterion in genomics, neuroimaging, and other large-scale testing settings. When choosing between FDR and FWER, the guiding principle is context: FDR excels in exploratory, high-dimensional screening, while FWER remains appropriate for confirmatory and safety-critical analyses.