False Discovery Rate (FDR) - Statistics Graduate Level

1. Which statement best describes the FDR in this special case?
   Answer: It equals $P(R > 0) = q$, because every rejection is a false rejection when all nulls are true When all $m$ null hypotheses are true, every rejection is necessarily a false discovery. This is the "global null" scenario, and it's a classic setting for understanding what the FDR actually measures. Recall that FDR is defined as $\mathbb{E}\left[\frac{V}{\max(R,1)}\right]$, where $V$ is the number of false rejections and $R$ is the total rejections. When all nulls are true, $V = R$ always, so the ratio $V/\max(R,1)$ equals 1 whenever $R > 0$ and 0 otherwise. Therefore, $\text{FDR} = 1 \cdot P(R > 0) + 0 \cdot P(R = 0) = P(R > 0)$. Under BH at level $q$ with independent uniform p-values, this probability equals exactly $q$, confirming A as correct. The BH procedure is designed so that even in this worst-case scenario, FDR $= q$, not something larger. Answer B is wrong because dividing expected rejections by $m$ gives you the expected proportion of all hypotheses rejected — this is not the FDR formula, which conditions on at least one rejection occurring. Answer C confuses the event "each rejection is false" (which is always true here, but doesn't mean all hypotheses get rejected) with the probability that every hypothesis is rejected — a much stronger and unrelated event. Answer D is wrong because the FDR definition elegantly handles the $R = 0$ case by defining the ratio as 0, so the unconditional expectation is perfectly well-defined. A key study tip: always return to the formal definition $\mathbb{E}[V/\max(R,1)]$ when reasoning about FDR — many traps arise from confusing it with related but distinct quantities like FWER or the expected rejection count.
2. Which relationship between their rejection sets must hold, regardless of the observed p-values?
   Answer: Every Bonferroni rejection is also a BH rejection, although BH may reject additional hypotheses When comparing multiple testing procedures, the key is understanding what each procedure's rejection threshold actually is — not just the error concept it controls. Bonferroni rejects hypothesis $i$ when $p_{(i)} \leq q/m$. The BH procedure sorts p-values as $p_{(1)} \leq p_{(2)} \leq \cdots \leq p_{(m)}$ and rejects all hypotheses up to index $k^* = \max\{i : p_{(i)} \leq iq/m\}$. Notice that the BH threshold for the smallest p-value is $q/m$ — identical to Bonferroni's single cutoff. For any larger p-value at rank $i > 1$, BH uses threshold $iq/m > q/m$, which is more lenient. This means any hypothesis Bonferroni rejects (with $p_{(i)} \leq q/m$) automatically satisfies $p_{(i)} \leq iq/m$, so BH rejects it too — but BH can additionally reject hypotheses that Bonferroni misses. That confirms D is correct. A has the logical relationship exactly backwards. FDR control is weaker than FWER control — BH allows some false discoveries, while Bonferroni permits none in expectation. A weaker criterion rejects more, not fewer, hypotheses. B is tempting but confuses a special case for a general rule. Even when no p-value falls at or below $q/m$, the containment relationship in D still holds vacuously (Bonferroni rejects nothing, and the empty set is a subset of BH's rejections). C is wrong because the procedures are mathematically related through the shared threshold $q/m$ at rank 1, creating the guaranteed subset relationship. Study tip: Remember that BH's rank-1 threshold equals Bonferroni's cutoff exactly — this single fact implies Bonferroni $\subseteq$ BH always.
3. What is the FDR when the two families' rejection results are pooled and treated as one reported collection?
   Answer: $2q - q^2$, because every pooled rejection is a false rejection, and the probability of at least one rejection equals $1-(1-q)^2$ When every hypothesis in a study is null, every rejection is a false discovery — so the FDR equals the probability of making at least one rejection. This is the key insight that unlocks this problem. Here, you have two independent families, each containing one null hypothesis, each tested at level $q$. Because all hypotheses are null, every possible rejection is a false discovery. The pooled FDR is therefore the probability that at least one of the two tests rejects. Using the complement rule for independent events: $\text{FDR} = 1 - (1-q)(1-q) = 2q - q^2$ This confirms D is correct. A is wrong because controlling FDR within each family does not automatically control the combined FDR at the same level when you pool results across families. Within-family control is a local guarantee; pooling creates a new, larger collection where joint error behavior must be re-examined. B is wrong because a pooled false discovery does not require both null hypotheses to be rejected — it requires at least one rejection. The value $q^2$ is the probability of rejecting both, which is an intersection event, not the union. C is wrong in a subtle but important way: $2q$ overcounts by ignoring the overlap (the event where both are rejected simultaneously). By inclusion-exclusion, the correct union probability is $2q - q^2$, not $2q$. Simple addition only works when the events are mutually exclusive. Study tip: Whenever you see "pooled FDR" with all-null families, immediately think "probability of at least one false rejection" and apply the complement rule — this reframes an FDR question into a clean probability calculation.
4. What is the BH-adjusted p-value for the hypothesis at rank $2$?
   Answer: $0.032$, because the minimum over ranks $2$ through $6$ occurs at rank $3$ When working with Benjamini-Hochberg (BH) adjusted p-values, the key formula to remember is that the adjusted p-value for rank $i$ is not simply $p_{(i)} \cdot m/i$. Instead, it requires a backward minimum sweep: $\tilde{p}_{(i)} = \min_{j \geq i}\left\{\frac{m \cdot p_{(j)}}{j}\right\}$, truncated at 1. This enforces monotonicity — adjusted p-values must be non-decreasing as rank increases. For rank 2, you compute $m \cdot p_{(j)}/j$ for all $j \geq 2$ and take the minimum. With $m = 6$: - $j=2$: $6(0.014)/2 = 0.042$ - $j=3$: $6(0.016)/3 = 0.032$ - $j=4$: $6(0.041)/4 = 0.0615$ - $j=5$: $6(0.13)/5 = 0.156$ - $j=6$: $6(0.40)/6 = 0.40$ The minimum over these values is $0.032$, occurring at rank 3. So the BH-adjusted p-value for rank 2 is $\mathbf{0.032}$, confirming C. A is wrong because BH adjustment always alters the p-value — simply preserving the original value ignores the multiplicity correction entirely. B makes the common mistake of stopping at rank 2 without sweeping forward; it computes $0.042$ but misses that rank 3 yields a smaller adjusted value. D multiplies $p_{(3)}$ by $m = 6$ rather than $m/j = 6/3$, confusing the total number of tests with the ratio used in the BH formula. A useful strategy: always compute the raw BH values $m \cdot p_{(j)}/j$ for all ranks from $i$ to $m$, then take the minimum. Stopping early at rank $i$ is the most common trap on questions like this.
5. What are this procedure's FDR and positive FDR, respectively?
   Answer: FDR is $0.01$, and positive FDR is $0.10$ When you see a question about FDR and positive FDR, the key distinction is whether you condition on any rejections occurring. Both metrics measure the expected proportion of false rejections among all rejections, but they differ in how they handle the case of zero rejections. FDR is defined as $E\left[\frac{V}{R}\right]$, where $V$ is the number of false rejections and $R$ is the total rejections, with the ratio defined as 0 when $R = 0$. Positive FDR (pFDR) conditions on at least one rejection occurring: $E\left[\frac{V}{R} \,\middle|\, R > 0\right]$. In this problem, with probability 0.90 there are no rejections (ratio = 0), and with probability 0.10 there are 10 rejections, exactly 1 false. So the false discovery proportion when rejections occur is $\frac{1}{10} = 0.10$. For FDR: $E\left[\frac{V}{R}\right] = (0.90)(0) + (0.10)(0.10) = 0.01$. For pFDR: you condition on the event that rejections occur (probability 0.10), so $E\left[\frac{V}{R} \,\middle|\, R > 0\right] = 0.10$. The 90% no-rejection scenarios are excluded entirely. This confirms answer A: FDR = 0.01, pFDR = 0.10. Answer B swaps the two values — a direct result of confusing which metric conditions on $R > 0$. Answer C incorrectly sets pFDR = 1.00, perhaps confusing "one false rejection" with "all rejections are false." Answer D sets both equal to 0.01, ignoring the conditioning in pFDR entirely. Your study tip: remember that pFDR is always at least as large as FDR, because conditioning on $R > 0$ removes the zero-ratio outcomes that pull the expectation down.
6. How many hypotheses does the second BH analysis reject?
   Answer: One, because only $0.01$ remains below its new rank-specific BH critical value Whenever you see a BH question involving changes to the family of hypotheses, your instinct should be to recompute the BH critical values from scratch rather than assume prior rejections are preserved. Recall the BH procedure: sort all $m$ p-values as $p_{(1)} \leq \cdots \leq p_{(m)}$, then reject all hypotheses up to rank $k^* = \max\{k : p_{(k)} \leq kq/m\}$. The critical value for rank $k$ is $kq/m$, so adding hypotheses changes $m$ and rescales every threshold. In the first analysis, $m = 4$ and $q = 0.10$. The thresholds are $0.025, 0.050, 0.075, 0.100$. Comparing: $p_{(1)} = 0.01 \leq 0.025$ ✓, $p_{(2)} = 0.04 \leq 0.050$ ✓, $p_{(3)} = 0.08 > 0.075$ ✗. BH rejects two hypotheses. In the second analysis, $m = 8$ and $q = 0.10$. The four new p-values are all $1$, so the ordered set is $0.01, 0.04, 0.08, 0.50, 1, 1, 1, 1$. Now the threshold at rank $k$ is $k(0.10)/8 = k/80$. Checking: $p_{(1)} = 0.01 \leq 1/80 = 0.0125$ ✓, $p_{(2)} = 0.04 \leq 2/80 = 0.025$ ✗. Only one rejection survives, confirming B is correct. A is wrong because BH rank ratios don't simply halve; each threshold must be computed individually. C is the most tempting trap — it assumes adding null p-values is harmless, but those extra hypotheses inflate $m$ and shrink every threshold, costing you a rejection. D is wrong because more ranks help only if corresponding p-values are small enough to clear their new, lower thresholds. Your key takeaway: adding hypotheses always changes $m$, which uniformly lowers BH thresholds — never assume prior rejections automatically persist after the family grows.
7. Which interpretation of the reported analysis is most accurate?
   Answer: Across repetitions, the expected false-discovery proportion is at most $0.10$, with the proportion defined as zero when there are no rejections When interpreting multiple testing procedures like Benjamini-Hochberg (BH), the critical skill is distinguishing between what the procedure actually controls versus what students commonly wish it controlled. BH controls the False Discovery Rate (FDR), defined as the expected value of the false discovery proportion (FDP), where FDP = (number of false rejections)/(total rejections) when rejections > 0, and 0 otherwise. Choice B is correct because it states precisely this guarantee: across repeated applications of the procedure under the same conditions, the expected FDP is at most $q = 0.10$. The phrase "with the proportion defined as zero when there are no rejections" correctly captures the standard definition, which avoids a 0/0 issue. This is a statement about a long-run average over hypothetical repetitions, not about this particular experiment alone. Choice A confuses FDR with a conditional expectation given the observed rejection set. The BH guarantee is marginal (unconditional), not conditional on the specific 20 discoveries observed. The posterior expected number of false discoveries given exactly these 20 rejections is a different, harder quantity that BH does not directly control. Choice C describes something resembling a Bayesian local false discovery rate (lfdr) — the posterior probability that an individual hypothesis is null. BH controls nothing at the individual-hypothesis level in this sense; it operates on the procedure's aggregate behavior. Choice D describes control of the familywise error rate (FWER) — the probability of any false rejection — which is a stricter criterion controlled by procedures like Bonferroni, not BH. Your study tip: always anchor your interpretation to the definition of what's being controlled. FDR = expected proportion (long-run average), not a per-experiment conditional probability or an individual-test probability.
8. Which conclusion about FDR control is most appropriate?
   Answer: The count-based filter can preserve BH validity under the stated null independence, whereas filtering by small p-values generally invalidates the usual guarantee When evaluating FDR control after pre-filtering, the critical question is whether the filter is independent of the test statistics under the null. The Benjamini-Hochberg (BH) procedure's FDR guarantee relies on certain properties of the null p-value distribution — specifically, that null p-values are stochastically super-uniform. Any filter that distorts this distribution among the retained nulls can break the guarantee. A count-based filter retains genes based on total read depth, a quantity determined before observing group-specific signals. Under the null hypothesis for a given gene, its read count is independent of its p-value, so the conditional null p-value distribution among retained genes remains uniform. BH applied to these retained genes therefore preserves its FDR control. This is the reasoning behind answer A, which is correct. Now consider filtering by smallest unadjusted p-values — say, keeping only genes with $p < 0.5$. This truncates the null p-value distribution, removing the large p-values that would otherwise anchor the BH procedure's step-up threshold. The remaining null p-values are no longer uniform; they are concentrated near zero, making the procedure anti-conservative and invalidating the FDR guarantee. B is wrong because BH's validity does not follow automatically from simply plugging in the post-filter count $m$. The shape of the null p-value distribution matters, not just the number of tests. C is wrong because enriching for nonnull genes is a power benefit, not a validity argument — the problem is what happens to the null p-values, not the nonnulls. D is wrong because filtering itself is not disqualifying; only filters that depend on the test statistics under the null are problematic. Key tip: Always ask whether a filter is independent of the test statistic under the null. If yes, BH validity survives. If the filter selects on p-values directly, the null distribution is distorted and the guarantee fails.
9. Which set of hypotheses is rejected?
   Answer: The hypotheses corresponding to the first five ordered p-values Whenever you see a Benjamini–Hochberg (BH) procedure question, your job is to find the largest ordered p-value that still falls below its BH threshold, then reject all hypotheses up to and including that one. With $m = 8$ hypotheses and target FDR level $q = 0.05$, the threshold for the $i$-th ordered p-value is $\frac{i}{m} \cdot q = \frac{i}{8}(0.05)$. Compare each ordered p-value $p_{(i)}$ to its threshold:

   $i$	$p_{(i)}$	Threshold $\frac{i}{8}(0.05)$	$p_{(i)} \leq$ threshold?
   1	0.004	0.00625	✓
   2	0.009	0.01250	✓
   3	0.018	0.01875	✗
   4	0.021	0.02500	✓
   5	0.031	0.03125	✓
   6	0.041	0.03750	✗

   The largest $i$ for which $p_{(i)} \leq \frac{i}{8}(0.05)$ is $i = 5$, so you reject hypotheses 1 through 5, confirming answer C. Answer A is wrong because it stops at $i=2$, ignoring that $p_{(4)}$ and $p_{(5)}$ also satisfy their thresholds. Answer B rejects only the first three, but $p_{(3)} = 0.018$ actually exceeds its threshold of 0.01875 — a common trap. Answer D rejects six hypotheses, but $p_{(6)} = 0.041 > 0.03750$, so that boundary is crossed in the wrong direction. The key study tip: BH is a step-up procedure — scan upward from the largest p-value and find the last crossing, then reject everything below it, even if some intermediate p-values individually exceeded their thresholds.
10. Which procedure has the most defensible general FDR guarantee at nominal level $q$?
    Answer: Apply the Benjamini–Yekutieli procedure with critical values $iq/(mH_m)$, where $H_m = \sum_{j=1}^{m} 1/j$, because it controls FDR under arbitrary dependence When you encounter FDR control questions, your first instinct should be to ask: what dependence structure am I assuming? The answer determines which procedure is valid. The Benjamini–Hochberg (BH) procedure uses critical values $iq/m$ and controls FDR at level $q$ — but only under independence or a specific positive-dependence condition called PRDS (positive regression dependence on a subset). When dependence is arbitrary and unknown, BH's FDR guarantee evaporates. Benjamini and Yekutieli (2001) proved that under arbitrary dependence, FDR control is restored by applying the more conservative critical values $iq/(m H_m)$, where $H_m = \sum_{j=1}^{m} 1/j \approx \ln(m) + 0.577$. This BY procedure sacrifices power for universality — it controls FDR at level $q$ with no restrictions whatsoever on the joint distribution of p-values. That makes C the correct and most defensible choice. A is wrong because valid marginal p-values do not make dependence irrelevant to FDR control. Dependence among test statistics can cause BH to exceed the nominal FDR level under general dependence structures. B commits the same error more subtly — claiming dependence affects only power but not FDR inflation. This is false; counterexamples exist where BH's FDR exceeds $q$ under negative or complex dependence. D is wrong because Bonferroni controls FWER, not FDR. While FWER control implies FDR control (since FWER $\geq$ FDR), this inequality means Bonferroni is unnecessarily conservative as an FDR procedure and doesn't directly bound the expected false-discovery proportion at $q$. Your study tip: memorize the trio — BH works under independence/PRDS, BY works under arbitrary dependence, and always distinguish FWER from FDR procedures.

Start this quiz