Statistics Graduate Level Quiz: Family Wise Error Rate And Bonferroni
10 questions · exam conditions
0:00
Family Wise Error Rate And BonferroniQuestion 1 of 10

Four hypotheses are tested using valid p-values. Their prespecified marginal significance levels are, respectively, 0.0120.012, 0.0180.018, 0.0090.009, and 0.0110.011. No assumptions are made about dependence among the test statistics.

Which conclusion about family-wise error control is justified?

The procedure strongly controls the family-wise error rate at 0.050.05, although its actual error rate need not equal 0.050.05.
The procedure controls the family-wise error rate at 0.050.05 only under the global null, because partial null configurations require independence.
The procedure controls the family-wise error rate at 0.050.05 only if the four rejection events are mutually independent.
The procedure has family-wise error rate exactly 0.050.05 under every null configuration, regardless of dependence.
← Back to quizzes

Statistics Graduate Level Quiz

Statistics Graduate Level Quiz: Family Wise Error Rate And Bonferroni

Practice Family Wise Error Rate And Bonferroni in Statistics Graduate Level with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Family Wise Error Rate And Bonferroni, giving you a quick way to practice the rules, question types, and explanations that matter most for Statistics Graduate Level.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

Four hypotheses are tested using valid p-values. Their prespecified marginal significance levels are, respectively, 0.0120.012, 0.0180.018, 0.0090.009, and 0.0110.011. No assumptions are made about dependence among the test statistics.

Which conclusion about family-wise error control is justified?

  1. The procedure strongly controls the family-wise error rate at 0.050.05, although its actual error rate need not equal 0.050.05. (correct answer)
  2. The procedure controls the family-wise error rate at 0.050.05 only under the global null, because partial null configurations require independence.
  3. The procedure controls the family-wise error rate at 0.050.05 only if the four rejection events are mutually independent.
  4. The procedure has family-wise error rate exactly 0.050.05 under every null configuration, regardless of dependence.
Explanation: Whenever you see a question about multiple testing procedures, your first instinct should be to identify which type of error rate control is claimed and under what conditions it holds. The key concept here is the Bonferroni correction and its relationship to family-wise error rate (FWER) control. The Bonferroni inequality states that for any collection of events, P(at least one false rejection)iH0αiP(\text{at least one false rejection}) \leq \sum_{i \in H_0} \alpha_i, where the sum runs over the true null hypotheses. This inequality holds regardless of the dependence structure among test statistics — it follows purely from the union bound. Here, the marginal levels sum to 0.012+0.018+0.009+0.011=0.050.012 + 0.018 + 0.009 + 0.011 = 0.05, so no matter which subset of hypotheses is actually null, the probability of any false rejection is bounded by 0.050.05. This is strong control: the FWER bound holds under every partial null configuration, not just the global null. The actual FWER may be strictly less than 0.050.05 (it equals 0.050.05 only under perfect positive dependence in special cases), confirming A is correct. Choice B is wrong because it implies that strong control requires the global null or independence — but the Bonferroni bound needs neither. Choice C repeats the same independence fallacy; the union bound imposes no dependence assumptions. Choice D overclaims by asserting the FWER equals exactly 0.050.05 under every configuration, which is false — Bonferroni is conservative in general. Study tip: Memorize that Bonferroni provides strong FWER control under arbitrary dependence, but is typically conservative (actual FWER α\leq \alpha, not =α= \alpha). Questions often bait you into adding independence assumptions that simply aren't needed.

Question 2

A researcher computes ten mutually independent p-values corresponding to ten true null hypotheses. The researcher reports only the hypothesis having the smallest p-value and rejects that hypothesis if its p-value is at most 0.050.05, arguing that only one hypothesis was ultimately tested.

What is the actual family-wise error rate of this procedure?

  1. It is 0.0050.005 because the reported test implicitly receives a Bonferroni-adjusted level.
  2. It is 0.0500.050 because only one selected hypothesis can ultimately be rejected.
  3. It is approximately 0.4010.401 because selection uses the minimum of ten p-values. (correct answer)
  4. It is approximately 0.5000.500 because ten candidate hypotheses were screened at level 0.050.05.
Explanation: Whenever you see a question involving selecting the most extreme result from a set of tests, you're dealing with selective inference — a classic source of inflated Type I error that lurks behind "I only tested one hypothesis" reasoning. Here's the core insight: even though the researcher ultimately reports one p-value, the family-wise error rate (FWER) measures the probability that any false rejection occurs across the entire procedure. When ten independent p-values are each uniform on [0,1][0,1] under true null hypotheses, the probability that the minimum exceeds 0.050.05 is (10.05)10=0.95100.599(1 - 0.05)^{10} = 0.95^{10} \approx 0.599. Therefore, the probability that the minimum is at most 0.050.05 — meaning a rejection occurs — is 10.95100.4011 - 0.95^{10} \approx 0.401. That's the true FWER, confirming C. A is wrong because Bonferroni correction works in the opposite direction — it tightens the threshold to control error, not describe the uncorrected error rate. The actual FWER here is far above 0.0050.005. B is the core misconception the question is designed to expose. Reporting only one test doesn't undo the implicit search across ten. The selection process itself inflates error, regardless of how many results are ultimately announced. D inflates the intuition too far in the other direction. Screening ten tests at 0.050.05 does not yield a FWER of 0.5000.500; the correct calculation uses the complement of the joint survival probability, giving roughly 0.4010.401. Your takeaway: whenever results are selected based on extremity (smallest p-value, largest effect), treat the full set of candidates — not just the reported one — as the relevant family for error rate calculations.

Question 3

Four genomic hypotheses receive prespecified Bonferroni weights 0.400.40, 0.300.30, 0.200.20, and 0.100.10, which sum to one. The family-level significance level is 0.050.05, and the observed p-values are, respectively, 0.0210.021, 0.0140.014, 0.0110.011, and 0.00480.0048. A hypothesis is rejected when its p-value does not exceed its weighted allocation.

Which hypotheses are rejected?

  1. Reject only H1H_1 and H2H_2 because they have the two largest prespecified weights.
  2. Reject only H2H_2 and H4H_4 because their p-values meet their allocated thresholds. (correct answer)
  3. Reject H2H_2, H3H_3, and H4H_4 because each p-value is below the unweighted level.
  4. Reject all four hypotheses because the four observed p-values sum to less than 0.050.05.
Explanation: Weighted Bonferroni procedures allocate the family-wise error rate across hypotheses by assigning each one a fraction of α\alpha. The individual threshold for hypothesis HiH_i is simply wi×αw_i \times \alpha, where wiw_i is the prespecified weight and α=0.05\alpha = 0.05. You reject HiH_i if and only if its p-value is at or below that threshold — no cross-hypothesis logic involved. Computing each threshold: H1H_1 gets 0.40×0.05=0.0200.40 \times 0.05 = 0.020, H2H_2 gets 0.30×0.05=0.0150.30 \times 0.05 = 0.015, H3H_3 gets 0.20×0.05=0.0100.20 \times 0.05 = 0.010, and H4H_4 gets 0.10×0.05=0.0050.10 \times 0.05 = 0.005. Now compare: H1H_1: 0.021>0.0200.021 > 0.020 — not rejected. H2H_2: 0.0140.0150.014 \leq 0.015 — rejected. H3H_3: 0.011>0.0100.011 > 0.010 — not rejected. H4H_4: 0.00480.0050.0048 \leq 0.005 — rejected. Only H2H_2 and H4H_4 clear their individual hurdles, confirming answer B. Choice A is wrong because having a large weight gives a larger threshold, making rejection easier — yet H1H_1 still fails. Ranking by weight alone tells you nothing about whether a p-value actually beats its allocation. Choice C applies the flat, unweighted 0.050.05 threshold to all hypotheses, which ignores the entire point of the weighting scheme — H3H_3's threshold is only 0.0100.010, not 0.050.05. Choice D invents a rule that doesn't exist; summing p-values is not a valid inference procedure, and the Bonferroni framework never aggregates p-values this way. Your key takeaway: always compute the hypothesis-specific threshold wiαw_i \alpha before comparing — a small weight can make a hypothesis harder to reject even if its p-value looks impressively small.

Question 4

A gatekeeping procedure first tests three primary hypotheses, each at level 0.05/30.05/3. If at least one primary hypothesis is rejected, four secondary hypotheses are then tested, each at level 0.05/40.05/4. No further adjustment is made between the primary and secondary families.

Which statement most accurately characterizes this procedure?

  1. It controls family-wise error strongly at 0.050.05 because each stage separately applies Bonferroni correction, and those within-stage controls combine to cover all null configurations.
  2. It controls family-wise error strongly only when all primary and secondary test statistics are mutually independent, because dependence allows secondary rejections to occur even without a genuine primary signal.
  3. It fails even under the global null because a secondary rejection is possible whenever a primary hypothesis is rejected, causing the two stage-specific error budgets to sum to 0.100.10.
  4. It controls family-wise error under the global null, but strong control can fail under partial null configurations in which the gate opens with high probability and the secondary error budget is not shared with the primary budget. (correct answer)
Explanation: Gatekeeping procedures test hypotheses in sequential "families," and the critical concept you need to apply is the distinction between weak control (controlling FWER only under the global null) and strong control (controlling FWER under every partial null configuration). Under the global null — all seven hypotheses are true — no primary hypothesis should be rejected. Because the primaries are tested at 0.05/30.05/3 each, the probability of any primary rejection is at most 0.050.05 by Bonferroni. Since the gate never opens, the secondary stage contributes zero error. So weak control holds cleanly. The problem emerges under a partial null: suppose the three primary hypotheses are all false (truly non-null), so the gate opens with probability approaching 1. Now the four secondary hypotheses — which are true nulls — are each tested at 0.05/40.05/4. The probability of at least one false secondary rejection approaches 0.050.05 by Bonferroni, but this error is independent of the primary budget. The two budgets are never pooled or constrained to sum to 0.050.05 across families, so the secondary family alone can exhaust 0.050.05 without any primary false rejection occurring. Strong control therefore fails because the secondary error isn't subordinated to leftover primary budget. D captures this exactly. A is wrong because within-stage Bonferroni doesn't automatically combine across stages — the secondary budget isn't conditional on leftover primary alpha. B is wrong because the failure isn't about dependence; it occurs even under independence due to the unconstrained secondary budget. C is wrong because under the global null the gate stays closed, so budgets don't literally "sum"; weak control is intact. Remember: a gatekeeping procedure guarantees strong FWER control only if secondary alpha is explicitly drawn from remaining primary alpha, not allocated independently.

Question 5

Six hypotheses have ordered p-values 0.0060.006, 0.0090.009, 0.0120.012, 0.0210.021, 0.0400.040, and 0.2000.200. The family-level significance level is 0.050.05. The hypotheses are analyzed both by single-step Bonferroni and by Holm's step-down Bonferroni procedure.

How many hypotheses are rejected by each procedure?

  1. Single-step Bonferroni rejects one hypothesis, whereas Holm's procedure rejects four hypotheses.
  2. Single-step Bonferroni rejects two hypotheses, whereas Holm's procedure rejects three hypotheses.
  3. Single-step Bonferroni rejects one hypothesis, whereas Holm's procedure rejects three hypotheses. (correct answer)
  4. Single-step Bonferroni rejects three hypotheses, whereas Holm's procedure rejects four hypotheses.
Explanation: When tackling multiple testing problems, always apply each procedure's decision rule mechanically and carefully — the differences between methods are easy to blur under exam pressure. Single-step Bonferroni uses one threshold for all hypotheses: reject HiH_i if pi<α/m=0.05/60.0083p_i < \alpha/m = 0.05/6 \approx 0.0083. Scanning the ordered p-values: 0.006<0.00830.006 < 0.0083 ✓, but 0.009>0.00830.009 > 0.0083 ✗. Only one hypothesis is rejected. Holm's step-down procedure uses sequentially adjusted thresholds. Starting from the smallest p-value, compare p(i)p_{(i)} against α/(mi+1)\alpha/(m - i + 1):
  • p(1)=0.006p_{(1)} = 0.006 vs. 0.05/60.00830.05/6 \approx 0.0083 → reject ✓
  • p(2)=0.009p_{(2)} = 0.009 vs. 0.05/5=0.0100.05/5 = 0.010 → reject ✓
  • p(3)=0.012p_{(3)} = 0.012 vs. 0.05/4=0.01250.05/4 = 0.0125 → reject ✓
  • p(4)=0.021p_{(4)} = 0.021 vs. 0.05/30.01670.05/3 \approx 0.0167 → fail to reject ✗ → stop
Holm's rejects three hypotheses, confirming answer C. Choice A overcounts Holm's rejections at four — a student who forgets to stop at the first non-rejection makes this mistake. Choice B incorrectly gives Bonferroni two rejections, likely by rounding 0.05/60.05/6 up to 0.0100.010, which just barely catches p(2)=0.009p_{(2)} = 0.009. Choice D inflates both counts, suggesting the student may have applied a less conservative threshold throughout. The key study tip: Holm's procedure stops the moment a p-value exceeds its threshold — all subsequent hypotheses are retained, even if their p-values seem small relative to their thresholds.

Question 6

Eight mutually independent tests are performed, and all eight null hypotheses are true. Each test rejects when its p-value is at most 0.010.01.

What is the family-wise error rate, and how does it compare with the Bonferroni bound?

  1. It is approximately 0.01000.0100, whereas the Bonferroni bound is 0.08000.0800.
  2. It is approximately 0.07730.0773, whereas the Bonferroni bound is 0.08000.0800. (correct answer)
  3. It is exactly 0.08000.0800, which equals the Bonferroni bound in this setting.
  4. It is approximately 0.92270.9227, whereas the Bonferroni bound is 0.92000.9200.
Explanation: When you see a question about multiple testing, your first instinct should be to distinguish between the exact family-wise error rate (FWER) and the Bonferroni bound, which is only an approximation. The FWER is the probability of making at least one Type I error across all tests. With eight independent tests each at level α=0.01\alpha = 0.01, the probability of not rejecting any true null is (10.01)8=(0.99)80.9227(1 - 0.01)^8 = (0.99)^8 \approx 0.9227. Therefore, the FWER is 10.9227=0.07731 - 0.9227 = 0.0773. The Bonferroni bound simply multiplies: 8×0.01=0.088 \times 0.01 = 0.08. So the exact FWER is approximately 0.07730.0773, which is less than the Bonferroni bound of 0.08000.0800. This confirms B. Now for the distractors. A confuses the per-test error rate (0.010.01) with the FWER — a classic trap of ignoring the accumulation of error across tests. C claims the FWER exactly equals the Bonferroni bound; this would only be true if the bound were tight, but Bonferroni uses the inequality P(A1Ak)P(Ai)P(A_1 \cup \cdots \cup A_k) \leq \sum P(A_i), which is strict unless events are mutually exclusive. D reports the complement — the probability of no error (0.9227\approx 0.9227) — and incorrectly pairs it with 10.08=0.921 - 0.08 = 0.92, mixing up the FWER with its complement. A good rule of thumb: the Bonferroni bound always overestimates the true FWER when tests are independent, because it ignores the fact that simultaneous rejections are possible. The exact calculation via 1(1α)k 1-(1-\alpha)^k is always your more precise tool.

Question 7

Twenty null hypotheses are tested with valid p-values. Under the global null, all twenty p-values are almost surely identical to the same random variable distributed uniformly on [0,1][0,1]. A single-step Bonferroni procedure targets family-wise error rate 0.050.05.

What is the actual family-wise error rate of the procedure under this dependence structure?

  1. It is 0.00250.0025 because either all hypotheses cross the Bonferroni threshold or none do. (correct answer)
  2. It is approximately 0.04880.0488 because twenty independent opportunities for rejection remain.
  3. It is exactly 0.05000.0500 because Bonferroni exhausts the full family-level error budget.
  4. It is 1.00001.0000 because perfect dependence causes all hypotheses to be rejected together.
Explanation: Whenever you see a question mixing multiple testing corrections with dependence structure, your first instinct should be to ask: how does this dependence change the probability that at least one false rejection occurs? The Bonferroni procedure rejects hypothesis ii when piα/m=0.05/20=0.0025p_i \leq \alpha/m = 0.05/20 = 0.0025. Normally, the family-wise error rate (FWER) is defined as P(at least one false rejection)P(\text{at least one false rejection}). Under the standard independent case, this approaches but stays below α=0.05\alpha = 0.05. But here, the setup is extreme: all twenty p-values are almost surely identical — they share the exact same realized value. This is perfect positive dependence. Because every pip_i equals the same draw UUniform(0,1)U \sim \text{Uniform}(0,1), either all hypotheses are rejected simultaneously (when U0.0025U \leq 0.0025) or none are. The probability of any false rejection is therefore exactly P(U0.0025)=0.0025P(U \leq 0.0025) = 0.0025. Answer A is correct. Answer B is wrong because it treats the hypotheses as independent, computing something like 1(10.0025)200.04881-(1-0.0025)^{20} \approx 0.0488. But independence fails completely here — there is only one effective test. Answer C is wrong because Bonferroni is conservative in general and only exhausts the full α\alpha budget asymptotically under independence; perfect positive dependence makes it dramatically more conservative. Answer D confuses the probability of rejection with the FWER itself — perfect dependence doesn't guarantee rejection, it just synchronizes outcomes. The key takeaway: Bonferroni is conservative under positive dependence. The more correlated your test statistics, the fewer "independent shots" you effectively have, and the lower your true FWER falls below the nominal level.

Question 8

Five true null hypotheses are tested, and each test has marginal type I error probability exactly 0.010.01. Nothing is known about the dependence among the five rejection events.

What is the tightest distribution-free range for the family-wise error rate?

  1. The family-wise error rate must lie between 00 and 0.010.01, because the union of five events cannot exceed the smallest constituent probability.
  2. The family-wise error rate must equal 1(10.01)50.0491-(1-0.01)^5\approx 0.049, because five independent rejection events uniquely determine the union probability.
  3. The family-wise error rate must lie between 0.010.01 and 1(10.01)50.0491-(1-0.01)^5\approx 0.049, because independence provides the sharpest upper bound without distributional assumptions.
  4. The family-wise error rate must lie between 0.010.01 and 0.050.05, because the union is at least as large as any marginal event and at most the sum of the marginal probabilities. (correct answer)
Explanation: When you see a question about the family-wise error rate (FWER) under unknown dependence, your job is to find the sharpest distribution-free bounds — bounds that hold regardless of how the test statistics are correlated. Two classical inequalities do the heavy lifting here. First, Boole's inequality (the union bound) says P(A1Ak)i=1kP(Ai)P(A_1 \cup \cdots \cup A_k) \leq \sum_{i=1}^k P(A_i), giving an upper bound of 5×0.01=0.055 \times 0.01 = 0.05. Second, monotonicity of probability gives a lower bound: the union of events is at least as probable as any single event, so FWER0.01\text{FWER} \geq 0.01. Together, these bracket the FWER in [0.01,0.05][0.01,\, 0.05]. Both bounds are tight — perfect positive dependence (all tests reject together) achieves the lower bound of 0.010.01, and mutual exclusivity achieves the upper bound of 0.050.05. That confirms D is correct. A is wrong on two counts: the union is never smaller than a constituent probability, so claiming the FWER is below 0.010.01 contradicts monotonicity entirely. B is wrong because 1(10.01)51-(1-0.01)^5 is the union probability only under independence. Without knowing the dependence structure, this formula doesn't apply — it's not a universal identity. C is wrong for a subtler reason: it treats the independence-based formula as an upper bound, but under positive dependence the FWER can actually be below that value (all the way down to 0.010.01), so the independence value is neither a floor nor the tightest ceiling. Study tip: On distribution-free problems, always pair Boole's inequality (upper bound = sum of marginals) with monotonicity (lower bound = largest single marginal). These two tools together give you the tightest bounds you can guarantee without dependence assumptions.

Question 9

A clinical trial evaluates three doses on each of two confirmatory endpoints, producing six null hypotheses. All six null p-values are mutually independent and uniformly distributed. Investigators apply Bonferroni separately within each endpoint: each of its three dose comparisons is tested at level 0.05/30.05/3. They then make confirmatory claims across both endpoints.

Under the global null, what is the family-wise error rate across all six claims?

  1. It is 1(10.05/3)60.0961-(1-0.05/3)^6\approx 0.096. (correct answer)
  2. It is 1(10.05/6)60.0491-(1-0.05/6)^6\approx 0.049.
  3. It is exactly 6(0.05/3)=0.1006(0.05/3)=0.100.
  4. It is at most 0.0500.050 because each endpoint family is Bonferroni-controlled.
Explanation: Whenever you see multiple testing problems where procedures are applied within subgroups before claims are made across all hypotheses, you need to track the actual per-test alpha level and then compute the family-wise error rate (FWER) over the full collection of tests. Here, Bonferroni is applied separately within each of the two endpoints, dividing 0.05 by 3, so each individual null hypothesis is tested at level α=0.05/3\alpha^* = 0.05/3. This controls the FWER within each endpoint at 0.05 — but it does not control the FWER across all six hypotheses simultaneously. Since all six p-values are independent and each is rejected with probability 0.05/30.05/3 under the global null, the probability that at least one false rejection occurs across all six is: FWER=1(10.053)60.096\text{FWER} = 1 - \left(1 - \frac{0.05}{3}\right)^6 \approx 0.096 This confirms A is correct. B is wrong because 1(10.05/6)61-(1-0.05/6)^6 corresponds to testing each hypothesis at 0.05/60.05/6 — a full Bonferroni correction over all six. That's not what was done here. C applies the Bonferroni union bound: 6×(0.05/3)=0.106 \times (0.05/3) = 0.10. While this is a valid upper bound (and the exact Bonferroni inequality), the question asks for the actual FWER under independence, which is the exact binomial complement formula — slightly less than 0.10, not equal to it. D is the key trap: controlling FWER within each endpoint family at 0.05 does not guarantee global FWER control. Two separate 5%-level procedures can still inflate the combined error rate above 5%. Study tip: Always distinguish local FWER control (within a family) from global FWER control (across all families). Splitting hypotheses into subgroups and applying corrections within each subgroup is not equivalent to a single correction over all hypotheses.

Question 10

In a regression analysis, four prespecified coefficients are to be reported with two-sided confidence intervals. The investigator wants simultaneous coverage of at least 0.950.95 using a Bonferroni construction. Assume each standardized estimator has a standard normal distribution. Relevant quantiles are z0.9875=2.241z_{0.9875}=2.241, z0.99375=2.498z_{0.99375}=2.498, and z0.995=2.576z_{0.995}=2.576.

Which critical value should be used in each interval?

  1. Use z0.9875=2.241z_{0.9875}=2.241 because the family-level error is divided across four intervals.
  2. Use z0.99375=2.498z_{0.99375}=2.498 because the adjusted error must also be divided across two tails. (correct answer)
  3. Use z0.995=2.576z_{0.995}=2.576 because each tail receives one-fifth of the family-level error.
  4. Use z0.975=1.960z_{0.975}=1.960 because simultaneous coverage follows from marginal coverage.
Explanation: When constructing simultaneous confidence intervals using the Bonferroni method, you need to think carefully about two separate error-splitting steps: one across the number of intervals, and one across the two tails of each interval. The logic works like this. You want a family-wise error rate of at most α=0.05\alpha = 0.05 across m=4m = 4 intervals. Bonferroni splits this evenly, allocating α/m=0.05/4=0.0125\alpha/m = 0.05/4 = 0.0125 of error to each individual interval. But each interval is two-sided, so that 0.01250.0125 must be split again across two tails, giving 0.0125/2=0.006250.0125/2 = 0.00625 per tail. You therefore need the quantile that leaves 0.006250.00625 in each tail, which corresponds to the 10.00625=0.993751 - 0.00625 = 0.99375 quantile — exactly z0.99375=2.498z_{0.99375} = 2.498. That confirms B is correct. A is wrong because it only performs the first split (across 4 intervals) and uses z0.9875z_{0.9875}, which corresponds to a one-sided cutoff for α/m\alpha/m, ignoring the second split for the two-sided nature of each interval. C incorrectly divides by 5 instead of 4, which has no justification — there are 4 coefficients, not 5. D represents a fundamental misunderstanding: marginal (individual) 95% coverage does not guarantee simultaneous 95% coverage across multiple intervals; the Bonferroni adjustment exists precisely to correct for this. A useful memory rule: for Bonferroni with mm two-sided intervals at family level α\alpha, the per-tail error is α/(2m)\alpha/(2m), so you always divide by both mm and 22 before looking up your quantile.