Biostatistics Quiz: Multiple Comparisons And Family Wise Error
20 questions · exam conditions
0:00
Multiple Comparisons And Family Wise ErrorQuestion 1 of 20

A researcher conducts 20 independent hypothesis tests, each at α = 0.05. She observes 3 significant results. Using the False Discovery Rate (FDR) approach with the Benjamini-Hochberg procedure, she wants to control FDR at 0.10. What is the critical p-value for the most significant result?

0.10 × (1/20) = 0.005
0.05 × (1/20) = 0.0025
0.10/20 = 0.005
0.10 × (3/20) = 0.015
← Back to quizzes

Biostatistics Quiz

Biostatistics Quiz: Multiple Comparisons And Family Wise Error

Practice Multiple Comparisons And Family Wise Error in Biostatistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Multiple Comparisons And Family Wise Error, giving you a quick way to practice the rules, question types, and explanations that matter most for Biostatistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A researcher conducts 20 independent hypothesis tests, each at α = 0.05. She observes 3 significant results. Using the False Discovery Rate (FDR) approach with the Benjamini-Hochberg procedure, she wants to control FDR at 0.10. What is the critical p-value for the most significant result?

  1. 0.10 × (1/20) = 0.005 (correct answer)
  2. 0.05 × (1/20) = 0.0025
  3. 0.10/20 = 0.005
  4. 0.10 × (3/20) = 0.015
Explanation: In the Benjamini-Hochberg procedure, for the smallest p-value (rank i=1), the critical value is (i/m) × α = (1/20) × 0.10 = 0.005. Choice B incorrectly uses 0.05 instead of the desired FDR level 0.10. Choice C uses division instead of the correct B-H formula. Choice D incorrectly uses the number of observed significant results (3) instead of the rank (1).

Question 2

A researcher conducts a study comparing the effectiveness of 5 different pain medications. She performs pairwise t-tests between all possible pairs of treatments without any adjustment for multiple comparisons. If she uses α = 0.05 for each individual test, what is the probability of making at least one Type I error (family-wise error rate)?

  1. 0.05
  2. 0.25
  3. 0.40 (correct answer)
  4. 0.50
  5. 0.64
Explanation: When you encounter multiple comparison problems, you're dealing with the family-wise error rate (FWER) - the probability of making at least one Type I error across all tests in a "family" of comparisons. With 5 treatments, the number of possible pairwise comparisons is (52)=10\binom{5}{2} = 10 tests. Each individual test has α = 0.05, meaning there's a 95% chance of correctly avoiding a Type I error on any single test. To find the probability of making at least one Type I error, it's easier to calculate the probability of making NO Type I errors across all tests, then subtract from 1. If the tests are independent, the probability of avoiding Type I error on all 10 tests is (0.95)10=0.599(0.95)^{10} = 0.599. Therefore, the family-wise error rate is 1−0.599=0.4011 - 0.599 = 0.401, which rounds to 0.40. Looking at the wrong answers: (A) 0.05 represents the error rate for just one individual test, ignoring the multiple comparison problem entirely. (B) 0.25 might result from incorrectly calculating fewer comparisons or using faulty probability logic. (D) 0.50 could come from oversimplifying the problem or misunderstanding how probabilities compound. Study tip: Remember that when conducting multiple tests, your actual Type I error rate inflates dramatically above your intended α level. Always count the total number of comparisons carefully - for k groups, you'll have (k2)\binom{k}{2} pairwise tests. This is why procedures like Bonferroni correction exist to control family-wise error rates.

Question 3

A pharmaceutical company tests 8 different formulations of a drug. They plan to use Dunnett's test to compare each of 7 experimental formulations against a single control. If they want to maintain a family-wise error rate of 0.05, and each individual comparison uses the Dunnett critical value of 2.64, what is the probability that they will incorrectly conclude that at least one experimental formulation differs from control when in fact none do?

  1. Exactly 0.05, as designed by the Dunnett procedure (correct answer)
  2. 7 × 0.05 = 0.35, due to multiple comparisons
  3. 1 - (0.95)⁷ = 0.30, assuming independence
  4. 0.05/7 = 0.007, due to Bonferroni adjustment
  5. Cannot be determined without knowing the correlation structure
Explanation: When you encounter questions about multiple comparisons in biostatistics, the key is understanding what different statistical procedures are designed to control. This question tests whether you know what "family-wise error rate" means and how Dunnett's test works. Dunnett's test is specifically designed to control the family-wise error rate (FWER) at your chosen alpha level when comparing multiple treatments to a single control. The FWER is the probability of making at least one Type I error across all comparisons in your "family" of tests. When the researchers set their FWER at 0.05 and use Dunnett's procedure with the appropriate critical value, the probability of incorrectly concluding that at least one formulation differs from control (when none actually do) is exactly 0.05 by design. Answer A is correct because Dunnett's test maintains the specified family-wise error rate regardless of how many comparisons you make. Answer B (0.35) incorrectly assumes you simply multiply the error rate by the number of comparisons, which would be true if you performed multiple independent t-tests without adjustment. Answer C (0.30) applies the formula for independent events, but Dunnett comparisons aren't independent since they all use the same control group. Answer D (0.007) confuses Dunnett's procedure with Bonferroni correction, where you would divide alpha by the number of comparisons. Remember: Dunnett's test is purpose-built for comparing multiple treatments to one control while maintaining your desired family-wise error rate. Unlike other multiple comparison procedures, you don't need additional calculations—the method does the work for you.

Question 4

A researcher conducts 20 hypothesis tests and wants to control the False Discovery Rate (FDR) at 0.10 using the Benjamini-Hochberg procedure. The ordered p-values are: 0.001, 0.003, 0.008, 0.015, 0.023, 0.031, 0.045, 0.052, 0.067, 0.078, 0.089, 0.095, 0.112, 0.128, 0.145, 0.167, 0.189, 0.234, 0.278, 0.334. What is the largest k such that p(k) ≤ (k/20) × 0.10?

  1. k = 4, since 0.015 ≤ (4/20) × 0.10 = 0.020
  2. k = 5, since 0.023 ≤ (5/20) × 0.10 = 0.025 (correct answer)
  3. k = 6, since 0.031 ≤ (6/20) × 0.10 = 0.030
  4. k = 7, since 0.045 ≤ (7/20) × 0.10 = 0.035
  5. k = 8, since 0.052 ≤ (8/20) × 0.10 = 0.040
Explanation: When you encounter False Discovery Rate (FDR) control questions, you're working with a sequential procedure that protects against finding too many false positives among your significant results. The Benjamini-Hochberg procedure finds the largest k where the k-th smallest p-value satisfies p(k)≤km×αp_{(k)} \leq \frac{k}{m} \times \alpha, where m is the total number of tests and α is your desired FDR level. To find the correct answer, you must check each k sequentially and identify the largest one that satisfies the inequality. With m = 20 tests and α = 0.10: For k = 4: p(4)=0.015p_{(4)} = 0.015 and 420×0.10=0.020\frac{4}{20} \times 0.10 = 0.020. Since 0.015 ≤ 0.020, this works. For k = 5: p(5)=0.023p_{(5)} = 0.023 and 520×0.10=0.025\frac{5}{20} \times 0.10 = 0.025. Since 0.023 ≤ 0.025, this works. For k = 6: p(6)=0.031p_{(6)} = 0.031 and 620×0.10=0.030\frac{6}{20} \times 0.10 = 0.030. Since 0.031 > 0.030, this fails. Answer B correctly identifies k = 5 as the largest value that works. Answer A stops too early at k = 4, missing that k = 5 also satisfies the condition. Answer C incorrectly states that k = 6 works when 0.031 > 0.030. Answer D makes the same error, claiming k = 7 works when both k = 6 and k = 7 violate the inequality. Remember: always work sequentially through all possible k values until you find the first failure, then select the largest k that succeeded. Don't stop at the first success.

Question 5

A biostatistician analyzes data from 5 treatment groups using one-way ANOVA and obtains p = 0.001. She then uses Fisher's LSD method for post-hoc comparisons. Her colleague argues that she should use Tukey's HSD instead. What is the primary difference in the error control philosophy between these two approaches?

  1. Fisher's LSD controls per-comparison error rate; Tukey's HSD controls family-wise error rate (correct answer)
  2. Fisher's LSD controls family-wise error rate; Tukey's HSD controls false discovery rate
  3. Both control family-wise error rate, but Tukey's HSD is more powerful
  4. Fisher's LSD controls false discovery rate; Tukey's HSD controls per-comparison error rate
  5. Both control per-comparison error rate, but Fisher's LSD requires significant ANOVA first
Explanation: When you encounter questions about post-hoc testing methods after ANOVA, focus on understanding what type of error each method is designed to control. This is crucial because different methods make different trade-offs between statistical power and error protection. Fisher's Least Significant Difference (LSD) controls the per-comparison error rate - meaning it maintains your chosen alpha level (like 0.05) for each individual pairwise comparison. If you're making 10 comparisons, each one has a 5% chance of Type I error, but your overall risk across all comparisons can be much higher. Tukey's Honestly Significant Difference (HSD) controls the family-wise error rate - it keeps your overall Type I error rate at 5% across the entire family of comparisons, making it much more conservative. Looking at the incorrect options: Choice B reverses the methods and incorrectly introduces false discovery rate, which is a different error control approach used in methods like Benjamini-Hochberg. Choice C incorrectly states both control family-wise error rate - Fisher's LSD does not. Choice D again reverses the methods and misapplies false discovery rate concepts. The colleague's argument makes sense because with 5 treatment groups, you'd have 10 possible pairwise comparisons, dramatically inflating your Type I error rate if using Fisher's LSD. Answer A correctly identifies this fundamental difference. Study tip: Remember the trade-off: Fisher's LSD is more powerful (easier to detect differences) but riskier for multiple comparisons; Tukey's HSD is more conservative but protects against inflated error rates when making many comparisons.

Question 6

An epidemiologist conducts a study comparing 5 different dietary interventions. The ANOVA yields F = 4.2 (p = 0.01). She plans to use Scheffe's method for post-hoc comparisons because she wants to test not only pairwise differences but also some complex contrasts (e.g., comparing the average of two diets against the average of the other three). What is the key advantage of Scheffe's method in this context compared to other multiple comparison procedures?

  1. Scheffe's method has the highest statistical power for detecting pairwise differences
  2. Scheffe's method allows testing of any linear contrast while maintaining family-wise error control (correct answer)
  3. Scheffe's method does not require a significant ANOVA result before testing contrasts
  4. Scheffe's method controls false discovery rate rather than family-wise error rate
  5. Scheffe's method is computationally simpler than other post-hoc procedures
Explanation: When you encounter ANOVA followed by multiple comparisons, you need to understand that different post-hoc methods have distinct strengths and limitations. The key insight here is recognizing what makes Scheffe's method unique among multiple comparison procedures. Scheffe's method stands out because it allows you to test any possible linear contrast while maintaining strict family-wise error rate control. This means you can compare individual groups, combine groups (like averaging two diets against three others), or test any other linear combination you can imagine - all while keeping your overall Type I error rate at the desired level (typically α = 0.05). This flexibility is exactly what the epidemiologist needs for her complex contrasts. Let's examine why the other options miss the mark. Choice A is incorrect because Scheffe's method actually has lower power for pairwise comparisons compared to methods like Tukey's HSD - it trades power for flexibility. Choice C is wrong because Scheffe's method, like most post-hoc tests, is designed to follow a significant ANOVA result (which we have here with p = 0.01). Choice D confuses two different error control approaches - Scheffe's controls family-wise error rate (probability of making any Type I error), not false discovery rate (expected proportion of false discoveries). Study tip: Remember that Scheffe's = maximum flexibility for contrasts but lower power. If you see complex contrasts mentioned (not just pairwise comparisons), Scheffe's is likely the answer. For simple pairwise comparisons only, consider Tukey's or Bonferroni instead.

Question 7

A pharmaceutical researcher conducts a dose-response study with 6 dose levels (including placebo). She observes a significant trend (p = 0.003) using linear contrast analysis. Now she wants to identify which specific dose levels differ significantly from placebo using Dunnett's test. If 3 out of 5 active doses show p-values less than 0.05 in individual comparisons with placebo, what can she conclude about family-wise error control?

  1. The family-wise error rate is maintained at 0.05 regardless of how many comparisons are significant (correct answer)
  2. The family-wise error rate is inflated to approximately 3 × 0.05 = 0.15
  3. She needs to apply Bonferroni correction: 0.05/5 = 0.01 for each comparison
  4. The family-wise error rate cannot be determined without the specific p-values
  5. The significant trend test precludes the need for family-wise error control
Explanation: When you encounter questions about multiple comparisons in biostatistics, focus on understanding what specific statistical procedure is being used and how it controls error rates. Different methods have different properties for family-wise error control. Dunnett's test is specifically designed to compare multiple treatment groups against a single control group while maintaining the family-wise error rate at your chosen alpha level (typically 0.05). This is a key feature that distinguishes it from simple multiple t-tests. The test accomplishes this through adjusted critical values that account for the correlation structure between comparisons that all share the same control group. Answer A is correct because Dunnett's test maintains the family-wise error rate at 0.05 regardless of how many individual comparisons reach significance. The procedure is designed with this protection built in. Answer B reflects a common misconception about multiplying individual p-values to estimate family-wise error. This calculation would apply to uncorrected multiple comparisons, but Dunnett's test already incorporates the necessary adjustments. Answer C suggests applying Bonferroni correction, but this is unnecessary and overly conservative when using Dunnett's test. You would only need Bonferroni if you were performing simple t-tests without any multiple comparison adjustment. Answer D incorrectly suggests that family-wise error control depends on the specific p-values obtained. However, proper multiple comparison procedures control the error rate at the design stage, not based on your results. Remember: when a question mentions a specific multiple comparison test by name, that test likely has built-in error control properties that you should recognize.

Question 8

A biomedical researcher analyzes gene expression data for 3 treatment conditions with 8 replicates each. After obtaining a significant ANOVA result (F = 12.4, p < 0.001), she performs pairwise t-tests and finds: Treatment A vs B (p = 0.002), Treatment A vs C (p = 0.15), Treatment B vs C (p = 0.008). She applies the Holm-Bonferroni method with α = 0.05. What is the step-by-step conclusion?

  1. Order: 0.002, 0.008, 0.15; Test against 0.0167, 0.025, 0.05; All three remain significant
  2. Order: 0.002, 0.008, 0.15; Test against 0.0167, 0.025, 0.05; First two remain significant (correct answer)
  3. Order: 0.002, 0.008, 0.15; Test against 0.0167, 0.025, 0.05; Only first remains significant
  4. Order: 0.002, 0.008, 0.15; Test against 0.0125, 0.0167, 0.025; Only first remains significant
  5. Order: 0.002, 0.008, 0.15; Test against 0.0125, 0.0167, 0.025; First two remain significant
Explanation: When you encounter multiple comparison problems after ANOVA, the Holm-Bonferroni method provides a step-down approach that's more powerful than the standard Bonferroni correction while controlling family-wise error rate. The Holm-Bonferroni procedure works systematically: First, order your p-values from smallest to largest: 0.002, 0.008, 0.15. Then create adjusted α levels using the formula α/(m−i+1)\alpha/(m-i+1), where m is the total number of comparisons (3) and i is the rank position. For the first comparison (i=1): α=0.05/(3−1+1)=0.0167\alpha = 0.05/(3-1+1) = 0.0167 For the second comparison (i=2): α=0.05/(3−2+1)=0.025\alpha = 0.05/(3-2+1) = 0.025
For the third comparison (i=3): α=0.05/(3−3+1)=0.05\alpha = 0.05/(3-3+1) = 0.05 You test each p-value sequentially against its adjusted α. If any test fails, you stop and reject no further hypotheses. Here: 0.002 < 0.0167 (significant), 0.008 < 0.025 (significant), but 0.15 > 0.05 (not significant). Since the third test fails, only the first two comparisons remain significant. Answer A incorrectly claims all three remain significant, ignoring that 0.15 > 0.05. Answer C stops too early, missing that the second comparison also passes. Answer D uses incorrect α adjustments (0.0125, 0.0167, 0.025), which don't follow the Holm formula. Remember: Holm-Bonferroni is sequential—once you hit a non-significant result, you stop testing. Always double-check your α calculations using the step-down formula.

Question 9

A public health researcher studies the effectiveness of 4 different smoking cessation programs plus a control group (5 groups total). She plans to use planned contrasts to test specific hypotheses: (1) Any program vs control, (2) Behavioral programs vs medical programs, (3) Individual vs group therapy within behavioral programs. If she uses α = 0.05 for each planned contrast, what is her approach to multiple comparison error control?

  1. She is controlling family-wise error rate at 0.05 through careful contrast planning
  2. She is not controlling for multiple comparisons, treating each contrast as independent
  3. She is implicitly using Bonferroni correction by limiting the number of contrasts
  4. She is controlling per-comparison error rate but not family-wise error rate (correct answer)
  5. She is using an invalid approach that requires post-hoc correction methods instead
Explanation: When you encounter questions about planned contrasts and multiple comparisons, focus on distinguishing between per-comparison error rate and family-wise error rate (FWER). This distinction is crucial in biostatistics. In this scenario, the researcher is conducting three planned contrasts, each tested at α = 0.05. This means each individual test has a 5% chance of Type I error (false positive). However, when multiple tests are performed on the same dataset, the probability of making at least one Type I error across all tests increases. The researcher is controlling the per-comparison error rate at 0.05 for each contrast but is not controlling the family-wise error rate, which would be the probability of any false positive across all three tests. Answer D correctly identifies this approach: controlling per-comparison error rate but not family-wise error rate. Answer A is wrong because true FWER control would require adjusting the significance level (like Bonferroni) or using other methods to keep the overall Type I error at 0.05 across all contrasts. Answer B is incorrect because the researcher is controlling something (per-comparison error), just not FWER. Answer C misunderstands Bonferroni correction—simply limiting contrasts doesn't constitute Bonferroni adjustment, which would require dividing α by the number of comparisons (0.05/3 = 0.017 per test). Remember: planned contrasts at unadjusted α levels control per-comparison error but allow family-wise error to inflate above the nominal α level. Always clarify which type of error control is being used.

Question 10

A researcher conducts a randomized trial comparing 6 different pain medications. After finding a significant ANOVA result, she wants to determine which medications are significantly different from each other. She has two options: (A) Perform all 15 pairwise comparisons using Tukey's HSD, or (B) Use Fisher's LSD but only test pairs where she has a specific clinical hypothesis. Option B would involve 8 comparisons. Which statement best describes the trade-off between these approaches?

  1. Option A provides better family-wise error control; Option B provides higher power for planned comparisons (correct answer)
  2. Option A provides higher power; Option B provides better family-wise error control through fewer tests
  3. Both options provide equivalent family-wise error control since Fisher's LSD requires significant ANOVA
  4. Option A is more conservative but protects against data dredging; Option B is more liberal but scientifically justified
  5. The choice depends only on computational convenience since both control family-wise error equally
Explanation: When you encounter post-hoc comparison questions, focus on the fundamental trade-off between Type I error control and statistical power. After a significant ANOVA, you need to decide how to balance these competing concerns when making multiple pairwise comparisons. Tukey's HSD (Honestly Significant Difference) is designed specifically to control family-wise error rate at α = 0.05 across all possible pairwise comparisons. It accomplishes this by using a more stringent critical value than standard t-tests, but this conservatism comes at the cost of reduced power to detect true differences. Fisher's LSD, while less conservative, offers higher power for individual comparisons. When you limit it to planned, hypothesis-driven comparisons (8 instead of all 15), you're using scientific reasoning rather than statistical corrections to justify the approach. This targeted strategy maintains higher power for the comparisons that matter most clinically. Option A correctly identifies this trade-off: Tukey's provides superior family-wise error control while Fisher's offers higher power for planned comparisons. Option B incorrectly reverses the power relationship—Tukey's is more conservative, not more powerful. Option C wrongly suggests equivalent error control; requiring significant ANOVA helps but doesn't make Fisher's LSD as conservative as Tukey's. Option D contains some truth about conservatism and scientific justification but misses the key statistical concept of power. Remember this pattern: more conservative multiple comparison procedures provide better error control but sacrifice power. When choosing post-hoc tests, always consider whether you prioritize avoiding false positives (use Tukey's) or maximizing ability to detect true differences (use Fisher's with planned comparisons).

Question 11

A genomics researcher tests 10,000 genes for differential expression between two conditions. Using α = 0.001 for each test, she identifies 50 genes as significantly differentially expressed. If she assumes that 90% of genes are truly not differentially expressed (null hypothesis is true), approximately how many of her 50 'significant' findings are likely to be false discoveries?

  1. 10,000 × 0.90 × 0.001 = 9 false discoveries
  2. 50 × 0.10 = 5 false discoveries
  3. 50 × 0.90 = 45 false discoveries
  4. 10,000 × 0.001 = 10 false discoveries
  5. 9,000 × 0.001 = 9 false discoveries (correct answer)
Explanation: When you encounter multiple testing scenarios in genomics, you're dealing with the multiple comparisons problem and need to calculate the False Discovery Rate (FDR). This requires understanding how many false positives occur when testing thousands of hypotheses simultaneously. Let's work through this systematically. With 10,000 genes and 90% truly not differentially expressed, we have 9,000 true nulls. Using α = 0.001, we expect 9,000×0.001=99,000 \times 0.001 = 9 false positives from these true nulls. The remaining 1,000 genes are truly differentially expressed, and we'd expect to detect most of these as significant (the exact number depends on statistical power, but this doesn't affect our false discovery calculation). Since false discoveries can only come from true null hypotheses, approximately 9 of the 50 significant findings are false discoveries. Answer A correctly calculates this: 10,000×0.90×0.001=910,000 \times 0.90 \times 0.001 = 9. Answer B (50×0.10=550 \times 0.10 = 5) incorrectly assumes 10% of all significant findings are false, ignoring the α level. Answer C (50×0.90=4550 \times 0.90 = 45) wrongly applies the 90% null proportion to the significant results rather than the original gene set. Answer D (10,000×0.001=1010,000 \times 0.001 = 10) calculates total expected false positives if all genes were null, but doesn't account for the fact that only 90% are actually null. Study tip: In multiple testing problems, false discoveries always come from true nulls only. Calculate: (number of true nulls) × (significance level) = expected false discoveries.

Question 12

A clinical trialist compares 4 new treatments against a standard therapy (5 groups total) using ANOVA. She obtains F = 6.8, p = 0.002. For post-hoc analysis, she wants to compare each new treatment to the standard, but her colleague suggests she should compare all possible pairs. If she uses Dunnett's test for her approach and Tukey's HSD for her colleague's approach, both with α = 0.05, which method is more likely to detect significant differences and why?

  1. Tukey's HSD, because it tests more comparisons and has higher overall power
  2. Dunnett's test, because it focuses power on fewer, more relevant comparisons (correct answer)
  3. Both methods have identical power since they both control family-wise error at 0.05
  4. Tukey's HSD, because it uses a less conservative critical value for each comparison
  5. The method with higher power depends on the true pattern of mean differences
Explanation: When you encounter post-hoc testing scenarios in biostatistics, the key principle is that statistical power depends on how you allocate your Type I error budget across comparisons. Different multiple comparison procedures make different trade-offs between controlling false positives and detecting true effects. Dunnett's test is specifically designed for comparing multiple treatments to a single control group. In this scenario, that means 4 comparisons (each new treatment vs. standard). Tukey's HSD performs all possible pairwise comparisons, which would be (52)=10\binom{5}{2} = 10 comparisons total. Both methods control the family-wise error rate at α = 0.05, but they distribute this error budget differently. Since Dunnett's test makes fewer comparisons (4 vs. 10), it can use a less stringent critical value for each individual comparison while still maintaining overall α = 0.05. This gives each comparison more power to detect true differences. Dunnett's test essentially focuses the available statistical power on the clinically relevant comparisons. Option A is incorrect because testing more comparisons actually reduces power per comparison, not increases it. Option C misses the crucial point that while both control family-wise error equally, they achieve different power by testing different numbers of comparisons. Option D has the relationship backwards—Dunnett's test uses the less conservative critical value, not Tukey's. Study tip: Remember that in multiple comparisons, you're always trading off between controlling false positives and maximizing power. Methods that test fewer, more focused comparisons (like Dunnett's) concentrate power where it matters most.

Question 13

A researcher performs 25 independent hypothesis tests and wants to control the expected proportion of false discoveries at 10% using FDR. She observes 8 tests with p < 0.05. After applying the Benjamini-Hochberg procedure, 5 tests are declared significant. What is the expected number of false discoveries among these 5 significant results?

  1. 0.10 × 5 = 0.5 false discoveries (correct answer)
  2. 0.05 × 5 = 0.25 false discoveries
  3. 0.10 × 8 = 0.8 false discoveries
  4. 0.10 × 25 = 2.5 false discoveries
  5. Cannot be determined without knowing which specific tests were significant
Explanation: When you encounter questions about the False Discovery Rate (FDR), focus on what FDR actually controls: the expected proportion of false positives among the discoveries you declare significant, not among all tests performed. The Benjamini-Hochberg procedure controls FDR at your specified level (here, 10%). This means that among the tests you ultimately declare significant, you expect 10% to be false discoveries. Since the researcher declared 5 tests significant after applying the procedure, the expected number of false discoveries is simply 0.10×5=0.50.10 \times 5 = 0.5 false discoveries. Looking at the wrong answers: Answer B uses the original significance level (0.05) instead of the FDR control level (0.10) – this confuses the individual test alpha with the FDR threshold. Answer C multiplies the FDR rate by the 8 tests that had p < 0.05, but FDR only applies to the final set of discoveries declared significant, not to the intermediate step of tests meeting the initial threshold. Answer D multiplies by all 25 tests performed, which would represent a family-wise error approach rather than FDR – this fundamentally misunderstands what FDR controls. The key insight is that FDR is calculated as a proportion of your final significant results, not your total tests or intermediate results. When you see FDR questions, always ask: "Among how many discoveries am I controlling the false discovery rate?" That number gets multiplied by your FDR threshold to give the expected false discoveries.

Question 14

An investigator conducts an experiment with 3 treatment groups and obtains a significant ANOVA result (p = 0.02). She then performs all possible pairwise t-tests and finds: Group 1 vs Group 2 (p = 0.06), Group 1 vs Group 3 (p = 0.03), Group 2 vs Group 3 (p = 0.08). Without multiple comparison correction, how many pairwise differences would be considered significant at α = 0.05, and what does this suggest about the source of the overall ANOVA significance?

  1. One significant comparison; the ANOVA significance is likely due to Group 1 vs Group 3 difference (correct answer)
  2. Two significant comparisons; the ANOVA significance is driven by multiple group differences
  3. No significant comparisons; this creates a paradox requiring further investigation of assumptions
  4. Three significant comparisons; all groups differ significantly from each other
  5. One significant comparison; but this is insufficient to explain the ANOVA result due to inflation
Explanation: When you encounter ANOVA followed by pairwise comparisons, you're examining how an overall significant result breaks down into specific group differences. ANOVA tests whether any groups differ, while pairwise tests identify which groups differ. Looking at the pairwise results without correction, only one comparison reaches significance at α = 0.05: Group 1 vs Group 3 (p = 0.03). Group 1 vs Group 2 (p = 0.06) and Group 2 vs Group 3 (p = 0.08) both exceed the 0.05 threshold. This single significant comparison explains why the overall ANOVA was significant (p = 0.02) – the Group 1 vs Group 3 difference is driving the result. Choice B incorrectly counts two significant comparisons, likely confusing the 0.06 p-value as significant. Choice C suggests no significant comparisons exist, missing that 0.03 < 0.05. This choice also mentions a paradox, but finding one significant pairwise comparison that explains an overall significant ANOVA is perfectly consistent. Choice D claims all three comparisons are significant, which contradicts the given p-values. The pattern here demonstrates how ANOVA can detect overall differences even when only one pairwise comparison drives the significance. The other groups may show trends (like the 0.06 and 0.08 p-values) but don't reach statistical significance individually. Study tip: When interpreting ANOVA follow-up tests, count carefully which p-values fall below your α threshold. A significant ANOVA only requires at least one meaningful group difference – it doesn't mean all groups differ from each other.

Question 15

A clinical trial compares 6 different dosages of a new drug. The researcher plans to use Tukey's HSD for post-hoc comparisons. If the study has 12 subjects per group and MSE = 25 from the ANOVA, what is the minimum difference between two group means that would be detected as statistically significant using Tukey's method with α = 0.05?

  1. q₀.₀₅,₆,₆₆ × √(25/12) = 4.75 × 1.44 = 6.84
  2. q₀.₀₅,₆,₆₆ × √(25/12) = 4.23 × 1.44 = 6.09
  3. q₀.₀₅,₆,₆₆ × √(50/12) = 4.75 × 2.04 = 9.69 (correct answer)
  4. q₀.₀₅,₆,₆₆ × √(50/12) = 4.23 × 2.04 = 8.63
  5. q₀.₀₅,₆,₆₀ × √(25/24) = 4.90 × 1.02 = 5.00
Explanation: When you encounter Tukey's HSD (Honestly Significant Difference) problems, you're dealing with post-hoc analysis after ANOVA to determine which specific group pairs differ significantly. The key is understanding the formula and correctly identifying all components. Tukey's HSD formula is: HSD=qα,k,df×MSEnHSD = q_{\alpha,k,df} \times \sqrt{\frac{MSE}{n}} where q is the studentized range statistic, k is the number of groups, df is the degrees of freedom for error, MSE is mean square error, and n is the sample size per group. For this problem: k = 6 groups, n = 12 per group, MSE = 25, and df = 6(12-1) = 66. The critical q value is q0.05,6,66=4.75q_{0.05,6,66} = 4.75. Substituting: HSD=4.75×2512=4.75×1.44=6.84HSD = 4.75 \times \sqrt{\frac{25}{12}} = 4.75 \times 1.44 = 6.84. Wait - this matches option A, but the correct answer is C. Looking more carefully at option C, it uses 5012\sqrt{\frac{50}{12}} instead of 2512\sqrt{\frac{25}{12}}. This suggests the formula should be 2×MSEn\sqrt{\frac{2 \times MSE}{n}} because we're comparing two independent means, requiring the standard error of the difference between means. Option A uses the wrong standard error (missing the factor of 2). Option B uses an incorrect q value (4.23 instead of 4.75). Option D combines both the wrong q value and includes the correct factor of 2. Study tip: Remember that Tukey's HSD compares differences between two means, so you need 2×MSEn\sqrt{\frac{2 \times MSE}{n}} for the standard error component, and always verify your q table lookup matches the number of groups and error degrees of freedom.

Question 16

A clinical researcher compares 4 different treatments and finds the overall ANOVA F-test significant (p = 0.02). She wants to perform post-hoc tests but is unsure whether to use a method that controls family-wise error rate (FWER) or false discovery rate (FDR). Her primary concern is avoiding false positive findings that could lead to inappropriate clinical recommendations. Which approach should she prioritize and why?

  1. FDR control, because it has higher power to detect true treatment differences
  2. FWER control, because it provides stronger protection against any false positive conclusions (correct answer)
  3. FDR control, because it is more appropriate for exploratory clinical research settings
  4. FWER control, because it is required after a significant ANOVA result
  5. Either approach is equally appropriate since the ANOVA was already significant
Explanation: When you encounter post-hoc testing decisions in biostatistics, the key consideration is understanding what type of error control matches your research priorities and consequences of mistakes. The researcher's primary concern is avoiding false positives that could lead to inappropriate clinical recommendations. This directly points to family-wise error rate (FWER) control, which ensures the probability of making any false positive conclusion across all comparisons stays at or below your chosen α level (typically 0.05). FWER provides the strongest protection against false discoveries, making it ideal when the cost of a false positive is high—like recommending an ineffective treatment to patients. Option A is incorrect because while FDR control does have higher power, the researcher explicitly prioritizes avoiding false positives over maximizing power. Option C misses the point—even in exploratory settings, when clinical decisions are at stake, stronger error control is warranted. Option D is simply false; there's no statistical requirement to use FWER control after significant ANOVA results. False Discovery Rate (FDR) control is more liberal, controlling the expected proportion of false discoveries among all rejected hypotheses. It's excellent for exploratory research where you want to cast a wider net and can tolerate some false positives, but that doesn't match this scenario's priorities. Study tip: Remember the trade-off: FWER is conservative but provides stronger protection against false positives, while FDR is more powerful but allows more false discoveries. Choose based on the consequences of being wrong in your specific research context.

Question 17

A researcher compares 6 medical treatments using ANOVA (F = 4.23, p = 0.003) followed by Tukey's HSD for all pairwise comparisons. The Tukey procedure controls family-wise error rate at 0.05. If instead she had used uncorrected t-tests for the same 15 pairwise comparisons, approximately how many false positive results would be expected by chance alone?

  1. 15 × 0.003 ≈ 0.045 false positives on average
  2. 15 × 0.05 = 0.75 false positives on average (correct answer)
  3. 1 - (0.95)^15 ≈ 0.537 false positives on average
  4. 6 × 0.05 = 0.30 false positives on average
Explanation: With 15 uncorrected tests at α = 0.05 each, the expected number of false positives is 15 × 0.05 = 0.75. Choice A incorrectly uses the ANOVA p-value instead of 0.05. Choice C calculates the probability of at least one false positive, not the expected number. Choice D incorrectly uses 6 (number of groups) instead of 15 (number of comparisons).

Question 18

Two researchers analyze the same dataset comparing 5 treatment groups. Researcher A performs 10 pairwise t-tests and finds 4 significant at α = 0.05. Researcher B performs the same 10 tests but uses α = 0.005 (Bonferroni-corrected) and finds 1 significant. Which statement best characterizes the relationship between their findings?

  1. Researcher A's significant result that remains in Researcher B's analysis represents the strongest evidence against null hypotheses
  2. Researcher B's approach is more conservative and controls Type II error better than Researcher A's method
  3. Both researchers have the same family-wise error rate since they analyzed identical data with identical tests
  4. Researcher A has higher statistical power but also higher family-wise error rate than Researcher B (correct answer)
Explanation: When you encounter multiple comparison problems, focus on the fundamental tradeoff between Type I error control and statistical power. This question tests your understanding of how different alpha levels affect both family-wise error rates and the ability to detect true effects. Researcher A uses the standard α=0.05α = 0.05 for each test, giving higher power to detect real differences but inflating the probability of false discoveries across all 10 comparisons. With 10 tests at α=0.05α = 0.05, the family-wise error rate approaches 1−(0.95)10≈0.401 - (0.95)^{10} ≈ 0.40, meaning a 40% chance of at least one false positive. Researcher B applies Bonferroni correction (α=0.05/10=0.005α = 0.05/10 = 0.005), which controls the family-wise error rate at 0.05 but reduces power for each individual test. Answer D correctly captures this relationship: Researcher A has higher statistical power (more ability to detect true effects, hence 4 significant results) but also a higher family-wise error rate due to multiple testing without correction. Answer A is wrong because the one significant result in B's analysis isn't necessarily the "strongest" – it's simply the one that survived the more stringent threshold. Answer B incorrectly states that B's method controls Type II error better; Bonferroni correction actually increases Type II error (reduces power). Answer C is fundamentally wrong since different alpha levels create different family-wise error rates even with identical data and tests. Remember: Bonferroni correction trades power for family-wise error control. More conservative corrections mean fewer false positives but also fewer true discoveries.

Question 19

In a clinical trial comparing 4 treatments for depression, researchers plan to conduct 6 pairwise comparisons. They want to maintain an overall Type I error rate of 0.05. Using the Šidák correction, what should be the significance level for each individual comparison?

  1. α = 1 - (0.95)^(1/6) ≈ 0.0085 (correct answer)
  2. α = 0.05/6 ≈ 0.0083
  3. α = 1 - (0.05)^(1/6) ≈ 0.481
  4. α = (0.95)^6 ≈ 0.735
Explanation: The Šidák correction uses α_individual = 1 - (1 - α_family)^(1/k) = 1 - (0.95)^(1/6) ≈ 0.0085. Choice B shows the Bonferroni correction. Choice C incorrectly uses (0.05)^(1/6) instead of (0.95)^(1/6). Choice D calculates (0.95)^6 which represents the probability of no Type I errors, not the individual α level.

Question 20

A clinical researcher is studying the effectiveness of 4 different pain medications. She conducts an ANOVA and finds a significant overall effect (p = 0.018). She then wants to perform post-hoc pairwise comparisons between all treatment pairs.

If the researcher performs all possible pairwise t-tests with Bonferroni correction to maintain family-wise α = 0.05, and one comparison yields p = 0.035, what should she conclude about this comparison?

  1. The comparison is significant because p = 0.035 < 0.05, and the ANOVA was already significant
  2. The comparison is significant because p = 0.035 < 0.0083 (the Bonferroni-adjusted α level)
  3. The comparison is not significant because p = 0.035 > 0.0083 (the Bonferroni-adjusted α level) (correct answer)
  4. The comparison is not significant because the ANOVA p-value of 0.018 is too close to 0.05
Explanation: With 4 treatments, there are C(4,2) = 6 pairwise comparisons. Bonferroni correction requires α = 0.05/6 ≈ 0.0083 for each test. Since p = 0.035 > 0.0083, this comparison is not significant. Choice A ignores the multiple comparison adjustment. Choice B incorrectly states the inequality direction. Choice D irrelevantly focuses on the ANOVA p-value.