Biostatistics Quiz: Post Hoc Comparisons
19 questions · exam conditions
0:00
Post Hoc ComparisonsQuestion 1 of 19

A researcher analyzes data from 6 treatment groups and calculates 95% confidence intervals for all pairwise mean differences. She notices that 3 intervals exclude zero, 2 intervals barely include zero, and 10 intervals clearly include zero. Before drawing conclusions, what should be her primary concern?

The confidence intervals may not account for multiple testing, potentially leading to inflated Type I error
The large number of intervals including zero suggests insufficient statistical power
Confidence intervals are inappropriate for multiple group comparisons
The intervals that barely include zero indicate potential computational errors
95% confidence level is too liberal for multiple comparisons and should be increased to 99%
← Back to quizzes

Biostatistics Quiz

Biostatistics Quiz: Post Hoc Comparisons

Practice Post Hoc Comparisons in Biostatistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Post Hoc Comparisons, giving you a quick way to practice the rules, question types, and explanations that matter most for Biostatistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A researcher analyzes data from 6 treatment groups and calculates 95% confidence intervals for all pairwise mean differences. She notices that 3 intervals exclude zero, 2 intervals barely include zero, and 10 intervals clearly include zero. Before drawing conclusions, what should be her primary concern?

  1. The confidence intervals may not account for multiple testing, potentially leading to inflated Type I error (correct answer)
  2. The large number of intervals including zero suggests insufficient statistical power
  3. Confidence intervals are inappropriate for multiple group comparisons
  4. The intervals that barely include zero indicate potential computational errors
  5. 95% confidence level is too liberal for multiple comparisons and should be increased to 99%
Explanation: When you encounter multiple group comparisons with many confidence intervals, the critical issue is multiple testing—performing many statistical tests simultaneously inflates your chance of finding false positives purely by chance. With 6 treatment groups, this researcher performed (62)=15\binom{6}{2} = 15 pairwise comparisons. Each 95% confidence interval has a 5% chance of excluding zero due to random variation alone, even if no true difference exists. With 15 comparisons, the probability of at least one false positive becomes 1−(0.95)15=54%1 - (0.95)^{15} = 54\%—more likely than not! The 3 intervals excluding zero could easily be Type I errors rather than real treatment effects. Answer A correctly identifies this multiple testing problem. Without adjustments like Bonferroni correction or controlling the family-wise error rate, conclusions about statistical significance are unreliable. Answer B misinterprets the situation—having many intervals include zero doesn't necessarily indicate low power; it might simply reflect no true differences between most groups. Answer C is factually wrong. Confidence intervals are perfectly valid for multiple group comparisons; the issue is interpreting them without multiple testing corrections. Answer D focuses on a minor computational concern that's unlikely to be the "primary" issue. Intervals "barely including zero" are normal statistical outcomes, not computational errors. Study tip: Whenever you see multiple comparisons in biostatistics, immediately think "multiple testing problem." The more tests performed, the higher the chance of false discoveries without proper statistical adjustments.

Question 2

In a dose-response study with 4 treatment levels, a researcher finds a significant linear trend (p = 0.003) but wants to also examine which adjacent dose levels differ significantly. She performs 3 adjacent pairwise comparisons and finds p-values of 0.021, 0.048, and 0.067. What is the most appropriate way to interpret these results?

  1. Since the linear trend is significant, all adjacent comparisons can be considered significant without correction
  2. Apply Bonferroni correction to the 3 adjacent comparisons, making only the first one significant
  3. The significant trend validates the two comparisons with p < 0.05, but not the third
  4. Treat the trend test and adjacent comparisons as separate families of tests with different correction needs (correct answer)
  5. Use Tukey's HSD for all 6 possible pairwise comparisons to be comprehensive
Explanation: When you encounter dose-response studies with multiple statistical tests, the key question is how to handle multiple comparisons while preserving the interpretability of different types of analyses. The correct approach is to treat the trend test and pairwise comparisons as separate families of tests requiring different correction strategies. The linear trend test addresses one specific hypothesis about the overall dose-response relationship, while the adjacent pairwise comparisons form a separate family of three related hypotheses about specific dose differences. Each family should be evaluated independently because they answer fundamentally different research questions. Option A is incorrect because finding a significant trend doesn't automatically validate all pairwise comparisons without correction. The trend test and individual comparisons are testing different hypotheses and both are subject to Type I error inflation. Option B incorrectly assumes you must apply Bonferroni correction to the pairwise comparisons, but this overly conservative approach may not be necessary depending on your research priorities and the exploratory nature of the comparisons following a significant trend. Option C creates an arbitrary rule linking trend significance to p-value thresholds for comparisons. This approach lacks statistical justification and doesn't address the multiple comparison problem systematically. Remember that in dose-response studies, you're often conducting both confirmatory tests (like trend analysis) and exploratory analyses (like pairwise comparisons). These serve different purposes and should be treated as separate analytical families, each with their own appropriate correction methods based on your research objectives and error control priorities.

Question 3

A researcher conducts an ANOVA comparing pain scores across four treatment groups (n=10 per group) and finds F(3,36) = 4.2, p = 0.012. She then performs six pairwise t-tests without any correction and finds that three comparisons have p < 0.05. What is the most appropriate conclusion?

  1. Three treatment pairs are significantly different since their individual p-values are less than 0.05
  2. The significant ANOVA result validates all pairwise comparisons with p < 0.05
  3. Multiple testing inflation likely occurred, so a correction method should be applied before concluding significance (correct answer)
  4. Since the overall ANOVA was significant, no correction is needed for the pairwise tests
  5. The results are invalid because post-hoc tests require equal sample sizes across groups
Explanation: When you encounter ANOVA followed by multiple pairwise comparisons, you're dealing with the multiple testing problem - one of the most important concepts in biostatistics. The key issue is that performing many statistical tests simultaneously inflates your chance of finding false positives. The researcher's approach creates a classic multiple testing scenario. With 4 groups, there are (42)=6\binom{4}{2} = 6 possible pairwise comparisons. Even if no true differences exist, performing 6 tests at α = 0.05 gives approximately a 26% chance of finding at least one "significant" result by pure chance. This is why option C is correct - multiple testing inflation likely occurred, and a correction method (like Bonferroni, Holm, or FDR) should be applied before concluding significance. Option A incorrectly assumes that individual p-values retain their nominal significance level when multiple tests are performed. Option B misunderstands the relationship between ANOVA and post-hoc tests - a significant F-test tells you differences exist somewhere, but doesn't validate specific pairwise comparisons without proper correction. Option D perpetuates the dangerous misconception that a significant omnibus test eliminates the need for multiple testing corrections in post-hoc analyses. Remember this pattern: whenever you see multiple comparisons without correction methods mentioned, suspect inflated Type I error rates. On biostatistics exams, questions combining ANOVA with multiple pairwise tests almost always test your understanding of multiple testing problems. Always consider whether appropriate corrections were applied before accepting "significant" results from multiple comparisons.

Question 4

A researcher comparing three groups finds F(2,27) = 6.8, p = 0.004. Using Bonferroni correction for post-hoc tests, she obtains the following pairwise p-values: Group 1 vs 2: p = 0.018; Group 1 vs 3: p = 0.032; Group 2 vs 3: p = 0.089. How many pairwise comparisons would be considered statistically significant?

  1. Three comparisons, since the overall ANOVA was significant at p = 0.004
  2. Two comparisons, since Groups 1 vs 2 and Groups 1 vs 3 have p < 0.05
  3. One comparison, since only Groups 1 vs 2 survives Bonferroni correction
  4. Zero comparisons, since none of the p-values are less than 0.017 (correct answer)
  5. Cannot determine without knowing the exact sample sizes in each group
Explanation: When you see ANOVA followed by post-hoc testing, you need to understand how multiple comparison corrections work. The Bonferroni correction adjusts your significance threshold to control family-wise error rate when making multiple comparisons. With three groups, you make three pairwise comparisons. The Bonferroni correction divides your original alpha level (typically 0.05) by the number of comparisons: αcorrected=0.05/3=0.017\alpha_{corrected} = 0.05/3 = 0.017. This means each individual comparison must have p<0.017p < 0.017 to be considered significant, not the usual p<0.05p < 0.05. Looking at the pairwise p-values: Group 1 vs 2 (p = 0.018), Group 1 vs 3 (p = 0.032), and Group 2 vs 3 (p = 0.089). None of these are below 0.017, so zero comparisons are statistically significant after correction. Answer A incorrectly assumes that a significant overall ANOVA automatically makes all pairwise comparisons significant—this ignores the multiple testing problem entirely. Answer B uses the uncorrected alpha of 0.05, missing the point of Bonferroni correction. Answer C incorrectly identifies Group 1 vs 2 as significant when p = 0.018 > 0.017. Answer D correctly recognizes that none of the p-values meet the Bonferroni-corrected threshold of 0.017. Study tip: Always calculate the corrected alpha when you see Bonferroni mentioned: divide 0.05 by the number of comparisons. Don't be fooled by p-values that would be significant without correction—the adjusted threshold is what matters for determining significance.

Question 5

When comparing the performance of Bonferroni vs. Holm's step-down method for post-hoc comparisons, which statement most accurately describes their relative properties?

  1. Bonferroni is more powerful but provides weaker control of familywise error rate than Holm's method
  2. Holm's method is more powerful while maintaining the same strong control of familywise error rate as Bonferroni (correct answer)
  3. Both methods have identical power, but Holm's method is computationally simpler to implement
  4. Bonferroni should be used when sample sizes are unequal, while Holm's method requires equal group sizes
  5. Holm's method controls comparisonwise error rate, while Bonferroni controls familywise error rate
Explanation: When you encounter questions about multiple comparison procedures, focus on two key properties: power (ability to detect true differences) and familywise error rate control (keeping overall Type I error at the desired level). Both Bonferroni and Holm's step-down method provide strong familywise error rate control at level α\alpha, meaning the probability of making any Type I error across all comparisons is ≤ α\alpha. However, they differ in power. Holm's method is uniformly more powerful than Bonferroni because it uses a step-down approach that becomes less conservative as you proceed through the ordered p-values. While Bonferroni applies the same strict threshold (α/m\alpha/m) to all comparisons, Holm's method uses increasingly lenient thresholds for larger p-values, allowing it to detect more true differences while maintaining the same error control. Looking at the wrong answers: Choice A incorrectly states that Bonferroni is more powerful (it's actually less powerful) and wrongly claims it provides weaker error control (both methods have identical strong control). Choice C is wrong because the methods definitely don't have identical power—Holm's superior power is its main advantage. Choice D mischaracterizes both methods entirely, as neither has requirements about equal group sizes, and the choice between them isn't based on sample size considerations. Remember this hierarchy: Holm's method dominates Bonferroni because it's more powerful with identical error control. When choosing between multiple comparison procedures, always prefer methods that give you more power without sacrificing error control.

Question 6

A researcher performs ANOVA on 5 groups and finds F(4,45) = 3.2, p = 0.021. She then conducts all pairwise comparisons using Fisher's LSD method. What is the primary limitation of this approach?

  1. Fisher's LSD requires larger sample sizes to maintain adequate power for detection
  2. The method doesn't account for the multiple testing problem, inflating familywise Type I error rate (correct answer)
  3. Fisher's LSD can only be used when the overall ANOVA result is non-significant
  4. The approach assumes equal variances, which may be violated with five different groups
  5. Fisher's LSD is inappropriate when there are more than four groups being compared
Explanation: When you encounter ANOVA followed by post-hoc comparisons, the key issue is controlling Type I error across multiple tests. With 5 groups, there are (52)=10\binom{5}{2} = 10 possible pairwise comparisons, each with its own chance of false positive results. Fisher's LSD (Least Significant Difference) method performs each pairwise comparison at the nominal alpha level (typically 0.05), treating them as independent tests. However, when you conduct multiple comparisons, the probability of making at least one Type I error across all tests—called the familywise error rate—becomes much larger than 0.05. With 10 comparisons at α = 0.05 each, the familywise error rate approaches 40%, meaning you have a substantial chance of incorrectly declaring at least one non-existent difference as significant. Answer B correctly identifies this multiple testing problem as Fisher's LSD's primary limitation. The method inflates your overall Type I error rate because it doesn't adjust for multiple comparisons. Answer A is incorrect because sample size requirements aren't Fisher's LSD's main limitation—it's the error rate inflation. Answer C gets the logic backward: Fisher's LSD is typically used only after a significant ANOVA result, not when it's non-significant. Answer D misses the point entirely—while equal variances are an ANOVA assumption, this isn't specific to Fisher's LSD or its primary limitation. Study tip: Remember that whenever you see multiple comparisons mentioned, think "Type I error inflation." Methods like Bonferroni or Tukey's HSD address this problem by adjusting significance levels.

Question 7

A study compares mean reaction times across three age groups (young, middle, elderly). The researcher hypothesizes that elderly participants will be slower than both young and middle-aged groups, but makes no prediction about young vs. middle-aged. What post-hoc approach would be most appropriate?

  1. Perform all three pairwise comparisons using Tukey's HSD to be comprehensive
  2. Use planned contrasts for the two elderly comparisons without correction, and apply Bonferroni to all three
  3. Conduct only the two planned elderly comparisons with Bonferroni correction (α/2)
  4. Test the two planned comparisons involving elderly participants without multiple testing correction (correct answer)
  5. Use Dunnett's test with elderly group as the control condition
Explanation: When you encounter questions about post-hoc testing, the key principle is matching your statistical approach to your research hypothesis. The distinction between planned comparisons (based on specific hypotheses) and exploratory comparisons (fishing expeditions) determines whether you need multiple testing corrections. The correct answer is D because the researcher has two specific, theory-driven hypotheses about elderly participants being slower than both other groups. These are planned comparisons, not exploratory data mining. When you have a small number of planned comparisons based on genuine a priori hypotheses, no multiple testing correction is needed because you're not inflating Type I error through extensive searching. Option A is wrong because Tukey's HSD is designed for all possible pairwise comparisons when you have no specific hypotheses. Here, the researcher has no prediction about young vs. middle-aged, making this unnecessarily conservative and reducing power for the comparisons of actual interest. Option B incorrectly mixes planned and unplanned approaches. You wouldn't apply Bonferroni to all three comparisons when only two are theoretically motivated—this defeats the purpose of having planned comparisons. Option C applies unnecessary correction to planned comparisons. The Bonferroni correction (α/2) is overly conservative here because the researcher isn't conducting a fishing expedition but testing specific, predetermined hypotheses. Remember: planned comparisons based on clear theoretical predictions don't require multiple testing corrections, even when you have more than one. The correction is only needed when you're exploring multiple possibilities without strong prior hypotheses.

Question 8

In a dose-response study with 4 dosage levels (0, 10, 20, 30 mg), a researcher wants to test for linear trend while also examining all pairwise differences. The ANOVA yields F(3,36) = 5.1, p = 0.005. What is the most statistically sound approach?

  1. Test the linear trend contrast first; if significant, no further corrections needed for pairwise tests
  2. Combine the trend test with 6 pairwise tests and apply Bonferroni correction to all 7 tests
  3. Use orthogonal polynomial contrasts for trend, then apply Tukey's HSD for pairwise comparisons (correct answer)
  4. Perform the trend test without correction, then use Fisher's LSD for pairwise comparisons
  5. Apply Holm's method to the trend test and all pairwise comparisons together
Explanation: When analyzing dose-response relationships, you need to balance two distinct statistical goals: testing for systematic trends (like linear dose effects) and examining specific group differences. This requires understanding when different correction methods apply and how they interact. The most rigorous approach uses orthogonal polynomial contrasts for trend analysis combined with Tukey's HSD for pairwise comparisons. Orthogonal contrasts are statistically independent, meaning the linear trend test doesn't inflate your Type I error rate for subsequent analyses. Since these contrasts use all available information efficiently and don't overlap with pairwise comparisons, you can perform the trend test without penalty. Tukey's HSD then provides the most appropriate correction for multiple pairwise comparisons, controlling family-wise error rate across all six possible pairs. Option A incorrectly assumes that a significant trend test eliminates the need for corrections in pairwise testing—these address different questions and both require proper error control. Option B overcorrects by applying Bonferroni to both trend and pairwise tests together, when orthogonal contrasts don't require this combined correction. This approach is overly conservative and reduces power unnecessarily. Option D pairs an uncorrected trend test with Fisher's LSD, which provides no protection against Type I error inflation in the pairwise comparisons. Remember this pattern: when you see dose-response studies asking about both trend and pairwise analyses, look for orthogonal polynomial contrasts plus Tukey's HSD. This combination maximizes statistical power while maintaining appropriate error control for each type of comparison.

Question 9

A researcher comparing 4 treatment groups finds that Tukey's HSD identifies 2 significant pairwise differences, while Bonferroni correction identifies 3 significant differences from the same dataset. What is the most likely explanation?

  1. Tukey's method is more conservative than Bonferroni for this number of comparisons (correct answer)
  2. An error occurred since Tukey's HSD should always find more significant differences than Bonferroni
  3. Bonferroni is more powerful than Tukey's method when there are exactly 4 groups
  4. The datasets must be different since both methods should yield identical results
  5. Tukey's method requires larger effect sizes due to its step-down procedure
Explanation: When you encounter questions about multiple comparison procedures, focus on understanding the relative conservativeness of different methods. The key insight is that more conservative methods have higher thresholds for significance, making them less likely to detect differences. Tukey's HSD (Honestly Significant Difference) and Bonferroni correction both control family-wise error rate, but they differ in conservativeness depending on the number of comparisons. With 4 groups, you're making (42)=6\binom{4}{2} = 6 pairwise comparisons. For this number of comparisons, Tukey's method becomes more conservative than Bonferroni, requiring larger differences to reach significance. This explains why Tukey found only 2 significant differences while Bonferroni found 3. Choice A correctly identifies that Tukey's method is more conservative than Bonferroni for this scenario. Choice B is fundamentally wrong – there's no rule that Tukey's HSD should always find more differences than Bonferroni. In fact, the relative performance depends on the number of groups being compared. Choice C incorrectly suggests Bonferroni is "more powerful" when it's actually less conservative (lower threshold), but this isn't about statistical power in the technical sense. Choice D is incorrect because these methods can and often do yield different results from the same data – they use different approaches to control Type I error. Study tip: Remember that conservativeness in multiple comparisons is context-dependent. Tukey's HSD becomes increasingly conservative relative to Bonferroni as the number of groups increases beyond 3, making it the more stringent test for larger group comparisons.

Question 10

A pharmaceutical study compares a new drug at three doses (low, medium, high) against a placebo control. The researcher plans to use Dunnett's test for post-hoc comparisons. What is the primary advantage of this choice over Tukey's HSD?

  1. Dunnett's test provides stronger control of familywise error rate than Tukey's HSD
  2. Dunnett's test is more powerful because it only compares treatments to control, not all pairwise comparisons (correct answer)
  3. Dunnett's test can handle unequal sample sizes better than Tukey's HSD
  4. Dunnett's test requires fewer assumptions about normality and equal variances
  5. Dunnett's test is computationally simpler and provides exact p-values
Explanation: When you encounter post-hoc testing scenarios in biostatistics, the key is matching the test to your specific comparison strategy. This question tests your understanding of when different multiple comparison procedures are most appropriate. Dunnett's test is specifically designed for comparing multiple treatment groups to a single control group, which is exactly what this pharmaceutical study requires. By limiting comparisons to only treatment-versus-control (3 comparisons: low vs. placebo, medium vs. placebo, high vs. placebo), Dunnett's test maintains statistical power while controlling Type I error. This focused approach makes it more powerful than tests that account for all possible pairwise comparisons. Looking at the wrong answers: Choice A is incorrect because both Dunnett's and Tukey's tests control familywise error rate at the same level (typically α = 0.05) - neither is "stronger" in this regard. Choice C misrepresents the tests' capabilities; both can handle unequal sample sizes, though they use different computational approaches. Choice D is false because both tests operate under the same ANOVA assumptions of normality and homogeneity of variance - Dunnett's test doesn't relax these requirements. The correct answer is B because Dunnett's test gains power by being more selective. While Tukey's HSD would make 6 comparisons (all possible pairs among 4 groups), Dunnett's makes only 3 meaningful comparisons for this research question. Study tip: Remember the comparison pattern: Dunnett's for many-to-one comparisons (treatments vs. control), Tukey's for all possible pairwise comparisons. Match your test to your research question for optimal power.

Question 11

A nutrition study compares 4 dietary interventions with n=12 per group. The researcher conducts 6 pairwise t-tests and finds 2 comparisons with p < 0.01, 1 comparison with p < 0.05, and 3 comparisons with p > 0.05. If she applies the False Discovery Rate (FDR) approach with q = 0.05, what is the primary difference from traditional familywise error rate control?

  1. FDR controls the expected proportion of false discoveries among rejected hypotheses rather than probability of any false discovery (correct answer)
  2. FDR provides stronger control by guaranteeing no false discoveries will occur
  3. FDR is identical to familywise error rate control but uses different computational methods
  4. FDR only applies when the number of comparisons exceeds 10
  5. FDR controls per-comparison error rate rather than familywise error rate
Explanation: When you encounter multiple testing scenarios in biostatistics, you need to understand how different correction methods handle the accumulation of Type I error across comparisons. This question tests your knowledge of False Discovery Rate (FDR) versus traditional familywise error rate (FWER) control methods. The key distinction lies in what each method controls. Traditional FWER methods like Bonferroni control the probability of making any false discovery across all tests - they aim to keep the chance of even one false positive below your alpha level. FDR takes a different approach: it controls the expected proportion of false discoveries among the hypotheses you actually reject. If you reject 3 hypotheses and 1 is falsely rejected, FDR controls that ratio (1/3) rather than preventing that single false discovery entirely. Choice A correctly captures this fundamental difference - FDR controls the expected proportion of false discoveries among rejected hypotheses, not the probability of any false discovery occurring. Choice B is wrong because FDR actually provides less stringent control than FWER methods - it allows some false discoveries while controlling their proportion. Choice C misses the conceptual difference entirely - these methods have fundamentally different goals, not just different calculations. Choice D incorrectly suggests FDR has arbitrary numerical limitations when it actually applies regardless of the number of comparisons. Remember this distinction: FWER methods are conservative and prevent false discoveries, while FDR methods are more liberal and manage the rate of false discoveries among your significant findings. This makes FDR more powerful for detecting true effects in exploratory research.

Question 12

In a behavioral study with 3 groups, a researcher performs all pairwise comparisons and reports: "Group A vs B: t = 2.8, p = 0.008; Group A vs C: t = 1.9, p = 0.067; Group B vs C: t = 3.2, p = 0.003." She concludes that Groups A and B differ significantly from each other, and Groups B and C differ significantly from each other, but Groups A and C do not differ. What error has she likely made?

  1. She failed to check the assumption of equal variances before conducting t-tests
  2. She interpreted the results without applying appropriate correction for multiple testing (correct answer)
  3. She used the wrong degrees of freedom for the t-tests
  4. She should have used one-tailed tests instead of two-tailed tests
  5. She failed to verify that the overall ANOVA was significant before post-hoc testing
Explanation: When you encounter multiple comparisons in biostatistics, you're dealing with a fundamental statistical principle: the more tests you perform, the higher your chance of finding false positives. This researcher conducted three pairwise t-tests but treated each p-value as if it were from a single, isolated test. The correct answer is B because she failed to adjust for multiple testing. When performing multiple comparisons, your family-wise error rate (the probability of making at least one Type I error) increases dramatically. With three comparisons at α = 0.05, her actual error rate is approximately 1 - (0.95)³ = 0.143, nearly triple the intended 5%. She should have applied corrections like Bonferroni (dividing α by number of tests) or used post-hoc tests designed for multiple comparisons. Option A is incorrect because unequal variances would affect the validity of individual t-tests, but her interpretation error stems from multiple testing, not variance assumptions. Option C is wrong because incorrect degrees of freedom would affect the p-values themselves, not the interpretation of multiple p-values. Option D misses the point entirely—the choice between one-tailed and two-tailed tests doesn't address the multiple comparisons problem. Study tip: Whenever you see multiple statistical tests in one study, immediately think "multiple comparisons problem." Look for whether appropriate corrections were applied. This is a frequent exam trap—researchers often forget that conducting several tests inflates their Type I error rate beyond their stated significance level.

Question 13

A researcher studying memory performance across 4 age groups wants to test the specific hypothesis that performance declines monotonically with age. After finding a significant ANOVA, what post-hoc approach would most directly address this research question?

  1. Conduct all 6 pairwise comparisons using Tukey's HSD to examine every possible difference
  2. Use a planned linear contrast to test for monotonic trend, supplemented by Tukey's HSD for pairwise comparisons
  3. Apply Dunnett's test using the youngest group as the control condition
  4. Test the specific hypothesis using a planned linear contrast without additional correction (correct answer)
  5. Use orthogonal polynomial contrasts to test linear, quadratic, and cubic trends simultaneously
Explanation: When you encounter a research question testing a specific directional hypothesis after ANOVA, the key decision is whether you need exploratory comparisons or a targeted test of your predetermined hypothesis. Answer D is correct because the researcher has a specific, theory-driven hypothesis about monotonic decline with age. A planned linear contrast directly tests whether the pattern of means follows a linear decreasing trend across the four age groups. Since this was the researcher's original hypothesis (not a post-hoc exploration), no multiple comparison correction is needed—the Type I error rate is already controlled at α = 0.05 for this single planned test. Answer A is wrong because Tukey's HSD tests all possible pairwise differences but doesn't directly address whether there's a monotonic trend. You could have significant pairwise differences without a monotonic pattern, or have a monotonic trend without all adjacent pairs being significantly different. Answer B is wrong because it unnecessarily combines the correct approach (linear contrast) with additional testing. The Tukey's HSD component would require multiple comparison corrections and doesn't add value when testing the specific monotonic hypothesis. Answer C is wrong because Dunnett's test compares all other groups to a single control group (youngest), but this doesn't test for monotonic decline—it only tells you which groups differ from the youngest group, not whether there's a systematic linear trend. Study tip: When researchers state a specific directional hypothesis before collecting data, planned contrasts test that hypothesis directly without needing post-hoc correction procedures designed for exploratory analyses.

Question 14

When deciding between Tukey's HSD and the Bonferroni correction for post-hoc comparisons, which factor most strongly favors choosing Tukey's method?

  1. Unequal sample sizes across groups make Tukey's method more robust
  2. A large number of groups increases Tukey's power advantage over Bonferroni (correct answer)
  3. Tukey's method provides exact control of familywise error rate while Bonferroni only provides approximate control
  4. Tukey's method is less sensitive to violations of normality assumption
  5. Tukey's method can be used with planned comparisons while Bonferroni cannot
Explanation: When you encounter questions about choosing between post-hoc correction methods, focus on their fundamental trade-offs: power versus error control, and how these change with study design parameters. Tukey's HSD and Bonferroni correction both control familywise error rate, but they make different assumptions and have different power characteristics. The key insight is that as the number of groups increases, the number of pairwise comparisons grows exponentially (specifically, k(k−1)2\frac{k(k-1)}{2} comparisons for k groups). Bonferroni divides your alpha level equally among all comparisons, so with many groups, each individual comparison gets a very small alpha, drastically reducing power. Tukey's method, designed specifically for all pairwise comparisons, maintains better power as the number of groups increases because it accounts for the correlation structure between comparisons rather than treating them as independent. Option A is incorrect because Tukey's method actually assumes equal sample sizes and equal variances - unequal sample sizes favor Bonferroni or other methods. Option C reverses the truth: Bonferroni provides exact control of familywise error rate (it's mathematically conservative), while Tukey's provides approximate control based on the studentized range distribution. Option D is wrong because both methods have similar sensitivity to normality violations since they're both based on the same underlying ANOVA assumptions. Remember this pattern: When you see "large number of groups" in post-hoc comparison questions, think about how the correction method scales with increasing comparisons - this often determines which method is most appropriate.

Question 15

An educational researcher tests 5 teaching methods and obtains the following confidence intervals for pairwise mean differences using Tukey's HSD: Method A vs B: (-2.1, 5.3), Method A vs C: (1.2, 8.8), Method A vs D: (-0.5, 7.1), Method A vs E: (2.3, 9.9). Which comparisons suggest statistically significant differences?

  1. All four comparisons since they come from a significant overall ANOVA
  2. Methods A vs C and A vs E, since their confidence intervals don't include zero (correct answer)
  3. Only Method A vs E, since it has the largest lower bound
  4. Methods A vs B and A vs D, since their intervals include negative values
  5. Cannot determine without knowing the exact p-values for each comparison
Explanation: When you encounter Tukey's HSD confidence intervals for pairwise comparisons, you're looking at post-hoc analysis following ANOVA. The key principle is straightforward: if a confidence interval for the difference between two means includes zero, there's no statistically significant difference between those groups. If zero falls outside the interval, the difference is significant. Let's examine each interval systematically. For Method A vs B: (-2.1, 5.3), zero falls within this range, indicating no significant difference. For Method A vs C: (1.2, 8.8), the entire interval is positive and doesn't include zero, showing a significant difference. Method A vs D: (-0.5, 7.1) includes zero, so no significance. Finally, Method A vs E: (2.3, 9.9) is entirely positive, excluding zero, indicating significance. Answer A is wrong because statistical significance depends on individual confidence intervals, not just overall ANOVA results. Even with a significant F-test, specific pairwise comparisons may not reach significance. Answer C incorrectly focuses on effect size rather than significance—both A vs C and A vs E are significant regardless of which has the larger lower bound. Answer D misunderstands the interpretation entirely; intervals including negative values can still contain zero, which means no significance. Remember this rule: zero inside the confidence interval means no significant difference, zero outside means significant difference. This applies to any confidence interval for a difference between groups, making it a crucial concept for interpreting comparative studies.

Question 16

In a psychological experiment with 6 treatment conditions, a researcher wants to make specific planned comparisons rather than all pairwise tests. She identifies 4 orthogonal contrasts of theoretical interest. What is the most appropriate approach for controlling Type I error?

  1. Use Bonferroni correction with α/4 since there are 4 planned comparisons
  2. Apply Tukey's HSD since it's designed for multiple comparisons regardless of planning
  3. No correction is needed since the contrasts are orthogonal and planned a priori (correct answer)
  4. Use Holm's method to control familywise error rate across the 4 contrasts
  5. Apply Bonferroni correction with α/15 since there are C(6,2) = 15 possible comparisons
Explanation: When you encounter questions about multiple comparisons in ANOVA, the key decision point is whether you're making planned contrasts that were specified before seeing the data, or conducting exploratory post-hoc tests. The approach to Type I error control depends critically on this distinction and the mathematical properties of your contrasts. The correct answer is C because orthogonal planned contrasts have special mathematical properties that eliminate the need for multiple comparison corrections. When contrasts are orthogonal (their coefficients sum to zero and are mathematically independent), they partition the treatment variance into independent components. Since these 4 contrasts were planned a priori based on theoretical interest and are orthogonal, each can be tested at the full α level without inflating familywise error rate. Option A incorrectly applies Bonferroni correction, which is overly conservative for orthogonal planned contrasts. While Bonferroni would control Type I error, it unnecessarily reduces power since the orthogonality already provides protection. Option B suggests Tukey's HSD, but this post-hoc test is designed for unplanned pairwise comparisons when you want to test all possible pairs - not for specific planned contrasts. Option D recommends Holm's method, which is another multiple comparison correction appropriate for non-orthogonal planned comparisons, but unnecessary here. Study tip: Remember the hierarchy: orthogonal planned contrasts need no correction → non-orthogonal planned contrasts need modest correction (like Holm) → unplanned post-hoc tests need strong correction (like Tukey). The combination of "planned" + "orthogonal" is your signal that no correction is needed.

Question 17

A clinical trial comparing four drug dosages yields the following ordered p-values for pairwise comparisons: p₁ = 0.008, p₂ = 0.015, p₃ = 0.023, p₄ = 0.031, p₅ = 0.044, p₆ = 0.067. Using Holm's step-down procedure with α = 0.05, which comparisons would be rejected?

  1. All comparisons with p < 0.05, so the first five comparisons would be rejected
  2. Only the first comparison, since 0.008 < 0.05/6 = 0.0083, but 0.015 > 0.05/5 = 0.01 (correct answer)
  3. The first two comparisons, since both meet their respective adjusted criteria in the step-down process
  4. The first three comparisons, since 0.023 < 0.05/4 = 0.0125 in the sequential testing
  5. No comparisons, since the first p-value 0.008 > 0.05/6 = 0.0083
Explanation: When you encounter multiple comparison problems, you're dealing with the challenge of controlling Type I error when performing many statistical tests simultaneously. Holm's step-down procedure is a sequential method that adjusts significance thresholds to maintain overall family-wise error rate. Here's how Holm's procedure works: Order your p-values from smallest to largest, then test each one sequentially against increasingly lenient thresholds. For the smallest p-value, use α/m\alpha/m where m is the total number of comparisons. If it's significant, move to the next p-value using α/(m−1)\alpha/(m-1), and so on. Stop when you encounter the first non-significant result. With our data: Start with p1=0.008p_1 = 0.008 versus 0.05/6=0.00830.05/6 = 0.0083. Since 0.008<0.00830.008 < 0.0083, this comparison is significant. Next, test p2=0.015p_2 = 0.015 versus 0.05/5=0.010.05/5 = 0.01. Since 0.015>0.010.015 > 0.01, this fails the test, so we stop here and reject only the first comparison. Answer A incorrectly applies simple multiple comparisons without adjustment. Answer C mistakenly continues the procedure after the stopping point. Answer D makes the same error as C, continuing even further despite the procedure requiring you to stop at the first non-significant result. Remember the key principle: Holm's procedure is "step-down" because once you fail to reject a hypothesis, you must stop the entire process. This conservative approach protects against inflated Type I error while maintaining reasonable power.

Question 18

In a study comparing five dietary interventions, a researcher plans to use Tukey's HSD for post-hoc comparisons. With α = 0.05 and 40 total subjects (n=8 per group), what familywise error rate is Tukey's method designed to control?

  1. 0.05 for each individual comparison, resulting in 0.50 overall familywise error rate
  2. 0.05 divided by 10 comparisons, resulting in 0.005 per comparison
  3. 0.05 for the entire family of comparisons, regardless of the number of comparisons (correct answer)
  4. 0.01 for the entire family since there are 10 possible comparisons
  5. 0.10 to account for the increased sample size across multiple groups
Explanation: When you encounter post-hoc testing questions, focus on understanding what "familywise error rate" means and how different methods control it. The familywise error rate is the probability of making at least one Type I error across all comparisons in a family of tests. Tukey's HSD (Honestly Significant Difference) is specifically designed to maintain the familywise error rate at exactly α = 0.05, regardless of how many pairwise comparisons you're making. With five groups, you have (52)=10\binom{5}{2} = 10 possible pairwise comparisons, but Tukey's method ensures that your overall chance of falsely rejecting at least one true null hypothesis remains at 5%. Answer A reflects a common misconception about multiple comparisons. If you performed 10 independent tests each at α = 0.05, your familywise error rate would indeed approach 0.40 (not 0.50), but Tukey's method prevents this inflation. Answer B describes the Bonferroni correction, which divides α by the number of comparisons (0.05/10 = 0.005 per comparison). While Bonferroni also controls familywise error at 0.05, it uses a different approach than Tukey's method. Answer D incorrectly suggests that having more comparisons changes the target familywise error rate to 0.01, which isn't how any standard post-hoc method works. Remember this key distinction: Tukey's HSD maintains your chosen α level (0.05) as the familywise error rate, while methods like Bonferroni achieve this by making individual comparisons more stringent. Always ask yourself whether the method controls overall or individual error rates.

Question 19

A researcher comparing 4 cognitive training programs conducts post-hoc analysis and reports: "Using α = 0.05 with appropriate correction for multiple testing, Programs A and B showed significantly different outcomes (p < 0.001), while all other pairwise comparisons were non-significant." If she used Bonferroni correction, what can be inferred about the uncorrected p-values?

  1. The A vs B comparison had uncorrected p < 0.001, and all others had uncorrected p > 0.05
  2. The A vs B comparison had uncorrected p < 0.001, and all others had uncorrected p > 0.0083
  3. All comparisons except A vs B had uncorrected p-values between 0.0083 and 0.05
  4. The A vs B comparison had uncorrected p < 0.001, while others could have uncorrected p-values anywhere above 0.0083 (correct answer)
  5. Cannot determine the uncorrected p-values from the given information
Explanation: When analyzing multiple comparisons with Bonferroni correction, you need to understand how the correction adjusts significance thresholds. With 4 programs, there are (42)=6\binom{4}{2} = 6 possible pairwise comparisons. The Bonferroni correction divides the original α by the number of comparisons: 0.056=0.0083\frac{0.05}{6} = 0.0083. For a comparison to be significant after Bonferroni correction, its uncorrected p-value must be less than 0.0083. Since the A vs B comparison was significant with corrected p < 0.001, its uncorrected p-value must be less than 0.0083 (and could indeed be as low as the reported < 0.001). For the non-significant comparisons, we only know their uncorrected p-values exceeded 0.0083. However, they could be anywhere above this threshold – whether 0.01, 0.03, 0.08, or even 0.5. The Bonferroni correction doesn't provide information about how far above the corrected threshold these values fall. Answer A is wrong because non-significant comparisons could have uncorrected p-values less than 0.05 (just above 0.0083). Answer B incorrectly assumes all non-significant comparisons had uncorrected p-values exactly at the boundary. Answer C wrongly suggests we can determine an upper bound for the non-significant comparisons – there's no such constraint. Answer D correctly recognizes that while A vs B must have had uncorrected p < 0.0083, the other comparisons could have any uncorrected p-value above 0.0083. Study tip: Remember that multiple comparison corrections only tell you which side of the corrected threshold each p-value falls on, not their exact values or ranges.