Post-Hoc Comparisons - Business Statistics

1. An analyst conducts a study with six groups and obtains a significant ANOVA F-test. They are considering two post-hoc options for pairwise comparisons: Tukey's HSD or Fisher's LSD. Which of the following statements provides the strongest rationale for choosing Tukey's HSD over Fisher's LSD in this situation?
   Answer: Tukey's HSD maintains control of the family-wise error rate across all 15 pairwise comparisons, whereas Fisher's LSD does not. The key distinction between these tests is the control of the family-wise error rate (FWER). For more than three groups, Fisher's LSD does not maintain the FWER at the nominal alpha level (e.g., 0.05). With six groups, there are 15 pairwise comparisons, and the FWER for Fisher's LSD would be considerably higher than 0.05. Tukey's HSD is specifically designed to control the FWER at the specified alpha level for all pairwise comparisons, making it the appropriate choice for maintaining Type I error control.
2. An analyst performed a one-way ANOVA on sales data from five regional offices and found significant differences (p < 0.001). She now wants to identify which specific regions differ. The Bonferroni-corrected significance level should be set at 0.005 to maintain overall α = 0.05. However, she's concerned about statistical power. Which approach would provide the best balance between Type I error control and statistical power for this scenario?
   Answer: Switch to Tukey's HSD since it maintains familywise error rate while being less conservative than Bonferroni Tukey's HSD is specifically designed for pairwise comparisons following ANOVA and maintains the familywise error rate at α = 0.05 while being less conservative (more powerful) than Bonferroni correction. Choice A is overly conservative. Choice C (Fisher's LSD) doesn't adequately control familywise error for multiple comparisons. Choice D incorrectly suggests Scheffé has optimal power - it's actually the most conservative test.
3. A marketing firm runs a one-way ANOVA to compare the mean customer engagement scores of five different advertising campaigns. The result is statistically significant (p < .05). A junior analyst suggests that the firm should now conduct independent samples t-tests for all possible pairs of campaigns to see which specific campaigns differ. Which of the following is the most critical statistical flaw in the analyst's suggestion?
   Answer: The approach inflates the family-wise error rate, increasing the probability of making at least one Type I error. The primary reason not to run multiple t-tests after an ANOVA is the problem of multiple comparisons, which inflates the family-wise error rate (FWER). With five campaigns, there are 10 possible pairwise comparisons. If each is tested at α = 0.05, the probability of incorrectly rejecting at least one null hypothesis (a Type I error) across the 'family' of tests becomes much higher than 0.05. Post-hoc tests are specifically designed to control this FWER.
4. A manager is reviewing a statistical report that compared the productivity of 4 different sales teams. The report states: "A one-way ANOVA was significant (F(3, 76) = 4.5, p = 0.006). Subsequently, all six possible pairwise comparisons were made using uncorrected t-tests at α = 0.05. Two of these comparisons were found to be statistically significant." Based on the methodology described, what is the most pressing concern regarding the report's conclusion?
   Answer: The reported significant differences may be spurious (Type I errors) because the family-wise error rate was not controlled. The most significant flaw in the described analysis is the use of uncorrected t-tests for post-hoc comparisons. This procedure fails to control for the inflation of the family-wise error rate. With six comparisons at α = 0.05, the probability of finding at least one significant result by chance alone is much higher than 5%. Therefore, the two 'significant' findings could easily be Type I errors, and the conclusion is suspect.
5. A supply chain manager analyzes shipping times for four different carriers (A, B, C, D). After a significant ANOVA, she becomes interested in exploring an unplanned, complex comparison: whether the average shipping time for carriers A and B differs from the average for carriers C and D. Which post-hoc test is specifically designed to maintain control over the family-wise error rate for any and all possible linear comparisons, including complex ones like this?
   Answer: The Scheffé test The Scheffé test is the most flexible and also the most conservative post-hoc test. Its key feature is that it controls the family-wise error rate for the set of all possible linear contrasts (or comparisons), not just pairwise comparisons. This includes complex comparisons like comparing the average of one subset of groups to the average of another, making it the appropriate choice for this type of exploratory analysis.
6. An analyst performed ANOVA on sales performance across five regions and found F(4,45) = 4.67, p = 0.003. When conducting Scheffé post-hoc tests, she found that a complex contrast comparing the average of Regions 1 and 2 against the average of Regions 4 and 5 yielded a test statistic of F = 3.84. Given that the critical value for Scheffé's test is F(4,45) × 4 = 18.68, and individual pairwise critical value for Tukey would be 2.86, what should she conclude?
   Answer: The contrast is not significant because F = 3.84 < 18.68, suggesting no difference between these regional groupings When you encounter ANOVA followed by post-hoc tests, you're dealing with multiple comparisons that require adjusted critical values to control for Type I error inflation. The key is understanding which critical value applies to each specific test method. The correct answer is D because Scheffé's method requires comparing the test statistic F = 3.84 against the Scheffé critical value of 18.68. Since 3.84 < 18.68, the contrast is not statistically significant, meaning there's insufficient evidence of a difference between the average of Regions 1&2 versus Regions 4&5. Option A incorrectly applies Tukey's critical value (2.86) to judge a Scheffé test. While 3.84 > 2.86, you must use the appropriate critical value for the method being employed. Tukey's method is designed for pairwise comparisons, not complex contrasts. Option B suggests the result is "marginally significant," but there's no such interpretation in hypothesis testing—results are either significant or not based on the chosen α level and critical value. The suggestion to use a "less conservative" method would compromise the error control that post-hoc tests are designed to provide. Option C is factually wrong. Scheffé's method is specifically designed to handle any linear contrast, including complex ones involving multiple group comparisons. This flexibility is actually Scheffé's main advantage over other post-hoc methods. Study tip: Always match your critical value to your test method. Scheffé uses $F_{critical} = (k-1) \times F_{(k-1,df_{error})}$, while Tukey uses different critical values for pairwise comparisons only.
7. An educational researcher compared reading comprehension scores across six different teaching methods using ANOVA. After finding significant differences, she performed all possible pairwise comparisons using both Fisher's LSD and Tukey's HSD. Fisher's LSD identified 8 significant pairs out of 15 total comparisons, while Tukey's HSD identified only 5 significant pairs. What is the most likely explanation for this discrepancy?
   Answer: Fisher's LSD has higher statistical power but inflated Type I error rate due to lack of familywise error control Fisher's LSD maintains α = 0.05 for each individual comparison but doesn't control familywise error rate, making it more liberal (higher power, more Type I errors) than Tukey's HSD, which controls familywise error. This explains why LSD found more significant differences. Choice B incorrectly suggests Tukey is specifically for unequal n. Choice C incorrectly assumes the differences are errors. Choice D mischaracterizes Fisher's LSD purpose.
8. Based on the results described in the passage, what is the most accurate conclusion?
   Answer: Layout A leads to significantly longer session times than Layout B, and Layout C leads to significantly longer session times than Layout B. In post-hoc tests, a confidence interval for a mean difference that does not contain zero indicates a statistically significant difference. The interval for A-B ([0.5, 4.5]) is entirely positive, so μ_A > μ_B. The interval for B-C ([-5.0, -1.0]) is entirely negative, so μ_B < μ_C (or μ_C > μ_B). The interval for A-C ([-1.2, 3.8]) contains zero, so there is no significant difference between A and C. Therefore, both A and C are significantly better than B.
9. A significant ANOVA result for five groups leads to a post-hoc analysis using Tukey's HSD. The results show that the mean of Group 1 is significantly higher than the means of Groups 3, 4, and 5. However, no other pairwise comparisons (e.g., Group 1 vs 2, Group 3 vs 4) are significant. What is the most rigorous and defensible conclusion?
   Answer: The evidence suggests a difference between the population mean of Group 1 and those of Groups 3, 4, and 5, while for other pairs, a difference cannot be ruled out or confirmed. When you encounter ANOVA followed by post-hoc testing, remember that statistical significance tells you whether differences exist, but non-significance doesn't prove equality—it simply means you lack sufficient evidence to detect a difference. The correct interpretation focuses on what the data actually demonstrate. Tukey's HSD found significant differences between Group 1 and Groups 3, 4, and 5, providing evidence that Group 1's population mean truly differs from these three groups. For all other comparisons (like Group 1 vs. 2, or Group 3 vs. 4), the lack of significance means you cannot conclude either that differences exist or that they don't exist—the evidence is simply insufficient either way. Answer A incorrectly assumes that non-significant results prove the groups are "statistically identical." Non-significance never proves equality; it only indicates insufficient evidence for a difference. Answer B makes an unjustified causal leap—statistical differences don't necessarily imply causation, and the data doesn't support claims about what "caused" Group 1's higher mean. Answer C misunderstands the relationship between ANOVA and post-hoc tests. A significant ANOVA indicates at least one difference exists among the groups, which the post-hoc analysis confirmed (Group 1 vs. Groups 3, 4, 5). Finding fewer significant pairs than possible doesn't invalidate the overall ANOVA. Answer D correctly captures the nuanced interpretation: evidence exists for specific differences (Group 1 vs. Groups 3, 4, 5), while other potential differences remain neither confirmed nor ruled out. Remember: in hypothesis testing, non-significance means "insufficient evidence," not "no difference."
10. A financial analyst compares the mean annual returns of investment portfolios from four different firms. After a significant ANOVA result, the analyst decides to perform post-hoc tests to compare all possible pairs of firms. If both Tukey's HSD and a Bonferroni-corrected set of t-tests are conducted, which statement most accurately describes the likely relationship between their results?
    Answer: Tukey's HSD is specifically optimized for all pairwise comparisons and will likely have more power to detect differences than the Bonferroni method. Tukey's HSD (Honestly Significant Difference) test is specifically designed to control the family-wise error rate for a set of all possible pairwise comparisons. The Bonferroni correction is a general method that can be applied to any set of tests, but it is often overly conservative (i.e., has lower statistical power) when the number of comparisons is large. For all pairwise comparisons following an ANOVA, Tukey's HSD is typically more powerful than Bonferroni, meaning it is more likely to find a true difference if one exists.
11. A market researcher conducted ANOVA comparing customer satisfaction scores across four retail locations and found F(3,76) = 5.12, p = 0.003. She wants to compare each of the three branch stores against the flagship store (control) using an appropriate post-hoc test. The mean satisfaction scores are: Flagship = 7.8, Branch A = 6.9, Branch B = 7.2, Branch C = 8.4. If the pooled standard error for comparisons is 0.35 and she uses α = 0.05, which multiple comparison procedure should she use and why?
    Answer: Dunnett's test because it's specifically designed for multiple comparisons against a single control group When you encounter post-hoc testing scenarios in ANOVA, the key is matching the comparison structure to the appropriate test. Here, the researcher wants to compare three branch stores specifically against one flagship store (control), not make all possible pairwise comparisons between groups. Dunnett's test (C) is specifically designed for this exact scenario - multiple comparisons against a single control group. It controls the familywise error rate while maintaining better statistical power than other methods when your research question focuses on comparing treatments to a control. Since the researcher has a clear control group (flagship store) and wants to test each branch against it, Dunnett's test is the optimal choice. Option A is incorrect because Tukey's HSD is designed for all pairwise comparisons (comparing every group to every other group). While it would work, it's unnecessarily conservative here since you're not interested in comparing Branch A to Branch B, for example. Option B is wrong because the Bonferroni correction, while conservative, isn't specifically designed for control-versus-treatment comparisons. It's a general adjustment that sacrifices power unnecessarily in this structured comparison situation. Option D is incorrect because Fisher's LSD doesn't provide adequate Type I error protection when making multiple comparisons, even after a significant omnibus test. The "protected" LSD approach is generally considered insufficient for controlling familywise error rates. Study tip: Match your post-hoc test to your comparison structure: Tukey's for all pairwise comparisons, Dunnett's for multiple groups versus one control, and consider the balance between Type I error protection and statistical power.
12. Considering the researcher's specific questions and the need to control Type I error appropriately, what would be the most statistically sound approach for these comparisons?
    Answer: Use three planned orthogonal contrasts without multiple comparison adjustment since these are theory-driven hypotheses specified before data collection The three questions represent planned, theory-driven contrasts that appear to be orthogonal (independent). When contrasts are planned a priori and orthogonal, no adjustment for multiple comparisons is needed, maximizing statistical power. Choice B unnecessarily mixes methods. Choice C goes beyond the specific research questions. Choice D applies unnecessary correction to planned orthogonal contrasts.
13. Given the research team's specific planned comparisons described above, which post-hoc approach would be most statistically appropriate and efficient?
    Answer: Conduct the three planned orthogonal contrasts without adjustment since they were specified a priori When contrasts are planned a priori and are orthogonal (independent), no multiple comparison adjustment is needed, providing maximum statistical power. The described comparisons appear to be pre-planned based on research hypotheses. Choice A is unnecessarily conservative for planned comparisons. Choice B applies unnecessary correction to planned contrasts. Choice D (Scheffé) is overly conservative for a small number of planned comparisons.
14. A regulatory agency is reviewing data from a study comparing the levels of a contaminant produced by four different manufacturing processes. A Type I error—falsely concluding that one process is different from another—could lead to costly and unnecessary sanctions. After a significant ANOVA, which post-hoc procedure should the agency use to be the most conservative in declaring any differences?
    Answer: The Scheffé test, because it provides the strongest control over the family-wise error rate for all possible comparisons. The term 'conservative' in statistics refers to a lower willingness to reject the null hypothesis, which means prioritizing the control of Type I errors. The Scheffé test is known to be the most conservative post-hoc test, as it controls the family-wise error rate for an infinite number of possible linear contrasts, not just pairwise comparisons. This makes it the most cautious choice when the consequences of a Type I error are severe, as described in the scenario.
15. Following a significant one-way ANOVA on sales data from four regions, a post-hoc test reveals that mean sales in the North region are 'significantly higher' than in the South region (p = 0.02). What is the correct interpretation of the term 'significantly higher' in this business context?
    Answer: The observed difference in mean sales is unlikely to be a result of random chance if the true regional sales means were actually equal. When you encounter post-hoc test results following ANOVA, you're dealing with statistical significance, which has a very specific meaning that's often misunderstood in business contexts. The key is distinguishing between statistical significance and practical importance. Statistical significance (p = 0.02) tells you about the reliability of your finding, not its magnitude or cause. A p-value of 0.02 means there's only a 2% probability of observing this difference (or larger) if the regional means were actually equal in the population. This makes option A correct—the observed difference is unlikely due to random chance alone. Option B confuses statistical significance with practical significance. A statistically significant difference could actually be quite small in dollar terms and have little strategic value. Statistical tests don't measure business importance. Option C represents a classic overgeneralization error. Statistical significance never guarantees future outcomes—it only tells you about the reliability of your current sample results. Future performance could vary due to countless factors. Option D falls into the causation trap. Post-hoc tests only reveal which groups differ significantly; they provide zero information about why those differences exist. The superior North region performance could stem from demographics, competition, economic conditions, or numerous other factors beyond marketing strategy. Study tip: Remember that "statistically significant" always means "probably not due to chance"—nothing more, nothing less. It doesn't indicate size, importance, future certainty, or causation. Keep this distinction sharp to avoid common ANOVA interpretation errors.
16. A researcher compares four groups and obtains a significant ANOVA F-test (p = 0.02). They proceed with Fisher's LSD post-hoc test. A statistician argues that Tukey's HSD should have been used instead. What is the primary weakness of Fisher's LSD in this scenario that justifies the statistician's argument?
    Answer: Fisher's LSD does not adequately control the family-wise error rate when the number of groups is greater than three. The main criticism of Fisher's LSD procedure is that, while it requires a significant overall F-test (the 'protection' step), it does not control the family-wise error rate (FWER) at the nominal alpha level when there are more than three groups being compared. For four or more groups, the FWER can become substantially inflated. Tukey's HSD, in contrast, is designed to control the FWER for all pairwise comparisons regardless of the number of groups.
17. A pharmaceutical company compares a new drug against three existing drugs. The consequences of a Type I error (incorrectly claiming the new drug is superior) are severe. However, a Type II error (failing to detect a real improvement) would be a major missed opportunity. After a significant ANOVA, the goal is to conduct all pairwise comparisons. Which post-hoc procedure offers the best balance by effectively controlling the family-wise error rate while maintaining reasonable statistical power for this specific goal?
    Answer: Tukey's HSD, as it is designed for all pairwise comparisons and is considered a good compromise between controlling Type I error and retaining power. The scenario requires a balance between controlling Type I error (high priority) and maintaining power (also important). The Scheffé test is often too conservative for simple pairwise comparisons, greatly reducing power. Fisher's LSD and uncorrected t-tests do not adequately control the family-wise error rate. Tukey's HSD is specifically designed for all pairwise comparisons and is widely considered to be the best-balanced procedure for this common scenario, offering strong FWER control without being overly conservative.
18. A quality control manager tested four manufacturing processes using ANOVA and found F(3,28) = 6.45, p = 0.002. Using Dunnett's test to compare three treatment processes against a control process, she found the following mean differences from control: Process A: +2.8, Process B: +1.4, Process C: +3.6. If Dunnett's critical value is 2.50 and the standard error for each comparison is 1.2, which conclusion is most appropriate?
    Answer: Processes A and C show significant improvement since their differences exceed the critical difference of 3.0 Dunnett's test requires calculating critical difference = critical value × standard error = 2.50 × 1.2 = 3.0. Process A (2.8) and Process C (3.6) have differences ≥ 3.0, so they're significant. Choice A incorrectly compares mean differences directly to critical value. Choice C incorrectly assumes ANOVA significance means all comparisons are significant. Choice D miscalculates and Process A actually exceeds the critical difference.
19. A production manager analyzed defect rates across three shifts using ANOVA and found significant differences (p = 0.012). She calculated the following confidence intervals for pairwise differences using Tukey's HSD: Day vs. Evening: [-2.1, -0.3], Evening vs. Night: [-0.8, 1.4], Day vs. Night: [-2.9, -0.1]. Based on these results, which interpretation is correct?
    Answer: Day shift has significantly lower defect rates than both Evening and Night shifts, while Evening and Night shifts don't differ significantly Confidence intervals that don't contain zero indicate significant differences. Day vs. Evening [-2.1, -0.3] and Day vs. Night [-2.9, -0.1] don't contain zero (significant), while Evening vs. Night [-0.8, 1.4] contains zero (not significant). This means Day shift differs from both others. Choice B incorrectly assumes ANOVA significance means all pairwise comparisons are significant. Choice C misinterprets which comparisons are significant. Choice D misreads the pattern of differences.
20. An operations manager performs a one-way ANOVA to compare the mean production times of four different assembly line configurations. The F-test yields a p-value of 0.09 at an alpha level of 0.05. What is the most appropriate next step for the manager to take?
    Answer: Conclude there is insufficient evidence of a difference among the means and do not perform post-hoc tests. A fundamental prerequisite for conducting most post-hoc tests (like Tukey's, Scheffé's, etc.) is a statistically significant overall ANOVA F-test. Since the p-value (0.09) is greater than the alpha level (0.05), the null hypothesis of equal means cannot be rejected. Therefore, there is no statistical justification for proceeding to find which specific means are different. The correct conclusion is that there is insufficient evidence to claim any difference exists.

Start this quiz