- A pharmaceutical company conducts 20 independent hypothesis tests on different drug efficacy measures, each using α=0.05. Exactly 2 tests yield significant results (p < 0.05). Assuming all null hypotheses are actually true, what is the most reasonable interpretation of these findings?
Answer: The significant results are likely Type I errors, as approximately 1 would be expected by chance alone
When you encounter multiple hypothesis testing scenarios, the key concept is understanding how Type I error rates accumulate across independent tests. Each individual test has a 5% chance of producing a false positive when the null hypothesis is true.
With 20 independent tests at α=0.05, you'd expect approximately 20×0.05=1 significant result by chance alone, even when all null hypotheses are true. Observing 2 significant results falls very close to this expected value and is entirely consistent with random variation. This makes option D correct—the significant results are most reasonably explained as Type I errors occurring at roughly the expected rate.
Option A incorrectly assumes that any significant result indicates a true effect, ignoring the multiple testing context where false positives are expected. Option B misinterprets the 90% non-significant rate; this actually supports the null hypotheses being true rather than indicating drug ineffectiveness (the drug could be ineffective AND these could be chance findings). Option C wrongly suggests systematic bias when the pattern of 2 out of 20 significant results is precisely what you'd expect from random chance with independent testing.
The critical insight is that 2 significant results out of 20 tests doesn't deviate meaningfully from the expected 1 false positive—this falls well within normal random variation.
Study tip: In multiple testing questions, always calculate the expected number of false positives (number of tests × α level) first. If observed significant results are close to this number, suspect Type I errors rather than true effects. - In a clinical trial, the primary endpoint shows p = 0.06, while a secondary endpoint shows p = 0.02. Both tests used α=0.05. The researchers conclude that the treatment is effective based on the secondary endpoint. What is the main concern with this approach?
Answer: The researchers are cherry-picking significant results while ignoring the primary outcome measure
When you encounter clinical trial results with multiple endpoints, the key principle is that conclusions should be based on pre-specified primary outcomes, not on hunting for favorable results among secondary measures.
The correct answer is D because this scenario represents a classic case of cherry-picking or "p-hacking." The researchers designed their study with a specific primary endpoint to test their main hypothesis, but when that endpoint failed to reach significance (p = 0.06), they shifted focus to a secondary endpoint that happened to be significant (p = 0.02). This invalidates the study's inferential framework because they're essentially changing their hypothesis after seeing the data.
Let's examine why the other options miss the mark. Option A suggests using stricter alpha levels for secondary endpoints, but the real issue isn't the alpha level—it's that secondary endpoints weren't meant to be the basis for primary efficacy claims. Option B incorrectly states that interpretation is impossible; while the results are mixed, the primary endpoint should take precedence in interpretation. Option C makes an overly broad claim that secondary endpoints are inherently less reliable, when actually they can be quite valid for their intended exploratory purposes.
The fundamental problem is prioritizing convenience over scientific rigor. Secondary endpoints are valuable for generating hypotheses and understanding mechanisms, but using them to rescue a failed primary analysis undermines the entire statistical framework of hypothesis testing.
Study tip: Remember that in clinical trials, the primary endpoint is king. When you see researchers celebrating secondary results while downplaying primary failures, that's a major red flag for biased interpretation.
- A journal editor receives a manuscript reporting p = 0.052 and requests that the authors 'discuss the implications of this near-significant result.' The authors argue that p-values should be reported as exact values without reference to arbitrary thresholds. Who has the better argument?
Answer: The authors are correct because the exact p-value provides more information than threshold-based interpretations
This question tests your understanding of p-value interpretation and the ongoing debate about statistical significance thresholds in biostatistics. When you encounter questions about p-values near 0.05, focus on what information is actually being conveyed versus arbitrary categorizations.
The authors have the stronger argument because exact p-values provide readers with the complete statistical information needed to make their own interpretations. A p-value of 0.052 tells you the precise probability of observing your data (or more extreme) under the null hypothesis. This allows readers to assess the strength of evidence on a continuous scale rather than forcing a binary significant/not-significant decision. The exact value also enables proper meta-analyses and helps readers understand how close the result was to conventional thresholds.
Option A is incorrect because p = 0.052 versus p < 0.05 represents an artificial categorical distinction. The difference between 0.049 and 0.052 is trivial and doesn't represent meaningfully different evidence against the null hypothesis.
Option C seems reasonable but is flawed because "guidance" shouldn't come from arbitrary threshold enforcement. Better guidance comes from teaching readers to interpret the continuous nature of evidence that p-values represent.
Option D incorrectly assumes that maintaining conventions is more important than providing complete information. While conventions have their place, the scientific community increasingly recognizes that rigid adherence to p < 0.05 can be misleading.
Remember: p-values are continuous measures of evidence, not binary indicators of truth. Always report exact values and avoid treating 0.05 as a magical boundary that transforms evidence quality.
- A meta-analysis combines 5 studies, each with p-values of 0.08, 0.12, 0.09, 0.11, and 0.07. None were individually significant at α=0.05. The combined analysis yields p = 0.02. What does this suggest?
Answer: Each study may have had insufficient power to detect a real but small effect
When you encounter meta-analysis questions in biostatistics, focus on the fundamental concept of statistical power—the ability to detect a true effect when it exists. Meta-analysis works by pooling data to increase overall sample size and power.
The correct answer is C because each individual study likely had insufficient power to detect a real but small effect. Notice that all five p-values (0.08, 0.12, 0.09, 0.11, 0.07) cluster just above the α=0.05 threshold, suggesting a consistent but weak signal across studies. When meta-analysis combines these studies, it increases the effective sample size, boosting statistical power and allowing detection of the true underlying effect (p = 0.02).
Answer A is wrong because studies aren't "flawed" simply for lacking power—they may have been appropriately designed for their available resources, but individually underpowered for small effects. Answer B reflects a common misconception; you absolutely can combine non-significant results to achieve significance if there's a consistent underlying effect that individual studies were underpowered to detect. This is precisely why meta-analysis exists. Answer D is incorrect because raising the significance level (making α larger) would make it easier to find significance in individual studies but wouldn't address the fundamental power issue.
Remember this pattern: when you see multiple studies with p-values hovering just above 0.05 that become significant when combined, think "power problem solved by larger sample size," not methodological flaws or statistical errors. - A randomized trial reports the primary outcome as 'not significant (p = 0.18)' but emphasizes a subgroup analysis showing p = 0.01. The authors conclude the treatment is effective in the subgroup. What are the main concerns with this interpretation?
Answer: Subgroup analyses are exploratory and prone to false positives, especially when the overall result is negative
When you encounter a study where the primary outcome is non-significant but the authors emphasize a significant subgroup result, you're looking at a classic statistical interpretation pitfall known as "subgroup hunting" or multiple testing bias.
The correct answer is C because subgroup analyses are inherently exploratory and carry high risk of false positives, especially when the overall trial is negative. Here's why: when you divide your sample into subgroups, you're essentially performing multiple statistical tests. Each test has a 5% chance of showing significance by random chance alone. With multiple subgroups, the probability of finding at least one "significant" result increases dramatically, even when no true effect exists. When the primary analysis is already negative (p = 0.18), emphasizing a positive subgroup finding is particularly suspect.
A is incorrect because subgroup analyses use the same statistical methods as primary analyses - the issue isn't methodological incompatibility. B misunderstands the relationship between overall and subgroup results. A non-significant overall result doesn't contradict a significant subgroup finding - it's actually expected when subgroup hunting occurs. D, while potentially true, misses the main concern. Small sample sizes in subgroups are problematic, but the primary issue is the multiple testing bias, not just power.
Remember this key principle: when the primary endpoint fails, be extremely skeptical of post-hoc subgroup claims. True subgroup effects should be pre-specified, biologically plausible, and ideally replicated in independent studies. This is a high-yield concept for biostatistics exams.
- A researcher conducts a study with n=25 and obtains p = 0.30. She argues that the lack of significance is due to insufficient power and plans to increase the sample size to n=100. What assumption is she making?
Answer: That there is a true effect to detect, which may or may not be a valid assumption
When you encounter questions about statistical power and sample size decisions, focus on the underlying assumptions researchers make about effect existence and detectability.
The researcher's logic reveals a critical assumption: she believes a true effect exists in the population that her initial study failed to detect due to insufficient power. By increasing sample size from 25 to 100, she's operating under the premise that with greater power, she'll be able to detect this presumed true effect. This assumption may or may not be valid—the non-significant result could reflect either insufficient power to detect a real effect or the genuine absence of an effect.
Let's examine why the other options miss the mark. Choice A incorrectly suggests effect size grows with sample size, but effect size is a population parameter that remains constant regardless of your sample size—only your ability to detect it changes. Choice C misunderstands significance levels, which researchers set deliberately (typically at 0.05) and don't automatically adjust based on sample size. Choice D confuses the relationship between sample size and error types—larger samples increase power (reduce Type II error) but don't change the Type I error rate, which is determined by your chosen significance level.
The correct answer is B because it captures the fundamental assumption underlying the researcher's decision: that there's a true effect worth detecting.
Study tip: Remember that increasing sample size only helps if there's actually an effect to find. When you see power analysis questions, always consider whether the researcher is assuming effect existence versus effect absence.
- Two studies test the same research hypothesis. Study A (n = 100) reports p = 0.03, and Study B (n = 1000) reports p = 0.04. Both used α=0.05. What is the most accurate interpretation?
Answer: Study A provides stronger evidence against the null hypothesis based solely on the p-value comparison
When evaluating statistical evidence across studies, you need to consider both statistical significance and effect size magnitude, not just p-values alone. The p-value tells you the probability of observing your results if the null hypothesis were true, but it's heavily influenced by sample size.
Study B provides stronger evidence despite the slightly larger p-value. With n=1000 versus Study A's n=100, Study B detected a significant effect with much greater precision. Large samples make it harder to achieve statistical significance by chance alone, so when you do find significance in a large study, it represents more robust evidence. The effect size in Study B is likely more reliable and generalizable.
Choice A falls into the common trap of p-value worship—assuming smaller p-values always mean stronger evidence. This ignores how sample size affects p-values. Choice C incorrectly treats all statistically significant results as equivalent evidence, missing the crucial role of sample size in determining reliability. Choice D suggests the studies can't be compared, but comparing results across different sample sizes is exactly what meta-analyses and evidence synthesis do routinely.
The key insight is that statistical significance is just the first hurdle. Once you've cleared α=0.05, the strength of evidence depends on factors like sample size, effect size confidence intervals, and study design quality. A barely significant result in a tiny study often represents weaker evidence than a clearly significant result in a large, well-powered study.
Remember: bigger samples generally provide more trustworthy evidence when both studies achieve statistical significance. - A pharmaceutical company tests a new drug using α=0.05 and obtains p = 0.048. The company claims this proves their drug is effective. A regulatory agency argues for using α=0.01 instead. What is the key issue with the company's claim?
Answer: Statistical significance does not prove effectiveness, and the choice of α should be predetermined
This question tests your understanding of hypothesis testing fundamentals and the distinction between statistical significance and practical significance. When you encounter questions about p-values and significance levels, focus on the proper interpretation and methodology rather than just the numerical comparison.
The key issue here involves two critical principles: the predetermined nature of significance levels and the meaning of statistical significance. The significance level (α) must be established before data collection and analysis to avoid bias. When researchers choose α after seeing results, they're essentially moving the goalposts to favor their desired outcome. Additionally, statistical significance only indicates that an observed difference is unlikely due to chance alone—it doesn't prove that a treatment is clinically meaningful, safe, or "effective" in practical terms.
Choice A is incorrect because while p = 0.048 is close to α=0.05, proximity alone doesn't invalidate the statistical test—the real issues are methodological and interpretive. Choice C misses the main point by focusing on which α level is "better" rather than addressing the fundamental flaws in the company's approach. Choice D incorrectly assumes that meeting a conventional significance threshold automatically validates effectiveness claims, ignoring both the predetermined α requirement and the limitation of statistical significance.
Remember this key principle: statistical significance ≠ clinical significance or proof of effectiveness. Always establish your significance level before collecting data, and interpret results within the broader context of clinical relevance, effect size, and practical importance. - A researcher conducts a two-tailed hypothesis test with α=0.05 and obtains a p-value of 0.03. If the same data were analyzed using a one-tailed test in the direction of the observed effect, what can be concluded about the relationship between the new p-value and the significance level?
Answer: The one-tailed p-value would be 0.015, leading to rejection of the null hypothesis at α=0.05
For a two-tailed test with p = 0.03, the one-tailed p-value in the direction of the effect would be 0.03/2 = 0.015. Since 0.015 < 0.05, we reject the null hypothesis. Choice B incorrectly doubles instead of halving. Choice C correctly calculates the p-value but incorrectly states the conclusion would be unchanged (both reject, but for different reasons). Choice D incorrectly assumes the p-value stays the same. - A researcher performs a hypothesis test and obtains p = 0.12. Upon reviewing the analysis, she realizes the significance level should have been set at α=0.15 rather than α=0.05 based on the study context. How does this realization affect the statistical conclusion and the strength of evidence?
Answer: The conclusion changes from non-significant to significant, but the strength of evidence against the null hypothesis remains unchanged
With α = 0.15, the result becomes statistically significant (p = 0.12 < 0.15), changing the conclusion. However, the p-value (0.12) represents the strength of evidence and doesn't change when α changes. Choice A incorrectly suggests evidence strength changes with α. Choice C incorrectly states the conclusion is unchanged. Choice D incorrectly suggests evidence becomes weaker.
- A meta-analysis combines results from five studies. Four studies report p-values of 0.08, 0.09, 0.07, and 0.06 (all non-significant at α=0.05), while the fifth study reports p = 0.02. If each study has equal weight, what conclusion is most appropriate regarding the overall evidence?
Answer: The overall evidence supports the alternative hypothesis because all p-values are relatively small and one achieves significance
All p-values are relatively small (≤0.09), suggesting consistent evidence against H₀, even though most don't reach the arbitrary 0.05 threshold. The pattern shows consistent directional evidence. Choice A overemphasizes the 0.05 threshold. Choice C incorrectly interprets non-significant results as evidence for H₀. Choice D inappropriately suggests excluding the significant result.
- A researcher reports p = 0.08 and states: 'While not statistically significant at α=0.05, this p-value indicates a trend toward significance.' What is the most appropriate evaluation of this interpretation?
Answer: The interpretation is misleading because it treats p-values as if they have inherent meaning beyond the chosen α
When you encounter questions about p-value interpretation, focus on understanding what p-values actually measure versus how researchers sometimes misinterpret them. P-values tell us the probability of observing our data (or more extreme) if the null hypothesis were true, but they don't have inherent meaning beyond our predetermined significance threshold.
The correct interpretation recognizes that p = 0.08 is simply a non-significant result at α=0.05. There's no statistical basis for calling this a "trend toward significance" – this phrase treats p-values as if they exist on a meaningful continuum where values close to 0.05 have special importance. Option B correctly identifies this as misleading because it suggests the p-value has inherent meaning beyond our chosen alpha level.
Looking at the wrong answers: Option A incorrectly validates the "trend" concept by suggesting p-values between 0.05 and 0.10 represent meaningful trends – this is a common misconception with no statistical foundation. Option C seems reasonable but actually perpetuates the problematic thinking by accepting that "proximity to significance" is meaningful. Option D goes too far in the opposite direction, incorrectly stating that p > 0.05 means "no effect whatsoever" – we simply fail to reject the null hypothesis, which doesn't prove no effect exists.
Study tip: Remember that statistical significance is binary at your chosen alpha level – results are either significant or not. Phrases like "trending toward significance," "marginally significant," or "approaching significance" are red flags indicating misinterpretation of p-values. - A researcher obtains p = 0.15 in a pilot study with n = 20 and calculates that n = 80 would provide 80% power to detect the observed effect size. She conducts the larger study and obtains p = 0.06. How should this sequence of results be interpreted?
Answer: Both studies suggest weak evidence, and the effect may not be practically important
When interpreting a sequence of studies, you need to look beyond simple significance thresholds and consider the broader pattern of evidence. Both p-values here (0.15 and 0.06) fall in the "gray zone" where results are neither clearly significant nor clearly null.
The correct interpretation is C because both studies actually tell a consistent story of weak evidence. A p-value of 0.15 suggests some signal but insufficient evidence for significance, while 0.06 indicates a slightly stronger signal that still falls short of the conventional 0.05 threshold. The fact that even with quadrupled sample size (n=20 to n=80) the effect barely approached significance suggests the true effect size may be smaller than initially estimated, raising questions about practical importance.
A is wrong because while the pilot study was indeed underpowered, p=0.06 doesn't "confirm" the effect exists—it's still not statistically significant. B incorrectly frames this as contradictory results when both are actually on the same side of the significance threshold and tell a coherent story of marginal evidence. D is wrong because pilot studies aren't "misleading" just because they're underpowered—they provide valuable preliminary information that should be interpreted in context with follow-up studies.
Study tip: Don't fall into the "significant vs. not significant" binary thinking trap. P-values near 0.05 (whether slightly above or below) often indicate weak evidence that warrants careful interpretation. Always consider the pattern across studies and whether effect sizes justify practical significance, not just statistical significance.
- In a bioequivalence study, the null hypothesis is that two treatments differ by more than 20% (indicating non-equivalence), and the alternative hypothesis is that they differ by 20% or less (indicating equivalence). A test yields p = 0.03 with α=0.05. What is the correct interpretation of this result?
Answer: There is significant evidence that the treatments are bioequivalent, rejecting the hypothesis of non-equivalence
This is an equivalence test where H₀ represents non-equivalence and H₁ represents equivalence. With p = 0.03 < 0.05, we reject H₀ (non-equivalence) and conclude bioequivalence. Choice A incorrectly interprets the hypothesis structure. Choice C applies standard hypothesis testing logic incorrectly. Choice D misunderstands that rejecting H₀ here supports equivalence.
- A researcher reports: 'We found a highly significant result (p = 0.001) with a very small effect size (Cohen's d = 0.1) in our study of 10,000 participants.' What is the most important consideration when interpreting this finding?
Answer: Statistical significance does not imply practical significance; the effect size suggests minimal clinical importance
When you encounter results combining statistical significance with effect sizes, you're dealing with one of the most important distinctions in biostatistics: the difference between statistical and practical significance. This scenario perfectly illustrates how large sample sizes can produce statistically significant results even when the actual effect is negligible.
The correct interpretation focuses on practical significance. With 10,000 participants, even tiny differences become statistically detectable, yielding very small p-values. However, Cohen's d = 0.1 represents a very small effect size—so small it's likely clinically meaningless. A treatment that produces statistically significant but practically insignificant improvements offers little real-world benefit. This is why answer D correctly emphasizes that statistical significance doesn't guarantee practical importance.
Answer A is wrong because large sample sizes don't invalidate significance testing—they actually increase statistical power to detect effects. Answer B represents a dangerous misconception: assuming that smaller p-values automatically mean greater practical importance. The p-value only tells you how confident you can be that an effect exists, not whether that effect matters clinically. Answer C incorrectly assumes computational error when this scenario is actually quite common in large-scale studies.
Remember this key principle: as sample sizes increase, you can detect smaller and smaller effects with high statistical confidence. Always examine effect sizes alongside p-values to determine whether statistically significant findings are actually meaningful. In clinical research especially, ask yourself whether the detected difference would change patient outcomes or clinical practice.
- An investigator sets α=0.01 for a study and obtains a p-value of 0.025. A colleague argues that if α had been set to 0.05, the results would be more convincing. What is the most appropriate response?
Answer: The colleague is incorrect because the p-value provides the same evidence regardless of the chosen α level
When you encounter questions about statistical significance and p-values, remember that the p-value represents the strength of evidence against the null hypothesis, while α is simply the threshold you choose for decision-making.
The p-value of 0.025 provides exactly the same evidence regardless of whether you compare it to α=0.01 or α=0.05. This value tells you there's a 2.5% probability of observing results this extreme or more extreme if the null hypothesis were true. That probability doesn't change based on your chosen significance level.
Option A is incorrect because a larger α doesn't make results "more statistically significant" – it only makes them more likely to be declared significant by lowering the bar for rejection. The actual strength of evidence (the p-value) remains unchanged.
Option C commits a common misconception by focusing on the significant/non-significant dichotomy rather than the underlying evidence. While the classification would change from non-significant to significant, this doesn't make the results more convincing.
Option D raises an important point about post-hoc α adjustment, but the colleague's argument isn't necessarily about changing α after seeing data – they're discussing what would have happened with different initial criteria.
The key insight is that statistical significance is a binary classification system, while the p-value provides continuous information about evidence strength. When evaluating research, focus on the p-value itself rather than arbitrary significance thresholds. A result with p = 0.025 carries the same evidential weight regardless of your α choice. - A clinical trial uses α=0.025 for the primary analysis to account for an interim analysis. The final result shows p = 0.04. The researchers state they cannot conclude efficacy. A reviewer argues they should use α=0.05 since this is the final analysis. Who is correct?
Answer: The researchers are correct because the α level must account for all planned analyses to control Type I error
When you encounter questions about multiple comparisons in clinical trials, think about the fundamental principle of Type I error control. The alpha level must be set to maintain the overall probability of falsely rejecting the null hypothesis across all planned analyses, not just individual tests.
The researchers are correct because they properly applied alpha spending to control the familywise error rate. When conducting an interim analysis followed by a final analysis, you're essentially performing multiple hypothesis tests on the same data. Without adjustment, each test at α=0.05 would give you a cumulative Type I error rate exceeding 5%. By using α=0.025 for the final analysis, they allocated part of their "alpha budget" to the interim look while maintaining overall Type I error control at 0.05.
Option A is wrong because interim analyses absolutely affect interpretation—they create a multiple comparisons problem that demands alpha adjustment. Option C misunderstands the approach: adjusting the significance level (alpha spending) is the standard, appropriate method rather than post-hoc p-value adjustment. Option D incorrectly suggests that 0.025 is arbitrarily conservative, when it's actually a calculated adjustment based on the study's sequential design.
The p-value of 0.04 exceeds their adjusted threshold of 0.025, so claiming efficacy would inflate Type I error risk beyond acceptable levels.
Study tip: Remember that in sequential trial designs, alpha spending is like a budget—you must allocate it across all planned analyses. The total can't exceed your target Type I error rate, regardless of how many looks you take at the data. - A study reports: 'The treatment showed a statistically significant improvement (p = 0.045) with a 2% increase in success rate.' A critic argues this result, while statistically significant, may not be practically meaningful. What does this illustrate?
Answer: Statistical significance and practical significance are distinct concepts that should both be considered
This question tests your understanding of a critical distinction in biostatistics: the difference between statistical significance and practical (clinical) significance. When you encounter results that are statistically significant but involve small effect sizes, you should always consider whether the finding is meaningful in real-world terms.
The correct answer is C because statistical significance and practical significance are indeed separate concepts that researchers must evaluate independently. A p-value of 0.045 indicates that the observed 2% improvement is unlikely due to chance alone, satisfying the criterion for statistical significance. However, whether a 2% increase in success rate represents a clinically meaningful improvement depends on the specific medical context, costs, risks, and patient outcomes involved.
Option A is wrong because statistical significance never automatically guarantees practical importance. Even tiny, clinically irrelevant differences can achieve statistical significance with large enough sample sizes. Option B incorrectly suggests that p-values near 0.05 are unreliable—while p-hacking is a concern, a p-value of 0.045 is still below the conventional 0.05 threshold. Option D makes an assumption about sample size that, while plausible, isn't definitively supported by the information given and misses the main point about the distinction between types of significance.
Remember this key principle: always ask two questions when evaluating research results—"Is this difference statistically significant?" and "Is this difference large enough to matter in practice?" Both questions are essential for proper interpretation of biostatistical findings.
- A researcher conducts a two-tailed hypothesis test with α=0.05 and obtains a p-value of 0.08. If the same data were analyzed using a one-tailed test in the correct direction, what can be concluded about the new p-value and the decision?
Answer: The new p-value would be 0.04, and the null hypothesis would be rejected at α=0.05
When you encounter questions about converting between one-tailed and two-tailed tests, remember that the key difference lies in how the critical region is distributed and how p-values are calculated.
In a two-tailed test, the critical region is split between both tails of the distribution, so the p-value represents the probability of obtaining results as extreme or more extreme in either direction. When you have a two-tailed p-value of 0.08, this means there's an 8% chance of getting results this extreme in either tail combined.
For a one-tailed test in the correct direction, you're only looking at one tail of the distribution. The p-value becomes half of the two-tailed p-value: 0.08÷2=0.04. Since 0.04 < 0.05, you would reject the null hypothesis at α=0.05. This makes answer A correct.
Answer B incorrectly doubles the p-value to 0.16, which would happen if you mistakenly thought one-tailed tests were more conservative. Answer C assumes the p-value stays the same, missing that one-tailed tests concentrate the critical region in one direction. Answer D correctly calculates the new p-value as 0.04 but incorrectly concludes the null hypothesis wouldn't be rejected, failing to compare it properly to the alpha level.
Remember this conversion rule: when switching from a two-tailed to a one-tailed test in the correct direction, divide the p-value by 2. This often changes borderline non-significant results into significant ones, which is why the choice between one-tailed and two-tailed tests should always be made before data collection. - A study with 1000 participants reports p < 0.001 for a mean difference of 0.1 units on a 100-point scale. A smaller study with 50 participants reports p = 0.04 for a mean difference of 2.5 units on the same scale. Which provides more compelling evidence?
Answer: The second study because it shows a larger effect size despite the smaller sample
When evaluating research findings, you need to distinguish between statistical significance (p-values) and clinical or practical significance (effect size). A small p-value doesn't automatically mean the finding is more important or compelling.
The second study demonstrates a much larger effect size: 2.5 units versus 0.1 units on the same 100-point scale. This represents a 25-fold difference in magnitude. While the first study achieved statistical significance more easily due to its large sample size (1000 vs 50), the actual difference it detected is practically negligible—just 0.1% of the total scale range. The second study found a meaningful difference of 2.5% of the scale range, which is far more likely to represent a clinically or practically important finding.
Choice A incorrectly prioritizes p-value and sample size over effect size. Large samples can detect trivially small differences as "statistically significant." Choice C wrongly assumes that achieving p < 0.05 makes studies equivalent—this ignores the magnitude of effects entirely. Choice D falls into the common trap of p-value worship, assuming smaller p-values always indicate stronger evidence regardless of practical importance.
The key insight is that with large enough samples, you can achieve statistical significance for virtually any difference, no matter how tiny and meaningless. Conversely, smaller studies that achieve significance despite their limited power likely detected genuinely large, important effects.
Study tip: Always ask yourself "So what?" when interpreting results. Statistical significance tells you the finding probably isn't due to chance, but effect size tells you whether anyone should actually care about it.
