College Statistics Quiz: Interpreting Results In Context
20 questions · exam conditions
0:00
Interpreting Results In ContextQuestion 1 of 20

A pilot study (n=20) for a new therapy for anxiety finds that patients' average anxiety scores decrease by 15 points on a 100-point scale, a large effect. However, the result is not statistically significant (p = 0.25). A subsequent, much larger study (n=1000) finds a statistically significant (p = 0.01) decrease of 1.2 points. Which statement best reconciles these findings?

The pilot study was flawed and its large effect should be ignored; the true effect is small, as shown by the larger study.
The pilot study likely had insufficient statistical power to detect significance, while the larger study could detect a small effect as significant.
The larger study is more practically significant because its result was statistically significant with a low p-value.
Contradictory results from the two studies suggest that the therapy has no real effect on anxiety.
← Back to quizzes

College Statistics Quiz

College Statistics Quiz: Interpreting Results In Context

Practice Interpreting Results In Context in College Statistics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Interpreting Results In Context, giving you a quick way to practice the rules, question types, and explanations that matter most for College Statistics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

A pilot study (n=20) for a new therapy for anxiety finds that patients' average anxiety scores decrease by 15 points on a 100-point scale, a large effect. However, the result is not statistically significant (p = 0.25). A subsequent, much larger study (n=1000) finds a statistically significant (p = 0.01) decrease of 1.2 points. Which statement best reconciles these findings?

  1. The pilot study was flawed and its large effect should be ignored; the true effect is small, as shown by the larger study.
  2. The pilot study likely had insufficient statistical power to detect significance, while the larger study could detect a small effect as significant. (correct answer)
  3. The larger study is more practically significant because its result was statistically significant with a low p-value.
  4. Contradictory results from the two studies suggest that the therapy has no real effect on anxiety.
Explanation: The correct answer is B. The pilot study's large effect size combined with a non-significant p-value is a classic sign of low statistical power due to a small sample size. The larger study had enough power to detect even a very small effect (1.2 points) as statistically significant. This scenario highlights the interplay of effect size, sample size, and statistical significance. Distractor A dismisses the pilot study's findings entirely, but the large effect size is still informative. Distractor C confuses statistical significance with practical significance; the 15-point drop is more practically significant than the 1.2-point drop. Distractor D fails to provide a nuanced interpretation of how sample size affects statistical results.

Question 2

A study reports that a new exercise regimen results in an average weight loss of 3 pounds over six months. The p-value for this result is 0.04. The study also reports a Cohen's d of 0.15. Cohen's d is a measure of effect size, where values around 0.2 are considered 'small.' What is the most complete interpretation?

  1. The regimen causes a statistically significant amount of weight loss, so it should be widely recommended.
  2. The effect is statistically significant, but the Cohen's d value indicates the magnitude of the effect is small from a practical standpoint. (correct answer)
  3. The Cohen's d is small, which contradicts the p-value. Therefore, the study's results are likely invalid.
  4. A p-value of 0.04 is close to the 0.05 threshold, indicating the evidence for any effect is weak and should be ignored.
Explanation: The correct answer is B. This question requires interpreting both a p-value and a formal effect size metric (Cohen's d). The p-value (0.04) indicates statistical significance. The Cohen's d (0.15) provides a standardized measure of the effect's magnitude, and a value of 0.15 is typically considered small. Thus, the result is statistically significant but practically small. Distractor A ignores the small effect size. Distractor C incorrectly suggests that a significant p-value and a small effect size are contradictory; they are not. Distractor D misinterprets the p-value as a measure of the strength of evidence in an absolute sense.

Question 3

A new smartphone battery is tested and found to last, on average, 10 minutes longer than the previous model. With a large sample of phones, this result is statistically significant (p = 0.005). The company advertises the new phone as having 'significantly longer battery life.' Which of the following statements is true?

  1. The claim is truthful because 'significantly' refers to the statistical significance of the result.
  2. The claim is misleading because a 10-minute increase is not practically significant for most users.
  3. The claim is false because practical significance is more important than statistical significance in advertising.
  4. Both A and B are true; the claim is technically true in a statistical sense but potentially misleading in a practical sense. (correct answer)
Explanation: When you encounter questions about statistical versus practical significance, you're dealing with one of the most important distinctions in applied statistics. Statistical significance tells you whether an observed difference is likely real (not due to chance), while practical significance tells you whether that difference matters in real-world terms. In this scenario, the 10-minute battery improvement is statistically significant (p = 0.005), meaning we can be confident the improvement is real. However, for most smartphone users, 10 minutes of extra battery life is negligible—hardly noticeable in daily use. This creates a tension between statistical truth and practical meaning. Answer D correctly captures this nuance. The company's claim is technically accurate because "significantly" can legitimately refer to statistical significance, but it's also potentially misleading because most consumers interpret "significantly longer" as meaningfully longer in practical terms. Answer A is incomplete—while the statistical interpretation is valid, it ignores the practical dimension that makes the claim problematic. Answer B is also incomplete—it correctly identifies the practical concern but dismisses the valid statistical meaning entirely. Answer C goes too far by calling the claim outright "false" when it has legitimate statistical grounding. The key insight is that both statistical and practical significance matter, but they serve different purposes. Statistical significance establishes that an effect exists; practical significance determines whether you should care about it. Study tip: Always ask two questions when evaluating research claims: "Is this difference real?" (statistical significance) and "Does this difference matter?" (practical significance). Both are necessary for meaningful conclusions.

Question 4

Two different studies investigate the effect of a new fertilizer on crop yield. Study A finds an increase of 50 kg/hectare with p = 0.04. Study B finds an increase of 20 kg/hectare with p = 0.005. Assuming both studies were well-conducted, which statement is the most valid conclusion?

  1. Study B provides stronger evidence of a practically important effect because its p-value is smaller.
  2. Study A's finding is more practically significant because the observed increase in yield was larger. (correct answer)
  3. The results are contradictory, so it is likely that the fertilizer has no real effect on crop yield.
  4. Both studies demonstrate a practically and statistically significant effect of the fertilizer.
Explanation: The correct answer is B. Practical significance is determined by the magnitude of the effect, not the p-value. The effect size in Study A (50 kg/hectare) is larger and therefore more practically significant than the effect size in Study B (20 kg/hectare). Study B's smaller p-value is likely a result of a larger sample size or lower variability, allowing it to detect a smaller effect with more statistical certainty. Distractor A incorrectly uses the p-value to judge practical importance. Distractor C misinterprets the results as contradictory when they simply reflect different effect sizes and statistical power. Distractor D makes a claim about practical significance for Study B that may not be warranted, as 20 kg/hectare could be a small effect.

Question 5

A national study involving 100,000 high school students found a statistically significant correlation of r=0.08r = 0.08 (p < 0.001) between students' final exam scores and the number of hours they spent in a particular after-school tutoring program. Which of the following is the most appropriate interpretation of this result?

  1. The tutoring program has a strong, positive effect on student exam scores, making it a highly valuable intervention.
  2. The statistical significance proves that the tutoring program causes an improvement in student exam scores.
  3. The relationship between tutoring and exam scores is statistically significant but very weak, suggesting it has low practical importance. (correct answer)
  4. The p-value is likely incorrect because a correlation of 0.08 is too close to zero to be statistically significant.
Explanation: The correct answer is C. A correlation coefficient of r=0.08r = 0.08 indicates a very weak positive linear relationship. While the large sample size makes this weak correlation statistically significant (unlikely to be a result of random sampling variation), its practical importance is minimal. The squared correlation, r2≈0.0064r^2 \approx 0.0064, means only about 0.64% of the variance in exam scores is explained by tutoring hours. Distractor A misinterprets a weak correlation as a strong effect. Distractor B incorrectly infers causation from correlation. Distractor D fails to recognize that with a very large sample size, even very small correlations can be statistically significant.

Question 6

A meta-analysis combines the results of 50 studies on a particular antidepressant. The overall estimate for the mean effect is a reduction of 0.5 points on a 50-point depression scale. The 99% confidence interval for this estimate is [0.4, 0.6], and the p-value is extremely small (p < 0.00001). What is the most appropriate conclusion?

  1. The results are highly precise and statistically significant, but the effect size itself is very small and likely not clinically meaningful. (correct answer)
  2. There is overwhelming evidence for a large and clinically important effect of the antidepressant.
  3. The confidence interval does not contain zero, which is sufficient evidence to conclude the drug has a major impact on depression.
  4. The large number of studies in the meta-analysis likely biased the results, making the small p-value unreliable.
Explanation: When interpreting statistical results, especially from meta-analyses, you need to distinguish between statistical significance, precision, and clinical meaningfulness - three separate concepts that don't always align. The correct answer is A because this scenario demonstrates a classic case where statistical power overwhelms practical importance. With 50 studies combined, you have enormous statistical power to detect even tiny effects. The narrow confidence interval [0.4, 0.6] shows high precision, and the tiny p-value confirms statistical significance. However, a 0.5-point reduction on a 50-point scale represents only a 1% improvement - clinically negligible for most depression treatments. Answer B is wrong because it confuses statistical significance with clinical importance. The effect size (0.5 out of 50 points) is objectively small, not large. Answer C makes the common error of assuming that statistical significance alone indicates meaningful impact - just because the confidence interval excludes zero doesn't tell you whether the effect matters in practice. Answer D incorrectly suggests that including more studies creates bias. Actually, larger sample sizes in meta-analyses typically reduce bias and increase precision, which is exactly what happened here. The small p-value is reliable; it's just detecting a real but trivial effect. Remember this key principle: with large enough sample sizes, you can achieve statistical significance for practically meaningless effects. Always evaluate effect size and clinical context alongside statistical tests. Look for questions that give you very small p-values with large sample sizes - they're often testing whether you can spot statistically significant but practically unimportant findings.

Question 7

A bank analyzes a dataset of 10 million loan applications. They find that applicants whose last name begins with the letter 'G' have an average approved loan amount that is $12 higher than applicants whose last name begins with 'H'. This difference is statistically significant (p < 0.0001). Which is the most valid conclusion?

  1. The bank's loan approval process is biased based on last names, and this requires immediate investigation.
  2. The very small p-value indicates this is an important finding with significant financial implications.
  3. The finding is statistically significant due to the massive sample size, but the effect is too small to be practically meaningful. (correct answer)
  4. This statistically significant result proves that people with last names starting with 'G' are more creditworthy than those with names starting with 'H'.
Explanation: The correct answer is C. With a sample size of 10 million, even a minuscule and meaningless difference can become statistically significant. A $12 difference in loan amounts is trivial and has no practical or real-world significance. The finding is a statistical artifact of the large dataset, not an indicator of a meaningful pattern. Distractor A jumps to a conclusion of bias without considering the trivial effect size. Distractor B incorrectly links the small p-value to practical importance. Distractor D makes an unwarranted causal and character-based inference from a spurious correlation.

Question 8

A pharmaceutical company conducts a large-scale clinical trial (n=30,000) for a new drug intended to lower LDL cholesterol. The results show that the drug lowers LDL cholesterol by an average of 1.2 mg/dL compared to a placebo, with a p-value of < 0.001. From a clinical standpoint, a reduction of at least 10 mg/dL is typically required for a drug to be considered effective.

  1. The drug has a highly significant and clinically important effect, as indicated by the extremely small p-value.
  2. The study's results are statistically significant, but the effect size is too small to be considered clinically or practically significant. (correct answer)
  3. The results are not reliable because the effect size is very small, which suggests the p-value is a statistical artifact.
  4. No conclusions can be drawn about the drug's effectiveness without knowing the confidence interval for the mean reduction.
Explanation: The correct answer is B. Statistical significance (indicated by p < 0.001) means the observed effect is unlikely due to chance. However, practical or clinical significance relates to the magnitude of the effect. A 1.2 mg/dL reduction is much smaller than the 10 mg/dL threshold for clinical importance, so the effect is not practically significant. Distractor A incorrectly equates statistical significance with practical importance. Distractor C wrongly implies a small effect makes the p-value an artifact; in large samples, small effects can be statistically significant. Distractor D is incorrect because the mean effect can be compared to the clinical threshold directly to assess practical significance, even without a CI.

Question 9

A large e-commerce website tests a new font for its product descriptions. With a sample of 2 million users, they find that the new font increases the average time spent on the page by 0.5 seconds. This result is statistically significant with a p-value of 0.002. What is the most reasonable conclusion for the company's design team?

  1. The new font is a major success and should be implemented immediately due to its statistically significant impact.
  2. Although the result is statistically significant, an increase of 0.5 seconds is likely not practically meaningful for business outcomes.
  3. The statistical significance indicates the effect is real, and this small time increase could lead to higher sales if it reflects greater engagement. (correct answer)
  4. The p-value of 0.002 means there is a 0.2% chance the result is a fluke, so the company should re-run the test to be sure.
Explanation: The correct answer is C. This question requires careful consideration of the business context. While a 0.5-second increase seems trivial (ruling out B as the best answer), in a high-volume e-commerce setting, even tiny changes in engagement can have a cumulative, practical impact on metrics like conversion rates. The statistical significance confirms the effect is unlikely due to chance. Therefore, concluding that this could be practically significant is the most nuanced interpretation. Distractor A overstates the case by calling it a 'major success.' Distractor B dismisses the possibility of practical significance too quickly. Distractor D misinterprets the p-value.

Question 10

A study on workplace wellness programs reports a 95% confidence interval for the mean difference in sick days per year between participating and non-participating employees as [-0.3, 2.1]. Which of the following is the correct interpretation of this interval?

  1. The program is practically significant because employees who participate could take up to 2.1 fewer sick days on average.
  2. The result is statistically significant because the interval is mostly positive, suggesting a reduction in sick days.
  3. The result is not statistically significant because the interval contains zero, meaning no effect is a plausible value for the mean difference. (correct answer)
  4. There is a 95% probability that the true mean difference in sick days falls between -0.3 and 2.1.
Explanation: The correct answer is C. For a two-sided test, a 95% confidence interval that contains the null value (in this case, a mean difference of 0) indicates that the result is not statistically significant at the α=0.05\alpha = 0.05 level. Since the interval [-0.3, 2.1] includes 0, we cannot reject the null hypothesis of no difference. Distractor A focuses on the upper bound while ignoring that the lower bound suggests a negative effect, and it makes a claim of practical significance without evidence of statistical significance. Distractor B incorrectly assesses statistical significance. Distractor D presents a common misinterpretation of confidence intervals; the interval either contains the true parameter or it does not.

Question 11

A new manufacturing process for microchips is found to reduce the defect rate from 0.050% to 0.048%. With millions of chips produced, this 0.002 percentage point decrease is statistically significant (p = 0.01). The new process costs an additional $2 million per year to operate. The practical significance of this change primarily depends on what information?

  1. The p-value from a more powerful statistical test.
  2. The financial savings gained from preventing the small number of additional defects. (correct answer)
  3. The standard deviation of the defect rate for the old and new processes.
  4. Confirmation that the decrease in defects was directly caused by the new process.
Explanation: The correct answer is B. The core question of practical significance here is one of cost-benefit analysis. The result is already known to be statistically significant. The decision to implement the change hinges on whether the economic value of reducing the defect rate by 0.002% (e.g., fewer warranty claims, less waste) outweighs the $2 million annual cost. Distractor A is incorrect because a different test or p-value doesn't address the practical cost issue. Distractor C (standard deviation) is part of the significance calculation but not the primary factor for the business decision. Distractor D (causation) is an important scientific question, but from a business perspective, the financial tradeoff is the key element of practical significance.

Question 12

A city implements a new traffic light timing system and measures the average commute time. A study of thousands of commutes finds a statistically significant reduction in average commute time of 15 seconds (p = 0.03). A local news report headlines, "New System Dramatically Cuts Commute Times." Which of the following best evaluates the headline?

  1. The headline is accurate because the result is statistically significant, meaning the time savings are real and important.
  2. The headline cannot be evaluated without knowing the margin of error associated with the 15-second estimate.
  3. The headline is inaccurate because statistical significance does not prove that the new system caused the reduction in commute times.
  4. The headline is misleading because a 15-second reduction is unlikely to be perceived as a 'dramatic' or practically significant improvement by most commuters. (correct answer)
Explanation: This question tests your ability to distinguish between statistical significance and practical significance—a crucial concept in interpreting research results. Statistical significance tells you whether an effect is likely real (not due to chance), while practical significance tells you whether that effect matters in the real world. The correct answer is D because while the 15-second reduction is statistically significant (p = 0.03), it's hardly "dramatic" from a practical standpoint. Most commuters wouldn't even notice saving 15 seconds on their daily drive, making the headline misleading and sensationalized. Let's examine why the other options miss the mark: Option A incorrectly equates statistical significance with practical importance. A p-value of 0.03 only means the result is unlikely due to chance—it doesn't tell you whether the effect size (15 seconds) is meaningful to commuters. Option B focuses on the margin of error, but that's not the main issue here. Even with perfect precision, a 15-second reduction wouldn't justify calling the improvement "dramatic." Option C raises the valid point about causation versus correlation, but the question stem describes this as a study of a new system implementation, which suggests a controlled intervention rather than just observational data. The bigger problem is the exaggerated language about a tiny effect. Study tip: When evaluating research claims, always ask two questions: "Is the effect real?" (statistical significance) and "Does the effect matter?" (practical significance). Large sample sizes can make tiny, meaningless differences appear statistically significant, so always consider the magnitude of the effect in real-world terms.

Question 13

A researcher fails to find a statistically significant effect of a new medication on patient recovery times (p = 0.30). However, the 95% confidence interval for the mean difference in recovery days is [-2, 8]. A practically significant difference would be any reduction greater than 1 day. What is the most appropriate conclusion?

  1. The results are inconclusive; the study may have been underpowered as the confidence interval includes both practically significant and insignificant values. (correct answer)
  2. The study proves that the medication has no effect on recovery time.
  3. The medication is likely harmful because the confidence interval includes a potential increase in recovery time of up to 8 days.
  4. The study shows a small but practically significant effect that was missed due to high variability in the data.
Explanation: When you encounter a question combining hypothesis testing results with confidence intervals, you need to interpret both pieces of information together to understand what the study actually tells us. The key insight here is recognizing what the confidence interval reveals about uncertainty. The interval [-2, 8] means we're 95% confident the true difference in recovery days falls somewhere between a 2-day improvement and an 8-day worsening. Since a practically significant effect is defined as any reduction greater than 1 day, this interval spans both practically significant values (reductions of 2 days) and practically insignificant values (small changes or increases). Answer A correctly identifies this as inconclusive evidence, likely due to insufficient statistical power. The wide confidence interval (spanning 10 days total) suggests high variability or a small sample size, making it difficult to detect effects even if they exist. Answer B incorrectly claims the study "proves" no effect. A non-significant p-value (0.30) simply means we failed to detect an effect, not that no effect exists. This is a classic misconception about null hypothesis testing. Answer C misinterprets the confidence interval by focusing only on the potential harm (8-day increase) while ignoring that this represents just one end of the uncertainty range, not a likely outcome. Answer D wrongly claims the study shows a practically significant effect. The non-significant p-value and wide confidence interval provide no evidence for any specific effect size. Remember: when confidence intervals are wide and span both meaningful and trivial effect sizes, consider whether the study had adequate power to detect the effect of interest.

Question 14

In a massive analysis of genomic data from 500,000 individuals, a particular gene variant is found to be associated with a 0.02-point increase in IQ score on a standard test with a mean of 100 and a standard deviation of 15. The p-value for this association is 1×10−101 \times 10^{-10}. How should this finding be interpreted?

  1. The result is statistically significant, but the effect size is so minuscule that it has no practical or explanatory importance for an individual's IQ. (correct answer)
  2. The gene variant has a powerful and important effect on IQ, as evidenced by the incredibly small p-value.
  3. The result is likely a Type I error (false positive) because the effect size is implausibly small.
  4. The association proves a causal genetic link, meaning individuals with this variant will have a higher IQ.
Explanation: This question tests your understanding of the crucial distinction between statistical significance and practical significance - a concept that becomes especially important with very large sample sizes. With 500,000 individuals, you have enormous statistical power to detect even tiny effects. A 0.02-point IQ increase is minuscule compared to the test's standard deviation of 15 points - it represents just 0.13% of one standard deviation. The extremely small p-value (1×10−101 \times 10^{-10}) simply reflects the massive sample size's ability to detect this tiny difference with high confidence, not the importance of the effect. Answer A is correct because it recognizes both the statistical significance (p-value < 0.05) and the trivial practical impact. A 0.02-point difference is meaningless for any individual. Answer B makes the classic error of equating a small p-value with a large effect. P-values measure the strength of evidence against the null hypothesis, not effect magnitude. With huge samples, even negligible effects yield tiny p-values. Answer C incorrectly suggests a Type I error. The result is likely real given the massive sample, just unimportant. Type I errors involve false positives, not genuine but trivial effects. Answer D commits the correlation-causation fallacy. Observational associations, even strong ones, don't prove causation. Genetic variants could be linked to IQ through complex pathways or confounding variables. Remember: Large studies can find statistically significant results that are practically meaningless. Always evaluate both the p-value AND the effect size to determine real-world importance.

Question 15

A school district considers two new reading programs. Program A was tested on 5,000 students and yielded a mean score increase of 2 points with p = 0.01. Program B was tested on 200 students and yielded a mean score increase of 10 points with p = 0.06. Both programs have similar costs. Which statement is the best basis for the district's decision?

  1. Program B appears to have a much larger practical effect, and its lack of statistical significance may be due to a smaller sample size. (correct answer)
  2. Program A is superior because its results are statistically significant, whereas Program B's are not.
  3. Neither program should be chosen because Program A's effect is too small and Program B's result could be due to chance.
  4. The p-values should be directly compared; since 0.01 is much smaller than 0.06, Program A is substantially more effective.
Explanation: When evaluating research studies, you need to distinguish between statistical significance and practical significance. Statistical significance (reflected in p-values) tells you whether an effect is likely real versus due to chance, while practical significance concerns whether the effect size is meaningful in real-world terms. Program A shows a statistically significant result (p = 0.01 < 0.05) with a 2-point increase, while Program B shows a non-significant result (p = 0.06 > 0.05) with a 10-point increase. However, Program B's much larger effect size (10 vs 2 points) suggests greater practical impact. The lack of statistical significance for Program B is likely due to its smaller sample size (200 vs 5,000 students), which reduces statistical power to detect effects. Answer A correctly identifies that Program B's larger practical effect matters, and acknowledges that sample size affects statistical significance. Answer B incorrectly prioritizes statistical significance over effect size, missing the practical importance. Answer C dismisses both programs without considering that Program B's large effect size could be practically valuable despite borderline significance. Answer D makes the common mistake of treating p-values as measures of effect size—smaller p-values don't necessarily indicate larger or more important effects. Study tip: Remember that statistical significance depends heavily on sample size. Large samples can make tiny, meaningless effects "significant," while small samples might fail to detect large, important effects. Always consider both statistical significance AND effect size when evaluating research results.

Question 16

A researcher conducts 20 separate hypothesis tests for 20 different potential side effects of a new drug, using an alpha level of 0.05 for each test. One test, for 'mild headaches,' yields a p-value of 0.03. The observed effect size for this side effect is very small. What is the most critical consideration when interpreting this finding?

  1. The finding is statistically significant and proves that the drug causes mild headaches.
  2. The practical significance is low due to the small effect size, so the finding can be safely ignored.
  3. With 20 tests, there is a high probability of observing at least one Type I error, and this significant result may be a false positive. (correct answer)
  4. The p-value of 0.03 is the probability that the null hypothesis is true, so there is only a 3% chance this finding is random.
Explanation: The correct answer is C. This question introduces the problem of multiple comparisons. When conducting many hypothesis tests, the chance of getting a 'significant' result just by random chance (a Type I error) increases. With 20 tests at α=0.05\alpha = 0.05, the probability of at least one false positive is about 1−(0.95)20≈0.641 - (0.95)^{20} \approx 0.64. Therefore, a single significant result, especially one with a small effect size, should be viewed with skepticism. Distractor A ignores the multiple testing problem. Distractor B may be true, but the multiple testing issue is a more critical statistical consideration. Distractor D provides a common misinterpretation of the p-value.

Question 17

A political campaign runs two different online ads. Ad A results in a 2.5% donation rate, while Ad B results in a 2.8% rate. In a small trial with 500 viewers for each ad, the difference is not statistically significant (p = 0.35). What is the most reasonable interpretation for the campaign manager?

  1. The ads are equally effective, so it does not matter which one is used.
  2. Ad B is clearly superior, and the non-significant p-value should be ignored.
  3. Ad A should be chosen because there is no statistically significant evidence that Ad B is better.
  4. The results are inconclusive, but the 0.3 percentage point difference could be practically significant at scale, warranting a larger test. (correct answer)
Explanation: When interpreting statistical test results, you need to distinguish between statistical significance, practical significance, and inconclusive evidence. A non-significant p-value doesn't automatically mean "no difference exists" – it means you lack sufficient evidence to detect a difference with your current sample size. The correct interpretation is D because with only 500 viewers per ad, this study has limited power to detect small differences. The observed 0.3 percentage point difference (2.8% vs 2.5%) may seem small, but at scale it could be meaningful. If the campaign reaches millions of people, even a 0.3% improvement in donation rates could translate to substantial additional revenue. The high p-value (0.35) simply indicates the sample size was too small to determine whether this observed difference reflects a true underlying difference or random variation. Option A incorrectly interprets non-significance as proof of equal effectiveness. Statistical tests cannot prove equality – they can only fail to detect differences. Option B commits the opposite error by ignoring the p-value entirely and treating the observed difference as definitive. This overlooks that random sampling variation could easily produce a 0.3% difference even if the ads are truly equal. Option C makes a flawed decision based on insufficient evidence, essentially choosing the status quo without proper justification. Remember: when you see a small effect size with a non-significant result and small sample size, consider whether the effect could be practically meaningful at scale. This often warrants collecting more data rather than concluding "no difference exists."

Question 18

A researcher is developing a new method for detecting a rare disease. The new method's accuracy is 99.8%, while the old method's is 99.7%. With a very large validation set, this 0.1% improvement is shown to be statistically significant (p = 0.01). Under which condition would this small improvement be most practically significant?

  1. If the new method is much more expensive and difficult to administer than the old method.
  2. If the disease is mild and easily treated once detected.
  3. If the effect size, measured by a metric like odds ratio, is also very small.
  4. If the method is used for a nationwide screening program involving millions of people. (correct answer)
Explanation: When evaluating research results, you need to distinguish between statistical significance and practical significance. Statistical significance tells you whether an observed difference is likely real (not due to chance), while practical significance tells you whether that difference actually matters in the real world. Here, a 0.1% improvement in accuracy is statistically significant with p = 0.01, meaning this difference is probably real. But whether it's practically meaningful depends entirely on the context and scale of application. Option D is correct because when you apply a screening method to millions of people, even tiny improvements become enormously impactful. A 0.1% improvement across 10 million people means 10,000 additional correct diagnoses. For a rare disease, this could represent catching hundreds or thousands more cases that would otherwise be missed—a massive public health benefit. Option A is wrong because higher costs and difficulty would make the small improvement less practically significant, not more. Option B is incorrect because if the disease is mild and easily treated, missing a few cases due to the slightly lower accuracy isn't as concerning—the stakes are lower. Option C represents a fundamental misunderstanding: if the effect size is very small, that would indicate the improvement is less practically significant, not more. Remember this key principle: small effect sizes can have huge practical importance when applied at scale. Always consider the scope of application when evaluating whether statistically significant findings translate to meaningful real-world impact.

Question 19

A pharmaceutical company conducts a large-scale clinical trial (n=30,000) for a new drug intended to lower LDL cholesterol. The results show that the drug lowers LDL cholesterol by an average of 1.2 mg/dL compared to a placebo, with a p-value of < 0.001. From a clinical standpoint, a reduction of at least 10 mg/dL is typically required for a drug to be considered effective.

  1. The drug has a highly significant and clinically important effect, as indicated by the extremely small p-value.
  2. The study's results are statistically significant, but the effect size is too small to be considered clinically or practically significant. (correct answer)
  3. The results are not reliable because the effect size is very small, which suggests the p-value is a statistical artifact.
  4. No conclusions can be drawn about the drug's effectiveness without knowing the confidence interval for the mean reduction.
Explanation: The correct answer is B. Statistical significance (indicated by p < 0.001) means the observed effect is unlikely due to chance. However, practical or clinical significance relates to the magnitude of the effect. A 1.2 mg/dL reduction is much smaller than the 10 mg/dL threshold for clinical importance, so the effect is not practically significant. Distractor A incorrectly equates statistical significance with practical importance. Distractor C wrongly implies a small effect makes the p-value an artifact; in large samples, small effects can be statistically significant. Distractor D is incorrect because the mean effect can be compared to the clinical threshold directly to assess practical significance, even without a CI.

Question 20

A national study involving 100,000 high school students found a statistically significant correlation of r=0.08r = 0.08 (p < 0.001) between students' final exam scores and the number of hours they spent in a particular after-school tutoring program. Which of the following is the most appropriate interpretation of this result?

  1. The tutoring program has a strong, positive effect on student exam scores, making it a highly valuable intervention.
  2. The statistical significance proves that the tutoring program causes an improvement in student exam scores.
  3. The relationship between tutoring and exam scores is statistically significant but very weak, suggesting it has low practical importance. (correct answer)
  4. The p-value is likely incorrect because a correlation of 0.08 is too close to zero to be statistically significant.
Explanation: The correct answer is C. A correlation coefficient of r=0.08r = 0.08 indicates a very weak positive linear relationship. While the large sample size makes this weak correlation statistically significant (unlikely to be a result of random sampling variation), its practical importance is minimal. The squared correlation, r2≈0.0064r^2 \approx 0.0064, means only about 0.64% of the variance in exam scores is explained by tutoring hours. Distractor A misinterprets a weak correlation as a strong effect. Distractor B incorrectly infers causation from correlation. Distractor D fails to recognize that with a very large sample size, even very small correlations can be statistically significant.