Evaluating Reports Based on Data - Statistics and Probability

1. What is the correct meaning of a $95\%$ confidence level in a report?

   The method captures the true parameter in $95\%$ of samples. It's about the long-run performance of the interval construction method.

2. Choose the better summary for skewed income data: mean or median?

   Median. Median resists the pull of high-income outliers.

3. What is the correct interpretation of a $95\%$ confidence interval for a population parameter?

   $95\%$ of such intervals capture the true parameter. The method produces intervals that contain the parameter 95% of the time.

4. What is the correct interpretation of a $95\%$ confidence statement in a report?

   The method captures the true value in $95\%$ of repeated samples. Not that we're $95\%$ sure about this specific interval.

5. Which conclusion is correct when a report states $p=0.03$ and $\alpha=0.05$?

   Reject $H_0$. Since $p < \alpha$, the result is statistically significant.

6. Which study type most supports a causal claim: randomized experiment or observational study?

   Randomized experiment. Random assignment controls for confounding variables.

7. Choose the correct conclusion: a $95\%$ CI for a mean difference is $(-1.2,0.4)$.

   Not statistically significant at $\alpha=0.05$ (interval includes $0$). CI contains null value $0$, so fail to reject $H_0$.

8. Which statement is correct about statistical significance versus practical importance?

   Significance may occur even when the effect is small. Large samples can detect tiny, meaningless differences.

9. Which design feature reduces confounding in an experiment: random assignment or voluntary response?

   Random assignment. Randomly assigning treatments balances confounders across groups.

10. What is the main risk when a report generalizes from a convenience sample?

    Selection bias; sample likely differs from the target population. Convenience samples are not randomly selected from the population.

11. Choose the correct conclusion: a $95\%$ CI for a proportion difference is $(0.03,0.11)$.

    Statistically significant at $\alpha=0.05$ (interval excludes $0$). All values in CI are positive, showing a real difference.

12. What is the correct meaning of a $p$-value reported for a test of $H_0$?

    Probability of data at least as extreme, assuming $H_0$ is true. Not the probability that $H_0$ is true.

13. Identify the misleading feature: a bar chart axis starts at $50$ instead of $0$.

    Truncated axis exaggerates visual differences. Makes small differences appear larger than they are.

14. Identify the issue: a poll reports results from an online click-in survey about a policy.

    Voluntary response bias. Self-selected respondents don't represent the population.

15. What is the margin of error for a confidence interval written as estimate $\pm$ margin?

    Half the interval width. The $\pm$ value represents the margin of error.

16. What does it mean if a report says its sample is an SRS of the population?

    Every same-size sample has an equal chance of selection. SRS ensures unbiased representation of the population.

17. What does a $p$-value represent in a significance test?

    Probability of data at least as extreme, assuming $H_0$. Measures how likely the observed data would occur if the null hypothesis were true.

18. What is the main risk when a report generalizes from a voluntary response sample?

    Strong selection bias; results may not represent the population. Volunteers differ systematically from non-volunteers.

19. What is the definition of a statistically significant result at level $\alpha$?

    A result with $p \le \alpha$. The p-value must be less than or equal to the significance level.

20. What is the main difference between association and causation in evaluating a report?

    Association does not imply a cause-and-effect link. Correlation doesn't prove one variable causes changes in another.

21. Identify the misleading summary: a report uses the mean for data with extreme outliers.

    Mean is not resistant; median is more appropriate. Outliers inflate the mean but don't affect median.

22. Identify the correct decision rule using a confidence interval to test $H_0: \theta=\theta_0$ at $\alpha=0.05$.

    Reject if $\theta_0$ is not in the $95\%$ CI. If the null value falls outside the CI, it's unlikely given the data.

23. Which value must a report provide to judge statistical significance at level $\alpha$?

    A $p$-value (or enough information to compute it). Compare $p$ to $\alpha$ to determine significance.

24. Find the margin of error for a reported $90\%$ CI of $[48,52]$.

    $2$. Margin = $(52-48)/2 = 2$.

25. Identify the flaw: a report claims causation from an observational study with no random assignment.

    Causal claim is not justified without random assignment. Observational studies can't establish causation due to potential confounders.

26. Which conclusion is correct when a report states $p=0.12$ and $\alpha=0.05$?

    Fail to reject $H_0$. Since $p > \alpha$, the result is not statistically significant.

27. Find the problem: a report compares two treatments but subjects chose their own treatment.

    Self-selection confounding; groups may differ before treatment. Without randomization, treatment groups aren't comparable.

28. What conclusion is justified when a report gives a $p$-value with $p<\alpha$?

    Reject $H_0$; results are statistically significant at level $\alpha$. Small $p$-values indicate data unlikely under null hypothesis.

29. What is the main risk of a voluntary response sample in a report?

    Strong selection bias from self-selection. Only those with strong opinions tend to respond.

30. What is the key question to ask when a report claims a study shows cause and effect?

    Was the study a randomized experiment (not just observational). Only randomized experiments can establish causation, not observational studies.

31. What does a report's margin of error (MOE) typically describe for a survey estimate?

    Likely sampling error range around the estimate (at a stated level). MOE quantifies uncertainty due to random sampling variability.

32. What is the difference between statistical significance and practical importance?

    Significance is about chance; importance is about effect size/impact. Small effects can be significant with large samples.

33. Which conclusion is correct if a reported $95\%$ CI for $\mu$ is $[-1,4]$ and $H_0: \mu=0$?

    Fail to reject $H_0$ at $\alpha=0.05$. Since $0$ is inside $[-1,4]$, the null hypothesis is not rejected.

34. Identify the correct claim: an observational study found a strong association between $X$ and $Y$.

    Association only; causation is not justified without random assignment. Observational studies can't control for all confounders.

35. What is a confounding variable in evaluating a data-based report?

    A variable related to both explanatory and response variables. It affects both variables, creating a spurious relationship.

36. Identify the correct critique: a report highlights $p<0.001$ but the effect size is near $0$.

    Statistically significant but possibly not practically important. Very small p-value doesn't guarantee the effect matters in practice.

37. What is the main warning sign if a report emphasizes relative change but omits baseline rates?

    Relative change can exaggerate impact without the absolute difference. "$200\%$ increase" from $1$ to $3$ is only $2$ units.

38. Which conclusion is correct if a reported $95\%$ CI for $\mu$ is $[2,5]$ and $H_0: \mu=0$?

    Reject $H_0$ at $\alpha=0.05$. Since $0$ is outside $[2,5]$, the null hypothesis is rejected.

39. Identify the confounder risk: a report links ice cream sales to drowning deaths.

    Season/temperature is a confounding variable. Both increase in summer; correlation doesn't imply causation.

40. What feature of a study design most supports a causal conclusion in a report?

    Random assignment to treatment and control groups. Randomization eliminates confounding variables by balancing groups.

Study these flashcards