Potential Errors When Performing Tests - AP Statistics

1. Identify the potential error: performing a $z$ test for a mean when $\sigma$ is unknown.

   Use a $t$ test when $\sigma$ is unknown. Population SD rarely known; sample SD requires t-distribution for proper inference.

2. What is the Type I error in hypothesis testing, stated in terms of $H_0$?

   Rejecting $H_0$ when $H_0$ is true. This is a false positive - incorrectly concluding an effect exists.

3. Identify the error: Treating a nonresponse-biased sample as if it were a random sample.

   Nonresponse bias can invalidate inference. Missing responses may differ systematically from respondents.

4. What is the common formula error in a $z$ test for a proportion regarding standard error?

   Use pooled $\sqrt{\frac{p_0(1-p_0)}{n}}$ under $H_0$, not sample $p$. Under $H_0$, use hypothesized $p_0$, not sample proportion.

5. Identify the potential error: running many tests at $\alpha=0.05$ and treating each as independent evidence.

   Inflated overall Type I error rate (multiple comparisons problem). Each test has 5% error rate; multiple tests compound the overall error probability.

6. Which change increases power: increasing $n$, increasing $\alpha$, or increasing variability?

   Increasing $n$ or increasing $\alpha$ (not increasing variability). Larger samples and higher $\alpha$ increase power; more variability decreases it.

7. Identify the error: using a $2$-sided $p$-value when the stated alternative is one-sided.

   Use a one-sided $p$-value consistent with $H_a$. P-value calculation must match the alternative hypothesis direction specified.

8. Which option correctly describes how decreasing $\alpha$ affects Type I error risk?

   It decreases the chance of a Type I error. Lower $\alpha$ means stricter criterion for rejection, reducing false positives.

9. What is the correct conclusion wording when $p>\alpha$?

   Fail to reject $H_0$; insufficient evidence for $H_a$. Cannot prove $H_0$; only state that data doesn't provide strong evidence against it.

10. Identify the error: Using a $\chi^2$ test when expected counts are too small.

    Expected counts condition violated; results may be unreliable. $\chi^2$ test requires all expected counts $\geq 5$.

11. Identify the error: Interpreting a $p$-value as $P(H_0\text{ is true})$.

    A $p$-value is $P(\text{data or more extreme}\mid H_0)$. It's the probability of the data given $H_0$, not vice versa.

12. Identify the potential error: using a one-proportion $z$ test when $np_0<10$ or $n(1-p_0)<10$.

    Normal approximation may be invalid; conditions not met. Success-failure condition ensures normal approximation is appropriate.

13. What is the key design error when claiming causation from an observational study?

    Confounding; association does not imply causation. Without randomization, other variables may explain the relationship.

14. If sample size $n$ increases (all else fixed), what happens to power $1-\beta$?

    Power increases. Larger samples reduce variability, making effects easier to detect.

15. Identify the error: Concluding $H_0$ is true because the $p$-value is large.

    A large $p$-value means insufficient evidence, not proof $H_0$ is true. Failing to reject $H_0$ doesn't prove it's true.

16. Identify the error: Using a $t$ test on strongly skewed data with small $n$ and no checks.

    Conditions not met; $t$ procedures may be invalid. $t$ procedures assume approximate normality or large $n$.

17. Identify the error: Using a one-sided test after seeing which direction favors $H_a$.

    Post hoc tail choice invalidates the $p$-value. Test direction must be chosen before seeing data.

18. If $\alpha$ decreases while conditions stay the same, what happens to $\beta$?

    $\beta$ increases. Lower $\alpha$ makes rejecting $H_0$ harder, increasing Type II errors.

19. What does the significance level $\alpha$ represent in a test of significance?

    The probability of a Type I error. $\alpha$ is the threshold we set for rejecting $H_0$.

20. What is the standard condition error for two-sample inference when samples are dependent?

    Using two-sample methods instead of paired methods. Paired data violates independence assumption of two-sample tests.

21. What is the Type II error in hypothesis testing, stated in terms of $H_0$?

    Failing to reject $H_0$ when $H_0$ is false. This is a false negative - missing a real effect.

22. Identify the error: stating that the $p$-value is $P(H_0\text{ is true})$.

    $p=P(\text{statistic as extreme as observed}\mid H_0)$. P-value measures extremeness of data given $H_0$, not probability $H_0$ is true.

23. What does a small $p$-value indicate about the data under $H_0$?

    The observed result is unlikely if $H_0$ is true. Small p-values suggest the data would be surprising if $H_0$ were true.

24. Identify the error: Failing to state hypotheses in terms of population parameters.

    Hypotheses must be about parameters (e.g., $\mu$, $p$), not statistics. We test population values, not sample statistics.

25. Which option correctly describes how decreasing $\alpha$ affects Type II error risk (all else fixed)?

    It increases the chance of a Type II error. Harder to reject $H_0$ means more likely to miss real effects when they exist.

26. Which option is a correct decision rule using a $p$-value and $\alpha$?

    Reject $H_0$ if $p\le\alpha$; otherwise fail to reject $H_0$. Compare p-value to significance level; reject when evidence is strong enough.

27. What is the correct conclusion wording when $p\le\alpha$?

    Reject $H_0$; sufficient evidence for $H_a$. Data provides strong enough evidence to conclude the alternative is likely true.

28. What is the key design error when generalizing to a population from a convenience sample?

    Selection bias; results may not generalize. Non-random samples may not represent the population.

29. What does the significance level $\alpha$ represent?

    The probability of a Type I error. Set before testing; the maximum acceptable risk of making a Type I error.

30. Identify the potential error: treating nonrandom or biased sampling as if test results generalize to a population.

    Inference may be invalid due to lack of random sampling/assignment. Statistical tests assume random sampling; bias limits generalizability.

31. Identify the error: Running many tests and treating one small $p$-value as strong evidence.

    Multiple comparisons inflate the Type I error rate. Testing many hypotheses increases chance of false positives.

32. Identify the error: Claiming statistical significance implies practical importance.

    Statistical significance does not imply practical importance. Small $p$-values can occur for trivial effects with large $n$.

33. What does power represent in a test, using $\beta$ notation?

    $1-\beta$, the probability of rejecting a false $H_0$. Power measures test's ability to detect false null hypotheses.

34. Identify the error: choosing $H_a$ after seeing the data to match the observed direction.

    Direction of $H_a$ must be set before analyzing data. Data snooping invalidates the test; hypotheses must be predetermined.

35. What is the power of a hypothesis test in terms of $\beta$?

    $1-\beta$. Power is the probability of correctly rejecting a false null hypothesis.

36. Identify the error: concluding $H_0$ is true because you failed to reject $H_0$.

    Failing to reject $H_0$ does not prove $H_0$ is true. Absence of evidence is not evidence of absence; we can only reject or not reject.

37. What does $\beta$ represent in a hypothesis test?

    The probability of a Type II error. Depends on effect size, sample size, and $\alpha$; decreases as power increases.

38. Which error can be directly controlled by choosing $\alpha$: Type I or Type II?

    Type I error. We set $\alpha$ directly; $\beta$ depends on other factors.

Study these flashcards