Potential Errors When Performing Tests - AP Statistics

A large $p$-value means insufficient evidence, not proof $H_0$ is true. Failing to reject $H_0$ doesn't prove it's true.

A $p$-value is $P(\text{data or more extreme}\mid H_0)$. It's the probability of the data given $H_0$, not vice versa.

Statistical significance does not imply practical importance. Small $p$-values can occur for trivial effects with large $n$.

Conditions not met; $t$ procedures may be invalid. $t$ procedures assume approximate normality or large $n$.

Using two-sample methods instead of paired methods. Paired data violates independence assumption of two-sample tests.

Multiple comparisons inflate the Type I error rate. Testing many hypotheses increases chance of false positives.

Selection bias; results may not generalize. Non-random samples may not represent the population.

Confounding; association does not imply causation. Without randomization, other variables may explain the relationship.

Post hoc tail choice invalidates the $p$-value. Test direction must be chosen before seeing data.

$\beta$ increases. Lower $\alpha$ makes rejecting $H_0$ harder, increasing Type II errors.

Hypotheses must be about parameters (e.g., $\mu$, $p$), not statistics. We test population values, not sample statistics.

Use pooled $\sqrt{\frac{p_0(1-p_0)}{n}}$ under $H_0$, not sample $p$. Under $H_0$, use hypothesized $p_0$, not sample proportion.

Expected counts condition violated; results may be unreliable. $\chi^2$ test requires all expected counts $\geq 5$.

Nonresponse bias can invalidate inference. Missing responses may differ systematically from respondents.

Failing to reject $H_0$ when $H_0$ is false. This is a false negative - missing a real effect.

Type I error. We set $\alpha$ directly; $\beta$ depends on other factors.

The probability of a Type I error. $\alpha$ is the threshold we set for rejecting $H_0$.

Rejecting $H_0$ when $H_0$ is true. This is a false positive - incorrectly concluding an effect exists.

$1-\beta$, the probability of rejecting a false $H_0$. Power measures test's ability to detect false null hypotheses.

Reject $H_0$ if $p\le\alpha$; otherwise fail to reject $H_0$. Compare p-value to significance level; reject when evidence is strong enough.

Use a $t$ test when $\sigma$ is unknown. Population SD rarely known; sample SD requires t-distribution for proper inference.

Normal approximation may be invalid; conditions not met. Success-failure condition ensures normal approximation is appropriate.

$1-\beta$. Power is the probability of correctly rejecting a false null hypothesis.

The probability of a Type II error. Depends on effect size, sample size, and $\alpha$; decreases as power increases.

The probability of a Type I error. Set before testing; the maximum acceptable risk of making a Type I error.

Direction of $H_a$ must be set before analyzing data. Data snooping invalidates the test; hypotheses must be predetermined.

Use a one-sided $p$-value consistent with $H_a$. P-value calculation must match the alternative hypothesis direction specified.

Reject $H_0$; sufficient evidence for $H_a$. Data provides strong enough evidence to conclude the alternative is likely true.

Fail to reject $H_0$; insufficient evidence for $H_a$. Cannot prove $H_0$; only state that data doesn't provide strong evidence against it.