Potential Errors in
Hypothesis Testing
Understanding Type I and Type II errors, the relationship between \(\alpha\), \(\beta\), and power, and how to minimize mistakes in statistical inference.
What Is a Hypothesis Test?
A hypothesis test uses sample data to decide between two competing claims about a population parameter. We start with a null hypothesis \(H_0\) (the status quo) and an alternative hypothesis \(H_a\) (the claim we want evidence for).
But the decision to reject or fail to reject is based on probability, not certainty. This means errors are always possible.
Two Types of Error
The Decision Matrix
\(\alpha\) and Type I Error
The significance level \(\alpha\) is the probability of committing a Type I error: rejecting a true \(H_0\). It is chosen before the test is conducted.
\(\beta\) and Type II Error
\(\beta\) is the probability of a Type II error: failing to reject \(H_0\) when it is actually false. Unlike \(\alpha\), the value of \(\beta\) depends on the true parameter value, which we typically don't know.
The Power of a Test
Power is the probability of correctly rejecting a false \(H_0\). It measures a test's ability to detect a real effect.
Formal Definitions
Type I vs Type II
| Type I Error | Type II Error | |
|---|---|---|
| What happens | Reject a true \(H_0\) | Fail to reject a false \(H_0\) |
| Also called | False Positive | False Negative |
| Probability | \(\alpha\) | \(\beta\) |
| Controlled by | Choosing \(\alpha\) level | Sample size, effect size, \(\alpha\) |
| Analogy | Convicting an innocent person | Acquitting a guilty person |
How to Identify Errors in Context
Example 1: Drug Trial
A pharmaceutical company tests whether a new drug lowers blood pressure. They set up:
\(H_0\colon \mu = 140\) mmHg (no improvement)
\(H_a\colon \mu < 140\) mmHg (drug works)
Example 2: Fire Alarm System
A building's fire alarm tests the hypothesis:
\(H_0\colon \text{No fire present}\)
\(H_a\colon \text{Fire is present}\)
Example 3: Increasing Power
A researcher tests \(H_0\colon p = 0.50\) vs \(H_a\colon p > 0.50\) with \(n = 40\) and \(\alpha = 0.05\). She finds that \(\beta = 0.35\) when the true proportion is \(p = 0.60\). She wants more power.
The \(\alpha\)-\(\beta\) Trade-Off
Consequences in Context
On the AP exam, you will be asked to describe errors in context and identify which error is more serious. The key is translating the abstract definitions into real-world consequences.
Common Mistakes
Question 1 of 5
A factory tests \(H_0\colon p = 0.02\) (defect rate is 2%) vs \(H_a\colon p > 0.02\). A Type I error would mean:
Question 2 of 5
If a test has \(\alpha = 0.05\) and \(\beta = 0.20\), what is the power of the test?
Question 3 of 5
Which of the following will increase the power of a hypothesis test?
Question 4 of 5
A smoke detector tests \(H_0\colon \text{no fire}\) vs \(H_a\colon \text{fire present}\). Which error is more serious?
Question 5 of 5
A researcher uses \(\alpha = 0.10\) instead of \(\alpha = 0.05\). What happens to the probability of Type I error and the power of the test?