01 / 21

Potential Errors in
Hypothesis Testing

Understanding Type I and Type II errors, the relationship between \(\alpha\), \(\beta\), and power, and how to minimize mistakes in statistical inference.

AP Statistics

What Is a Hypothesis Test?

A hypothesis test uses sample data to decide between two competing claims about a population parameter. We start with a null hypothesis \(H_0\) (the status quo) and an alternative hypothesis \(H_a\) (the claim we want evidence for).

But the decision to reject or fail to reject is based on probability, not certainty. This means errors are always possible.

Two Types of Error

Type I Error
Rejecting \(H_0\) when \(H_0\) is actually true. A "false positive": we claim an effect exists when it doesn't.
Type II Error
Failing to reject \(H_0\) when \(H_0\) is actually false. A "false negative": we miss a real effect.

The Decision Matrix

Reality
\(H_0\) True
\(H_0\) False
Decision
Reject \(H_0\)
Fail to reject \(H_0\)
Type I Error
P = \(\alpha\)
Correct!
P = Power = \(1-\beta\)
Correct!
P = \(1-\alpha\)
Type II Error
P = \(\beta\)
False Positive False Negative Correct Decision

\(\alpha\) and Type I Error

The significance level \(\alpha\) is the probability of committing a Type I error: rejecting a true \(H_0\). It is chosen before the test is conducted.

1
\(\alpha = 0.05\) means a 5% chance of a false positive: you'd wrongly reject \(H_0\) in about 1 out of 20 tests when \(H_0\) is true.
2
\(\alpha = 0.01\) is stricter: only 1% chance of false positive. Used when the consequence of a wrong rejection is severe.
3
\(\alpha = 0.10\) is more lenient: 10% false positive rate. Used when missing a real effect is the greater concern.

\(\beta\) and Type II Error

\(\beta\) is the probability of a Type II error: failing to reject \(H_0\) when it is actually false. Unlike \(\alpha\), the value of \(\beta\) depends on the true parameter value, which we typically don't know.

\(H_0\) distribution
\(H_a\) distribution
Critical value
\(\alpha\)
\(\beta\)
Area under \(H_a\) curve to LEFT of critical value = \(\beta\)

The Power of a Test

Power is the probability of correctly rejecting a false \(H_0\). It measures a test's ability to detect a real effect.

Increase \(n\)
Larger sample sizes reduce variability and increase power.
Increase \(\alpha\)
A higher significance level makes it easier to reject \(H_0\), boosting power.
Larger effect
A bigger true difference from \(H_0\) separates the distributions more, raising power.

Formal Definitions

1
Type I Error: \(P(\text{reject } H_0 \mid H_0 \text{ is true}) = \alpha\)
2
Type II Error: \(P(\text{fail to reject } H_0 \mid H_0 \text{ is false}) = \beta\)
3
Power: \(P(\text{reject } H_0 \mid H_0 \text{ is false}) = 1 - \beta\)

Type I vs Type II

Type I Error Type II Error
What happens Reject a true \(H_0\) Fail to reject a false \(H_0\)
Also called False Positive False Negative
Probability \(\alpha\) \(\beta\)
Controlled by Choosing \(\alpha\) level Sample size, effect size, \(\alpha\)
Analogy Convicting an innocent person Acquitting a guilty person

How to Identify Errors in Context

1
State the hypotheses. Write \(H_0\) and \(H_a\) in context (what does the null claim?).
2
Describe Type I Error in context. "We conclude [effect], but in reality [no effect]." This is a false alarm.
3
Describe Type II Error in context. "We fail to find evidence of [effect], but in reality [effect exists]." This is a missed detection.
4
Identify consequences. Which error is worse in this scenario? This helps choose \(\alpha\).
5
Connect to power. If the scenario asks how to reduce a specific error, think about \(n\), \(\alpha\), and effect size.

Example 1: Drug Trial

A pharmaceutical company tests whether a new drug lowers blood pressure. They set up:
\(H_0\colon \mu = 140\) mmHg (no improvement)
\(H_a\colon \mu < 140\) mmHg (drug works)

Type I Error
Conclude the drug lowers blood pressure when it actually doesn't. Consequence: patients take an ineffective drug, potential side effects without benefit.
Type II Error
Fail to detect that the drug works when it actually does. Consequence: an effective treatment is abandoned, patients miss potential benefit.

Example 2: Fire Alarm System

A building's fire alarm tests the hypothesis:
\(H_0\colon \text{No fire present}\)
\(H_a\colon \text{Fire is present}\)

Type I Error
Alarm sounds when there is no fire. Consequence: unnecessary evacuation, disruption, but everyone is safe.
Type II Error
Alarm fails to sound when there IS a fire. Consequence: potential loss of life and property.

Example 3: Increasing Power

A researcher tests \(H_0\colon p = 0.50\) vs \(H_a\colon p > 0.50\) with \(n = 40\) and \(\alpha = 0.05\). She finds that \(\beta = 0.35\) when the true proportion is \(p = 0.60\). She wants more power.

1
Increase sample size: Going from \(n = 40\) to \(n = 100\) reduces variability in \(\hat{p}\), making it easier to detect the true \(p = 0.60\).
2
Increase \(\alpha\): Changing from 0.05 to 0.10 shifts the critical value, making rejection easier, but also increases the false positive risk.
3
Larger effect: If the true \(p\) were 0.70 instead of 0.60, the distributions separate more and power increases automatically.

The \(\alpha\)-\(\beta\) Trade-Off

\(\alpha\) (significance level)
Probability 0.01 0.05 0.10 0.20
\(\beta\)
Power
Typical \(\alpha\)

Consequences in Context

On the AP exam, you will be asked to describe errors in context and identify which error is more serious. The key is translating the abstract definitions into real-world consequences.

1
Type I in context: Start with "We conclude that [alternative claim], but in reality [null is true]." Then state the real-world harm.
2
Type II in context: Start with "We fail to find evidence that [alternative claim], but in reality [alternative is true]." Then state the missed opportunity or danger.
3
Which is worse? Consider: does the consequence involve safety, money, health, or justice? The more severe consequence should guide the choice of \(\alpha\).

Common Mistakes

✕ Confusing the two errors
Type I = reject a true \(H_0\) (false positive). Type II = fail to reject a false \(H_0\) (false negative). Many students swap them.
✕ Not writing in context
Saying "reject \(H_0\) when it's true" earns no credit. You must describe what that means in the specific problem scenario.
✕ Saying "accept \(H_0\)"
We never "accept" \(H_0\). We "fail to reject" it. Accepting implies we've proven it true, which we cannot do.

Question 1 of 5

A factory tests \(H_0\colon p = 0.02\) (defect rate is 2%) vs \(H_a\colon p > 0.02\). A Type I error would mean:

Question 2 of 5

If a test has \(\alpha = 0.05\) and \(\beta = 0.20\), what is the power of the test?

Question 3 of 5

Which of the following will increase the power of a hypothesis test?

Question 4 of 5

A smoke detector tests \(H_0\colon \text{no fire}\) vs \(H_a\colon \text{fire present}\). Which error is more serious?

Question 5 of 5

A researcher uses \(\alpha = 0.10\) instead of \(\alpha = 0.05\). What happens to the probability of Type I error and the power of the test?