Potential Errors When Performing Tests - AP Statistics

SECTION 1

Historical Context & Motivation

Statistical hypothesis testing did not arrive fully formed; it evolved through decades of intellectual debate about how scientists should weigh evidence against uncertainty. The fundamental tension—how do we decide when data are surprising enough to reject a claim?—naturally carries the risk of reaching the wrong conclusion. Two pioneers, Jerzy Neyman and Egon Pearson, formalized the idea that hypothesis tests are decision procedures that can fail in exactly two directions: rejecting a true null hypothesis or failing to reject a false one. Their framework gave statisticians the vocabulary—Type I and Type II errors—that remains central to every AP Statistics course and every modern research study.

1900s • Foundations of Significance Testing

Karl Pearson develops the chi-square goodness-of-fit test, introducing the concept of measuring how well observed data conform to a theoretical distribution—an early step toward formalized hypothesis testing.

1925 • Fisher's p-value Framework

Ronald Fisher publishes "Statistical Methods for Research Workers," popularizing p-values and the notion of significance levels. However, Fisher's framework does not formally define an alternative hypothesis or error types.

1933 • Neyman–Pearson Lemma

Jerzy Neyman and Egon Pearson publish their foundational paper, formalizing hypothesis testing as a decision between two hypotheses and explicitly naming Type I error (α) and Type II error (β). This framework introduces the concept of statistical power.

1945–1960 • Adoption in Applied Sciences

Medical trials, quality control in manufacturing, and social science research adopt the Neyman–Pearson decision framework, making error control a practical concern in fields far beyond pure mathematics.

2000s–Present • Replication Crisis & Renewed Focus

Widespread failures to replicate published findings across psychology and medicine prompt a reexamination of how researchers handle Type I and Type II error rates, reinforcing the importance of understanding these concepts.

The central question this lesson addresses is deceptively simple: when we perform a hypothesis test for a population proportion, what can go wrong with our conclusion, how do we quantify those risks, and what levers do we have to control them? Answering this question is not merely an academic exercise—it is the foundation for making responsible inferences from data in medicine, policy, and science.

SECTION 2

Core Principles & Definitions

Every hypothesis test begins with two competing claims about a population parameter: the null hypothesis (H₀), which represents the status quo, and the alternative hypothesis (Hₐ), which represents the researcher's claim. Because we base our decision on sample data rather than a census, there is always the possibility that the data mislead us. Two distinct mistakes can occur, and they are not symmetric in their consequences.

Type I Error (False Positive)

Rejecting H₀ when H₀ is actually true. The probability of committing this error equals α, the significance level chosen before the test is conducted.

Type II Error (False Negative)

Failing to reject H₀ when H₀ is actually false. The probability of this error is denoted β, which depends on the true value of the parameter and the sample size.

Significance Level (α)

The pre-set threshold for how much evidence against H₀ we require before rejecting it. Common choices are α = 0.05 and α = 0.01. Lowering α reduces the Type I error rate but increases the Type II error rate.

Power (1 − β)

The probability of correctly rejecting H₀ when it is false. High power means the test is sensitive enough to detect a real departure from the null. Power increases with larger sample sizes and larger true effect sizes.

✦

KEY TAKEAWAY

Think of a hypothesis test like a smoke detector. A Type I error is a false alarm—the detector goes off when there is no fire. A Type II error is a missed alarm—the detector stays silent while a fire is burning. You can make the detector more sensitive (lower α threshold), but then it is more likely to cry wolf. The engineering challenge—just like the statistical one—is finding the right balance between sensitivity and specificity.

SECTION 3

Visual Explanation: The Decision Matrix

The relationship between reality and our test decision can be summarized in a 2 × 2 decision matrix. The columns represent the true state of the world (which we never know for certain), and the rows represent our decision based on sample data. Two cells are correct outcomes, and two cells are errors. The diagram below illustrates this framework with color coding: green for correct decisions and red/amber for errors.

The decision matrix shows four possible outcomes of a hypothesis test. The upper-left cell (Type I error) and lower-right cell (Type II error) represent incorrect conclusions. The two green cells represent correct decisions. Notice that the probability of a Type I error is entirely set by the researcher's choice of α, while the probability of a Type II error is influenced by multiple factors.

A critical insight from this matrix is that you can never make both errors simultaneously—they arise under mutually exclusive realities. If H₀ is true, the only error possible is Type I. If H₀ is false, the only error possible is Type II. This is why reducing α (moving the rejection threshold) inevitably affects β: when you demand stronger evidence before rejecting H₀, you become more cautious about false positives but more vulnerable to missing real effects.

SECTION 4

Mathematical Framework

When performing a one-proportion z-test, we compute a test statistic and compare it to the standard normal distribution. The rejection region is determined by our chosen significance level α. Understanding the mathematics behind the error probabilities clarifies exactly how they are connected to sample size, the true proportion, and the significance level.

TEST STATISTIC FOR ONE-PROPORTION Z-TEST

z = (p̂ − p₀) / √(p₀(1 − p₀) / n)

Where p̂ is the sample proportion, p₀ is the hypothesized proportion under H₀, and n is the sample size.

TYPE I ERROR PROBABILITY

P(Type I Error) = P(reject H₀ | H₀ is true) = α

This is a direct consequence of our choice: by setting α = 0.05, we accept that 5% of the time we will incorrectly reject a true null hypothesis.

TYPE II ERROR PROBABILITY

β = P(fail to reject H₀ | H₀ is false)

β depends on the true proportion p, the hypothesized p₀, the sample size n, and the significance level α. There is no single formula to memorize; rather, β is computed by finding the probability that the test statistic falls outside the rejection region under the true parameter value.

POWER OF THE TEST

Power = 1 − β = P(reject H₀ | H₀ is false)

Power increases when (1) the sample size n increases, (2) the true proportion is farther from p₀ (larger effect size), or (3) α is increased. Researchers typically aim for power ≥ 0.80.

The interplay between α and β is often described as a tradeoff: for a fixed sample size, decreasing α (demanding more stringent evidence) shifts the critical value farther from the center of the null distribution, making it harder to reach the rejection region even when H₀ is false, thereby increasing β. The only way to decrease both α and β simultaneously is to increase the sample size, which narrows both distributions and separates them more cleanly.

SECTION 5

Factors Affecting Errors & Real-World Consequences

Understanding the factors that influence each type of error is essential for designing responsible studies and interpreting results correctly. The diagram below illustrates how key factors shift the balance between Type I and Type II error rates, with particular attention to the proportions context on the AP Statistics exam.

This diagram shows four factors that influence error rates. The significance level (α) is the only factor that directly controls Type I error. Sample size, effect size, and power all affect Type II error. The dashed line between the two error circles emphasizes the inherent tradeoff.

Contextualizing Errors in Proportion Tests

On the AP Statistics exam, you will often be asked to describe what a Type I or Type II error means in context. This means you must translate the abstract definitions into a specific scenario. For example, suppose a pharmaceutical company tests whether the proportion of patients who experience side effects from a new drug exceeds 0.10. A Type I error would mean concluding that the side-effect rate exceeds 0.10 when it actually does not—potentially pulling a safe drug from the market. A Type II error would mean failing to detect that the side-effect rate exceeds 0.10 when it truly does—allowing a harmful drug to remain available. The consequences of each error are asymmetric, and this asymmetry should inform the choice of α.

Factor	Effect on Type I Error (α)	Effect on Type II Error (β)
Increase α (e.g., 0.05 → 0.10)	Increases (more false positives)	Decreases (fewer missed effects)
Decrease α (e.g., 0.05 → 0.01)	Decreases (fewer false positives)	Increases (more missed effects)
Increase sample size n	No change (still equals α)	Decreases (more power)
Larger true effect \|p − p₀\|	No change (still equals α)	Decreases (easier to detect)

Summary of how key factors influence each error type in a one-proportion z-test

SECTION 6

Worked Example: Errors in a Proportion Test

A school board claims that 60% of parents in the district support a proposed curriculum change. A parent group suspects the true proportion is less than 0.60 and surveys a random sample of 200 parents. They plan to test at the α = 0.05 significance level. Out of the 200 sampled parents, 108 express support (p̂ = 0.54). Let us walk through the hypothesis test and identify what errors would mean in context.

Testing a Claim About Parent Support

1
Step 1 — State Hypotheses

H₀: p = 0.60 (the true proportion of parents who support the change is 0.60). Hₐ: p < 0.60 (the true proportion is less than 0.60). This is a one-sided (left-tailed) test.
2
Step 2 — Check Conditions

Random: The sample is stated to be random. Independence: n = 200 is almost certainly less than 10% of all parents in the district (10% condition). Normal: np₀ = 200 × 0.60 = 120 ≥ 10 and n(1 − p₀) = 200 × 0.40 = 80 ≥ 10. All conditions are met.
3
Step 3 — Compute the Test Statistic

z = (p̂ − p₀) / √(p₀(1 − p₀) / n) = (0.54 − 0.60) / √(0.60 × 0.40 / 200) = −0.06 / √(0.0012) = −0.06 / 0.03464 ≈ −1.732.

z ≈ −1.732
4
Step 4 — Find the p-value and Make a Decision

Using the standard normal table or a calculator, P(Z < −1.732) ≈ 0.0416. Since 0.0416 < 0.05 = α, we reject H₀. There is sufficient evidence at the 5% significance level to conclude that less than 60% of parents support the curriculum change.

p-value ≈ 0.0416 → Reject H₀
5
Step 5 — Describe Errors in Context

Type I Error: We concluded that fewer than 60% of parents support the change, but in reality, 60% (or more) actually do. Consequence: the school board may abandon a popular initiative based on misleading sample data. Type II Error: We failed to reject the claim that 60% support the change, but in reality, fewer than 60% actually support it. Consequence: the school board proceeds with a curriculum change that lacks majority support.

Always describe errors using the specific context of the problem, not just α and β.

💡

AP Exam Tip

On free-response questions, the AP readers require you to describe Type I and Type II errors in context. Simply writing "rejecting H₀ when H₀ is true" will not earn full credit. You must reference the specific parameter and scenario—for instance, "concluding that the proportion of parents who support the change is less than 0.60 when it is actually 0.60."

SECTION 7

Strengths, Limitations & the Error Tradeoff

The Neyman–Pearson framework gives us a powerful and structured way to make decisions under uncertainty, but it comes with inherent limitations. Understanding when the framework works well and when it can mislead you is critical for interpreting results responsibly, both on the AP exam and in professional practice.

Strengths	Limitations
Provides a clear, reproducible decision rule (reject or fail to reject) based on pre-specified α.	The binary decision (reject/fail to reject) discards information about the strength of evidence beyond the threshold.
Type I error rate is directly controlled by the researcher's choice of α.	Type II error rate is harder to control and depends on unknown quantities (the true parameter value).
Power analysis allows researchers to plan adequate sample sizes before collecting data.	A statistically significant result does not necessarily imply practical significance; a tiny but real difference can be detected with large n.
The framework is widely understood and accepted, enabling standardized communication across disciplines.	Multiple testing inflates the overall Type I error rate unless corrections (e.g., Bonferroni) are applied.

Strengths and limitations of the Neyman–Pearson hypothesis testing framework

⚖️

KEY TAKEAWAY

Choosing α is like calibrating a scale of justice. In a criminal trial, we set a very high bar for conviction (small α) because a Type I error (convicting an innocent person) is considered worse than a Type II error (acquitting a guilty person). In contrast, a screening test for a dangerous disease might use a larger α because missing a sick patient (Type II) could be fatal. The choice is always driven by which error has worse consequences in the given context.

SECTION 8

Connection to Advanced Topics

The concepts of Type I error, Type II error, and power extend well beyond one-proportion z-tests. On the AP Statistics exam and in college-level coursework, you will encounter these ideas in the context of two-proportion z-tests, chi-square tests for homogeneity and independence, and t-tests for means. The underlying logic of the decision matrix remains identical—only the test statistic and distribution change.

Topic in This Lesson	Advanced Extension
Type I / Type II errors in a single proportion test	Errors in two-proportion z-tests and chi-square tests (same decision matrix, different test statistics)
Fixed significance level α	Multiple comparison corrections (Bonferroni, Benjamini–Hochberg) that adjust α when performing many tests simultaneously
Power as 1 − β	Formal power analysis and sample size determination formulas used in research design
Consequences of errors described in context	Decision theory and loss functions that assign numeric costs to each type of error for optimal decision-making

How this lesson's concepts connect to advanced statistics topics

Looking ahead, introductory courses in Bayesian statistics offer a fundamentally different perspective on inference: rather than classifying decisions into binary error categories, Bayesian methods update the probability of a hypothesis given the data. However, the frequentist error framework you learn in AP Statistics remains the dominant paradigm in many scientific fields, and a deep understanding of its strengths and pitfalls will serve you well in any future statistics course.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher tests H₀: p = 0.40 versus Hₐ: p > 0.40 at the α = 0.05 level. She rejects H₀. Later, it is discovered that the true population proportion is actually 0.40. Which type of error has been committed?

A. Type I error, because she rejected H₀ when H₀ was actually true.
B. Type II error, because she rejected H₀ when H₀ was actually true.
C. Type I error, because she failed to reject H₀ when Hₐ was actually true.
D. Type II error, because she failed to reject H₀ when Hₐ was actually true.
E. No error has been committed because her p-value was below 0.05.

Show answer (A)

The correct answer is A. A Type I error occurs when H₀ is rejected even though H₀ is true. Since the researcher rejected H₀: p = 0.40 and the true proportion turns out to be 0.40, she committed a Type I error. Choice B is wrong because it correctly identifies the action (rejecting H₀ when it is true) but mislabels it as a Type II error. Choice C is wrong because the researcher did reject H₀, not fail to reject it—and the description actually matches a correct decision scenario. Choice D describes the definition of a Type II error (failing to reject H₀ when Hₐ is true), which does not match the scenario since she did reject. Choice E is wrong because having a p-value below α does not guarantee a correct decision—it means the result is statistically significant, but the null hypothesis could still be true.

Question 2: BASIC CALCULATION

A city health department tests H₀: p = 0.15 versus Hₐ: p ≠ 0.15 at α = 0.01, where p is the proportion of restaurant inspections that reveal critical violations. If the test has a power of 0.72 against the true proportion p = 0.20, what is the probability of a Type II error?

A. 0.01
B. 0.15
C. 0.28
D. 0.72
E. 0.99

Show answer (C)

The correct answer is C. Power is defined as 1 − β, so β = 1 − Power = 1 − 0.72 = 0.28. This means there is a 28% probability that the test will fail to detect the true proportion of 0.20. Choice A (0.01) is the significance level α, which is the probability of a Type I error, not Type II. Choice B (0.15) is the hypothesized null proportion, not a probability of error. Choice D (0.72) is the power itself, not β. Choice E (0.99) equals 1 − α, which is the probability of correctly failing to reject H₀ when it is true—not the probability of a Type II error.

Question 3: INTERMEDIATE

A polling organization tests H₀: p = 0.50 versus Hₐ: p > 0.50 at α = 0.05 using a sample of n = 400 voters. They obtain p̂ = 0.535 and compute a p-value of 0.081. The organization fails to reject H₀. However, the true proportion of voters who support the ballot measure is p = 0.54. Describe the error (if any) that has been made, identify which type of error it is, and describe the consequence of this error in context.

A. Type I error: the organization concluded a majority supports the measure when in fact p = 0.50.
B. Type II error: the organization failed to detect that the true proportion (0.54) exceeds 0.50, potentially underrepresenting actual voter support.
C. No error was made because the p-value (0.081) is close to α.
D. Type I error: the p-value exceeded α, so the test incorrectly retained the null.
E. Type II error: the organization rejected H₀ when the true proportion was actually 0.54.

Show answer (B)

The correct answer is B. Since the true proportion is 0.54 > 0.50, the alternative hypothesis is true. The organization failed to reject H₀, which means they made a Type II error—they missed the real effect. In context, this means the polling organization concluded there is not convincing evidence that a majority supports the ballot measure, even though a majority actually does (p = 0.54). This could lead campaigns to underinvest in promoting the measure. Choice A is incorrect because the organization did not reject H₀—they failed to reject it. Choice C is wrong because closeness to α does not prevent an error; the decision was still to not reject, and H₀ is false. Choice D misidentifies the error type; failing to reject a true null would be a correct decision, not a Type I error. Choice E is wrong because the organization did not reject H₀ in this scenario.

Question 4: APPLIED

6 pts

A quality control engineer at a bottling plant monitors the proportion of defective bottles. The current standard is that no more than 3% of bottles are defective. Each day, the engineer takes a random sample of 500 bottles and tests H₀: p = 0.03 versus Hₐ: p > 0.03 at α = 0.05. (a) Describe what a Type I error and a Type II error would mean in this context, including the consequences of each. (b) The engineer considers lowering α to 0.01. Explain the effect this change would have on the probabilities of Type I and Type II errors, and discuss whether this is a good idea given the consequences you described in part (a). (c) Instead of changing α, the engineer proposes increasing the daily sample size from 500 to 1,000 bottles. Explain how this change affects the probability of each type of error.

Show answer

RUBRIC (6 points): Part (a) — 2 points: • (1 point) Type I error: The engineer concludes that the defective rate exceeds 3% when it is actually 3% or less. Consequence: The plant may halt production or recalibrate equipment unnecessarily, wasting time and money. • (1 point) Type II error: The engineer fails to detect that the defective rate exceeds 3% when it truly does. Consequence: Defective bottles continue to reach consumers, risking product quality and customer safety. Part (b) — 2 points: • (1 point) Lowering α from 0.05 to 0.01 decreases the probability of a Type I error (from 0.05 to 0.01) but increases the probability of a Type II error (β increases because the rejection region becomes smaller, requiring more extreme evidence). • (1 point) Evaluation: Given that a Type II error means defective products reach consumers (a safety concern), lowering α is likely not a good trade because it makes the test less sensitive to real quality problems. The consequences of missing a true increase in defect rate may outweigh the costs of an occasional false alarm. Part (c) — 2 points: • (1 point) Increasing n from 500 to 1,000 does NOT change the probability of Type I error, which remains equal to α = 0.05. • (1 point) Increasing n decreases the probability of Type II error (increases power) because the standard error of p̂ decreases with larger n, making the sampling distribution narrower and the test more sensitive to departures from p₀ = 0.03. This is the preferred approach because it improves detection without increasing false positives.

Question 5: CRITICAL THINKING

7 pts

A medical researcher is designing a study to test whether a new treatment changes the proportion of patients who achieve remission from a disease. The current remission rate with standard treatment is p₀ = 0.30. She tests H₀: p = 0.30 versus Hₐ: p ≠ 0.30 at α = 0.05. (a) If the true remission rate with the new treatment is p = 0.35, explain why a sample of n = 100 might have low power. Use the standard error formula to support your reasoning. (b) The researcher argues: "If my test is not significant, it means the new treatment has no effect." Critique this statement by referencing the concepts of Type II error and power. (c) A colleague suggests that if the researcher runs the same test on 20 different patient subgroups at α = 0.05, approximately one subgroup will show a significant result by chance alone even if the treatment truly has no effect. Explain why this is the case and name one method for addressing this problem. (d) Given that approving an ineffective treatment wastes resources but failing to approve an effective treatment costs lives, argue for a specific choice of α and justify your reasoning in terms of Type I and Type II error consequences.

Show answer

RUBRIC (7 points): Part (a) — 2 points: • (1 point) The standard error under H₀ is √(0.30 × 0.70 / 100) = √(0.0021) ≈ 0.0458. The difference between the true proportion (0.35) and p₀ (0.30) is only 0.05, which is barely one standard error away from p₀. • (1 point) Because the effect size (0.05) is small relative to the standard error (0.0458), the sampling distribution under the true proportion p = 0.35 overlaps substantially with the sampling distribution under H₀, making it difficult to reject H₀. The test has low power because p̂ values consistent with H₀ are also very plausible under the true parameter. Part (b) — 2 points: • (1 point) The statement is incorrect. Failing to reject H₀ does not prove H₀ is true—it means there is not enough evidence to reject it. A nonsignificant result could be a Type II error: the treatment may genuinely improve remission, but the test was not powerful enough to detect the improvement. • (1 point) Power determines the probability of detecting a real effect. If power is low (e.g., below 0.80), there is a substantial probability (β > 0.20) of failing to reject H₀ even when the treatment works. The researcher should report the power of her test or provide a confidence interval rather than interpreting nonsignificance as evidence of no effect. Part (c) — 2 points: • (1 point) Under H₀ (no treatment effect), each test has a 5% chance of producing a significant result by random chance. With 20 independent tests, the expected number of false significant results is 20 × 0.05 = 1. The probability of at least one false positive is 1 − (0.95)²⁰ ≈ 0.64, so there is a 64% chance of at least one spurious significant result. • (1 point) One method to address this is the Bonferroni correction, which adjusts the significance level to α/k = 0.05/20 = 0.0025 for each individual test, keeping the overall (family-wise) Type I error rate at 0.05. Part (d) — 1 point: • (1 point) A well-reasoned argument for a specific α value with clear justification. For example: Given that failing to approve an effective treatment costs lives (a severe Type II error consequence), a moderate α such as 0.05 or even 0.10 may be appropriate for a preliminary trial. A very small α (e.g., 0.01) would reduce false positives but increase the chance of missing a life-saving treatment. The researcher should balance these consequences and plan an adequate sample size to achieve high power (at least 0.80) at the chosen α. Credit is given for any well-justified choice that correctly links the consequences to the error types.

SUMMARY

Lesson Summary

Summary

Every hypothesis test about a population proportion carries the risk of two types of errors. A Type I error (false positive) occurs when we reject H₀ even though it is true, and its probability equals the significance level α. A Type II error (false negative) occurs when we fail to reject H₀ even though the alternative hypothesis Hₐ is true, and its probability is denoted β. The power of the test (1 − β) measures the test's ability to correctly detect a real effect and increases with larger sample sizes and larger effect sizes.

The fundamental tradeoff is that decreasing α reduces Type I error but increases Type II error for a fixed sample size. The only way to reduce both simultaneously is to increase the sample size. On the AP exam, always describe errors in the specific context of the problem—stating what it would mean to reject or fail to reject H₀ if you are wrong—and consider which error has more serious practical consequences when justifying your choice of significance level.