01 / 21

Introducing Statistics:
Should I Worry About Error?

Learn to identify, measure, and evaluate sources of error in statistical studies: from sampling variability to bias, and understand when error truly matters.

AP Statistics

What Is Error in Statistics?

In statistics, error is the difference between what we observe in a sample and the true value of the population parameter. Every study has some error: the question is whether it is small enough to trust our conclusions.

Parameter
The true value for the entire population (often unknown).
Statistic
The value calculated from a sample used to estimate the parameter.

Why Should You Care About Error?

Imagine a poll says 52% of voters support a candidate, but the margin of error is \(\pm 4\%\). The true support could be anywhere from 48% to 56%: the race could go either way. Understanding error tells us how much to trust a result.

48% 52% 56% 50% Point Estimate Margin of Error: ± 4%

Two Families of Error

All statistical error falls into two broad categories. Understanding the difference is essential for designing good studies and interpreting results.

Sampling Variability
Random, expected fluctuation from sample to sample. Unavoidable, but decreases with larger samples. Also called random error.
Bias
A systematic tendency to over- or under-estimate the parameter. Does not decrease with larger samples. Must be designed out of the study.

Bias vs. Variability: The Bullseye

Low Bias Low Variability IDEAL Low Bias High Variability High Bias Low Variability = True Value

The Margin of Error

The margin of error quantifies the expected range of sampling variability. For a proportion at the 95% confidence level:

\( z^* \)
Critical value from the normal distribution for the chosen confidence level.
\( \hat{p} \)
Sample proportion: the fraction of successes observed in the sample.
\( n \)
Sample size. Larger \( n \) means smaller margin of error.

How Sample Size Reduces Error

Because the margin of error has \( \sqrt{n} \) in the denominator, quadrupling the sample size only halves the margin of error. This is the law of diminishing returns.

Sample Size (\(n\))
\(\text{ME}\)
0 250 500 750 1000
\(n=100\)
\(\text{ME}\approx9.8\%\)
\(n=400\)
\(\text{ME}\approx4.9\%\)
\(n=800\)
\(\text{ME}\approx3.5\%\)

Bias vs. Variability: Key Distinction

Sampling Variability Bias
Type Random error Systematic error
Direction Equally likely above or below Consistently one direction
Effect of larger \( n \) Decreases No change
Fix Increase sample size Improve study design
Detected by Repeated sampling Comparison to known values

Common Sources of Bias

Selection Bias
The sampling method systematically excludes certain groups. Example: surveying only people who answer their phones during the day.
Response Bias
Question wording or social pressure causes respondents to answer dishonestly. Example: leading questions.
Nonresponse Bias
People who choose not to respond differ systematically from those who do. Example: surveys with low return rates.

How to Evaluate Error

1
Identify the claim. What parameter is being estimated? What value is reported?
2
Check the margin of error. Is it reported? How wide is the confidence interval relative to the claim?
3
Assess sample size. Is \( n \) large enough to produce a reasonably small margin of error?
4
Look for bias. Was the sample random? Are there selection, response, or nonresponse issues?
5
Decide if it matters. Does the margin of error change the practical conclusion?

Example 1: The School Poll

A school newspaper surveys 100 students and finds that 60% support a new dress code. Compute the margin of error at 95% confidence.

Example 2: The National Survey

A national poll of \( n = 1{,}500 \) adults finds 47% approve of a policy. The poll uses random digit dialing. Should we worry about error?

✓ Variability
\(\pm 2.5\%\) is reasonably small. The large sample helps.
? Bias
Random digit dialing may miss people without landlines (selection bias) or have low response rates (nonresponse bias).

Example 3: The Website Survey

A website posts an open poll asking "Do you support the new tax?" and gets 10,000 responses. The result: 78% say no. The site claims "Americans overwhelmingly oppose the tax."

When Does Error Actually Matter?

Error matters most when the conclusion could flip depending on the true value. Ask: does the confidence interval include values that would change my decision?

Scenario A: Error MATTERS 50% threshold 46% 56% 51% Scenario B: Error DOESN'T matter 50% threshold 60% 70% 65%

Common Mistakes

✕ Big \( n \) Fixes Everything
A huge sample reduces variability but does nothing for bias. A biased survey of 100,000 people is still biased.
✕ Ignoring the ME
Reporting "52% support the candidate" without mentioning \(\pm 4\%\) hides the fact that the race may be a toss-up.
✕ Confusing CI Meaning
A 95% CI does not mean "95% chance the parameter is in this interval." It means 95% of such intervals capture the true value.

Summary: Error at a Glance

Sampling Variability Bias
Nature Random fluctuation Systematic shift
Measured by Margin of error / SE Study design review
Reduced by Increasing \( n \) Better sampling methods
Worry when CI is wide relative to claim Non-random or voluntary sample
Formula captures it? Yes No

Question 1 of 5

A researcher surveys 2,000 randomly selected adults and finds that 62% support a new policy. The margin of error is \(\pm 2.1\%\). Which statement is most accurate?

Question 2 of 5

A website posts an online poll and receives 50,000 responses. The margin of error is tiny: \(\pm 0.4\%\). Why should we still worry about this result?

Question 3 of 5

If a researcher wants to cut the margin of error in half, by what factor must they increase the sample size?

Question 4 of 5

A poll finds that 51% of voters favor Candidate A with a margin of error of \(\pm 3\%\). What is the best interpretation?

Question 5 of 5

Which of the following is a source of bias rather than sampling variability?