01 / 21

How Collected Data
Tells The Truth

Understanding when your data accurately represents reality and how bias, variability, and study design determine whether conclusions are trustworthy.

AP Statistics

What Does "Truth" Mean in Statistics?

In statistics, "truth" means our collected data faithfully reflects the population we are studying. Every dataset is just a snapshot: the question is whether that snapshot is clear and accurate or blurry and misleading.

Key Vocabulary You Need

Population
The entire group you want to study or make conclusions about.
Sample
A subset of the population actually observed or measured.
Parameter
A number that describes a characteristic of the entire population (often unknown).
Statistic
A number computed from the sample that estimates the parameter.

From Population to Conclusion

Population (parameter \(\mu\)) Sampling Sample (statistic \(\bar{x}\)) Compute Analysis \(\bar{x}, s, \hat{p}\) Conclusion About population Inference Does it tell the truth?

Bias vs. Variability: The Two Enemies

Two forces can prevent data from telling the truth. Bias makes your estimates systematically too high or too low. Variability makes your estimates scattered and imprecise from sample to sample.

Low Bias Low Variability IDEAL Low Bias High Variability High Bias Low Variability CONSISTENTLY WRONG

Common Sources of Bias

Selection Bias
The sample systematically excludes parts of the population. Example: surveying only honors students about school satisfaction.
Response Bias
Wording or context of questions pushes respondents toward certain answers. Example: leading questions in a survey.
Nonresponse Bias
People who don't respond differ systematically from those who do. Example: only satisfied customers reply to feedback forms.

A neighborhood survey with visible routes for missing voices

A neighborhood survey shows homes, invitation routes and returned responses.A survey can lose people before selection or after invitation. Keeping the target population, sampling frame, selected sample and respondents separate reveals where each problem arises.

Track who is missing at each stage

A 24-ID population narrows to a 20-ID frame, eight selected invitations and six responses.Suppose the target population has 24 IDs. The frame omits IDs 21–24; eight listed IDs are selected, and six respond. Frame omissions are undercoverage; selected people who do not respond are nonresponse. Bias direction depends on how missing responses differ.

Participation does not remove wording effects

Two versions of a survey question contrast neutral phrasing with wording that encourages a particular response.A respondent can still be influenced by the wording of a question. Compare the neutral and leading versions by identifying the assumption each asks the respondent to accept.

Reducing Variability with Sample Size

Larger samples produce less variability in statistics. The standard error of the sample mean decreases as \(n\) increases:

Sample Size (\(n\)) SE \(n=1\): High SE \(n=10\) \(n=40\): Low SE 1 10 25 50

Observational vs. Experimental Studies

Observational Study Experiment
Researcher role Observes without intervening Imposes a treatment
Causation? Cannot establish Can establish
Confounding Likely present Controlled via randomization
Example Survey of coffee drinkers' heart rates Randomly assign caffeine vs. placebo

The Danger of Confounding Variables

A confounding variable is an unmeasured factor that is associated with both the explanatory variable and the response variable. It makes you think X causes Y, when really Z is driving both.

Confounder (Z) Explanatory (X) Response (Y) Misleading association

Principles of Good Study Design

1
Randomization: Use random selection (for surveys) or random assignment (for experiments) to eliminate systematic bias.
2
Replication: Use a large enough sample so results are reliable and variability is low.
3
Control: Hold other variables constant or use a control group so you can isolate the effect of interest.
4
Blinding: Keep subjects (single-blind) or both subjects and researchers (double-blind) unaware of group assignments to prevent placebo effects.

How to Evaluate Whether Data Tells the Truth

1
Identify the population and sample. Who is being studied? Who was actually measured?
2
Check for bias. Was the sample randomly selected? Are there leading questions or nonresponse issues?
3
Assess variability. Is \(n\) large enough? Larger \(n\) means smaller SE and more precise estimates.
4
Check for confounding. Could a lurking variable explain the observed association?
5
Determine scope of conclusions. Can you generalize? Can you claim causation?

Example 1: School Lunch Survey

A school wants to know if students are satisfied with lunch options. They survey every student who visits the cafeteria during one lunch period on a Tuesday.

Example 2: Exercise and Grades

A researcher observes that students who exercise regularly have higher GPAs. She concludes: "Exercise causes higher grades."

Self-discipline (Z) Exercise (X) GPA (Y) Observational: no causation

Example 3: Well-Designed Experiment

Researchers want to test whether a new study app improves test scores. They randomly assign 200 students to use the app or a traditional textbook for 4 weeks, then compare exam results.

Random Assignment
Eliminates systematic differences between groups: confounders are balanced.
Replication
\(n = 200\) is large enough to detect meaningful differences with low variability.
Control Group
Textbook group serves as baseline: any score difference is due to the app.

Common Mistakes to Avoid

✕ Confusing Correlation with Causation
Just because two variables are associated doesn't mean one causes the other. Always check for confounders and study type.
✕ Voluntary Response Samples
Online polls and call-in surveys attract people with strong opinions, creating extreme bias in results.
✕ Generalizing from a Convenience Sample
Surveying only people nearby (friends, classmates) produces a non-representative sample that cannot generalize to the population.

Summary: When Does Data Tell the Truth?

Condition Data Tells Truth Data Misleads
Sampling method Random selection Convenience or voluntary
Sample size Large \(n\): low variability Small \(n\): high variability
Bias Minimal: fair questions, full participation Selection, response, or nonresponse bias
Confounders Controlled via randomization Lurking variables ignored
Causation claim Only from experiments with random assignment Claimed from observational data

Question 1 of 5

A researcher finds that people who own pets tend to have lower blood pressure. She concludes that pet ownership causes lower blood pressure. What is the primary flaw in this conclusion?

Question 2 of 5

A news website posts a poll asking readers: "Do you support the new tax proposal?" and 82% say no. Why should we be skeptical of this result?

Question 3 of 5

If the population standard deviation is \(\sigma = 12\) and we take a sample of size \(n = 36\), what is the standard error of the sample mean?

Question 4 of 5

Which of the following is the best way to reduce bias in a sample survey?

Question 5 of 5

Based on this study, which conclusion is most justified?