01 / 22

Introduction to
Experimental Design

Learn the foundational principles of controlled experiments: randomization, replication, and control, and how they let us establish cause-and-effect relationships.

AP Statistics

What Is an Experiment?

An experiment is a study in which the researcher deliberately imposes a treatment on individuals (called experimental units) in order to observe a response. Unlike observational studies, experiments allow us to draw conclusions about causation.

Core Vocabulary

Explanatory Variable
The variable the researcher manipulates (also called the factor).
Response Variable
The outcome measured to assess the effect of the treatment.
Treatment
A specific condition applied to units: a particular level (or combination of levels) of the factor(s).

Why Experiments Matter

Observational studies can reveal associations, but they cannot establish cause and effect. Lurking variables (confounders) may explain the relationship. Experiments solve this problem by using random assignment to balance out all other variables, isolating the treatment effect.

OBSERVATIONAL STUDY Variable X Variable Y association? Lurking variable? EXPERIMENT Treatment Response causation Random assignment balances confounders

The Three Principles of Experimental Design

1. Control
Keep all variables other than the factor the same across treatment groups so differences in response can be attributed to the treatment.
2. Randomization
Use a chance process to assign units to treatments, which balances both known and unknown confounders.
3. Replication
Apply each treatment to enough units so that the natural variation in responses can be distinguished from a true treatment effect.

Why Randomization Is Essential

Random assignment is the mechanism that allows us to conclude causation. By assigning units to groups via chance, we ensure that every potential confounding variable is roughly evenly distributed across treatment groups, on average.

All Subjects RANDOMIZE Treatment Group Receives treatment Control Group Placebo / no treatment Compare Responses

Experiment vs Observational Study

Experiment Observational Study
Treatment Imposed by researcher Not imposed: researcher observes
Assignment Random assignment to groups Subjects self-select or pre-existing groups
Causation Can establish cause-and-effect Cannot establish cause-and-effect
Confounders Balanced by randomization May be present and uncontrolled
Example Randomly assign a drug vs placebo Survey smokers vs non-smokers

Control Groups and the Placebo Effect

A control group provides a baseline for comparison. In medical and behavioral experiments, subjects may improve simply because they believe they are being treated: this is the placebo effect.

Placebo
A dummy treatment that looks identical to the real treatment but has no active ingredient (e.g., a sugar pill).
Blinding
Single-blind: subjects do not know which group they are in. Double-blind: neither subjects nor evaluators know.

Completely Randomized Design (CRD)

The simplest experimental design: all experimental units are randomly assigned to treatment groups with no restrictions. Each unit has an equal chance of receiving any treatment.

30 Subjects RANDOMIZE
Group A (\(n=10\))
Treatment 1
Group B (\(n=10\))
Treatment 2
Group C (\(n=10\))
Control Compare Responses

Randomized Block Design

When a known variable (such as age or gender) is likely to affect the response, subjects are first grouped into blocks of similar individuals. Then, within each block, random assignment to treatments occurs separately.

All Subjects Block 1: Males Block 2: Females RANDOMIZE RANDOMIZE Treatment Control Treatment Control Compare Responses Within and Across Blocks

Light defines the greenhouse blocks

A researcher examines a seedling in a greenhouse with brighter upper shelving and shaded lower shelving.The top shelf receives more light than the lower shelf. Place both fertilizer treatments on each shelf, control watering, soil and pot size, and measure height increase over four weeks.

Randomize within each shelf

Thirty unique plant IDs: T01–T15 on the brighter top shelf and B01–B15 on the lower shelf. Blue indicates new fertilizer and white standard; each shelf contains both treatments.For 30 seedlings, form two 15-plant shelf blocks and randomize both fertilizers within each shelf. The shown allocation assigns 7 new and 8 standard on top, 8 new and 7 standard below. Compare treatment growth within shelves so the light difference stays in the analysis.Open full diagram

Matched Pairs Design

A special case of blocking where each block contains exactly two experimental units that are as similar as possible, or the same unit measured twice (before/after or crossover).

Two Matched Subjects
Pair similar subjects (e.g., twins). Randomly assign one to treatment, the other to control.
Same Subject Twice
Each subject serves as their own control. Randomly assign the order of treatments.

A listening study with fixed response cards and moving treatment badges

Participants wear headphones at matching task stations while a researcher distributes cards.Measurements taken from the same person remain linked. Compare within-person differences instead of treating the two columns as independent samples.

Keep the pairs linked

Six linked before-and-after pairs become six differences; within-ID sign swaps generate 64 assignments with 12 in the marked absolute tail.Suppose the same six IDs have differences 2, −1, 3, 0, 2 and 1, averaging 7/6. Under a sharp zero-effect null with randomized or exchangeable within-ID labels, 2⁶ = 64 sign assignments are allowed; 12 have absolute mean at least 7/6. Repeated measurement alone does not establish that exchangeability.

How to Design an Experiment

1
Identify the question. What cause-and-effect relationship do you want to investigate?
2
Define the factor(s), levels, and response variable. Be specific about what you will manipulate and measure.
3
Choose the experimental units. Describe the subjects and how many you will use (replication).
4
Randomly assign units to treatment groups. Use a random number generator, not haphazard methods.
5
Control other variables. Keep all conditions identical except the treatment. Use blinding/placebos when appropriate.
6
Measure and compare the response variable across groups to assess the treatment effect.

Example 1: Fertilizer Study

A botanist wants to know if a new fertilizer increases tomato yield. She has 40 tomato plants of the same variety in a greenhouse.

1
Factor: Fertilizer type (new vs standard). Response: Tomato yield (kg) after 10 weeks.
2
Randomly assign 20 plants to the new fertilizer and 20 to the standard fertilizer using a random number table.
3
Control: Same soil, water, sunlight, pot size, and greenhouse temperature for all 40 plants.
4
Compare: After 10 weeks, measure yield for both groups and determine if any difference is statistically significant.

Example 2: Blocking in a Drug Trial

A pharmaceutical company tests a new headache medication. They suspect that severity of headaches (mild, moderate, severe) will affect the response.

1
Block the 90 participants into 3 groups of 30 by headache severity (mild, moderate, severe).
2
Within each block, randomly assign 15 to the new drug and 15 to a placebo.
3
Double-blind: Neither the patients nor the doctors measuring pain relief know who received the drug.
4
Compare pain relief scores within each block and overall.

Example 3: Matched Pairs Taste Test

A coffee company wants to know if customers prefer a new blend over the original. They recruit 50 volunteers.

1
Each subject tastes both blends (matched pairs: same subject is their own control).
2
Randomly assign the order: for each subject, flip a coin to decide which blend is tasted first.
3
Single-blind: Cups are labeled A and B (subjects do not know which is which).
4
Record each subject's preference rating (1-10) for each blend and compare the paired differences.

Common Mistakes

✕ Confusing Random Selection and Random Assignment
Random selection is about sampling from a population (for generalizability). Random assignment is about placing subjects into groups (for causation).
✕ No Control Group
Without a comparison group, you cannot tell whether the treatment caused the change or if it would have happened anyway.
✕ Too Few Replications
Using too few subjects per group makes the experiment unable to detect a real treatment effect (low statistical power).
✕ Calling an Observational Study an Experiment
If subjects are not randomly assigned to treatments by the researcher, it is not an experiment, regardless of how carefully data is collected.
✕ Ignoring Lurking Variables
Failing to control extraneous variables introduces confounding, making it impossible to isolate the treatment effect.

Experimental Designs Summary

Design How It Works Best When
Completely Randomized All units randomly assigned to treatments Units are fairly homogeneous
Randomized Block Group into blocks first, then randomize within blocks A known variable affects the response
Matched Pairs Blocks of size 2: matched subjects or same subject twice Subjects can be closely paired or serve as own control

Question 1 of 5

A researcher randomly assigns 60 volunteers to either take Vitamin C or a placebo daily for 3 months, then measures the number of colds each person gets. Which of the following is the response variable?

Question 2 of 5

A study finds that people who eat breakfast every day have lower rates of heart disease than people who skip breakfast. Can we conclude that eating breakfast causes lower rates of heart disease?

Question 3 of 5

In an experiment testing a new allergy medication, neither the patients nor the doctors measuring symptoms know who receives the drug and who receives the placebo. This describes a:

Question 4 of 5

A coach wants to test two training programs. She has athletes of varying skill levels (beginner, intermediate, advanced) and is worried this will affect results. She should use a:

Question 5 of 5

Which of the following is the primary purpose of random assignment in an experiment?