01 / 20

Selecting an
Experimental Design

Learn how to choose the right experimental design: completely randomized, randomized block, or matched pairs to maximize validity and minimize confounding.

AP Statistics

What Is an Experiment?

An experiment deliberately imposes a treatment on individuals to observe their responses. Unlike observational studies, experiments allow us to establish cause-and-effect relationships because the researcher controls who receives which treatment.

Experimental Units
The individuals or objects to which treatments are applied
Treatments
Specific conditions imposed on the experimental units
Response Variable
The outcome measured after the treatment is applied

Three Principles of Experimental Design

1
Control: Hold extraneous variables constant or compare treatments to a control group to reduce confounding.
2
Randomization: Use a chance process to assign treatments so groups are roughly equivalent before the experiment begins.
3
Replication: Use enough experimental units in each group so that real treatment effects can be distinguished from random variation.

Why Design Matters

Choosing the right experimental design controls variability and strengthens our ability to detect a real treatment effect. If our design is weak, the noise from lurking variables drowns out the signal from the treatment.

WEAK DESIGN Noise (confounding) Signal STRONG DESIGN Noise Signal (treatment effect)

Completely Randomized Design

In a completely randomized design (CRD), all experimental units are assigned to treatments entirely by chance. Every unit has an equal probability of receiving any treatment.

All Subjects
\( n \)
units Random Assignment Treatment 1 Measure response Treatment 2 Measure response Control Measure response Compare Results

Randomized Block Design

A randomized block design (RBD) groups experimental units into blocks based on a known source of variability, then randomly assigns treatments within each block. This reduces variability between groups.

Block 1: Males Subjects Random Treatment A Treatment B Compare Block 2: Females Subjects Random Treatment A Treatment B Compare

A university population with four visibly different sampling routes

A university campus provides a setting for comparing sampling and study-design routes.The way people or groups enter a study determines its design. Sampling within groups, selecting whole groups and assigning treatments answer different questions.

Randomize schools within blocks

Twelve schools are divided into urban and suburban blocks, each split into three schools per curriculum.Suppose 12 schools are blocked as urban or suburban, with three schools assigned to each curriculum in each block. Schools are the assignment units; measuring many students within a school does not create additional independent treatment assignments.

Matched Pairs Design

A matched pairs design is a special case of a randomized block design where each block has exactly two experimental units (or one unit measured twice). This is the most powerful way to compare exactly two treatments.

Type 1: Two Subjects
Pair similar subjects together. Randomly assign one to Treatment A and the other to Treatment B within each pair.
Type 2: One Subject
Each subject receives both treatments in random order. The subject serves as their own control (crossover design).

Comparing the Three Designs

Feature CRD RBD Matched Pairs
Blocking None Yes: groups by a variable Yes: pairs of 2
Number of Treatments Any Any Exactly 2
Best When Units are homogeneous Known lurking variable exists Comparing 2 treatments closely
Controls Variability Least Moderate Most
Complexity Simplest Moderate Moderate

How to Choose the Right Design

1
Count the treatments. If there are exactly 2 and you can pair subjects, consider a matched pairs design.
2
Identify lurking variables. Is there a known variable (age, gender, prior experience) that could affect the response? If yes, use a randomized block design.
3
Assess homogeneity. If experimental units are similar with no obvious lurking variable, a completely randomized design is sufficient.
4
Ensure replication. Regardless of design, use enough units in each group (or block) for the results to be statistically meaningful.

Decision Flowchart

How many treatments? Exactly 2 3+ Can you pair units? Yes Matched Pairs No Known lurking variable? Known lurking variable? Yes RBD No CRD Yes RBD No CRD

Example 1: Fertilizer Test

A farmer wants to test whether a new fertilizer increases crop yield. She has 30 identical plots in the same field and 3 fertilizers to compare (Fertilizer A, B, and a control). What design should she use?

1
Count treatments: 3 treatments (A, B, Control). Not exactly 2, so matched pairs is unlikely.
2
Lurking variables? The plots are identical and in the same field: no known lurking variable needs blocking.
3
Homogeneous units? Yes: all 30 plots are similar in soil, sunlight, and water.

Example 2: Study Drug Trial

A researcher tests two headache medications on 40 volunteers. She suspects that age strongly affects recovery time: younger patients recover faster regardless of medication. What design should she use?

1
Count treatments: 2 medications. Could potentially use matched pairs, but pairing on age alone is not precise enough for a block of 2.
2
Lurking variable? Yes: age. She should block on age groups (e.g., 18-30, 31-50, 51+).
3
Conclusion: Create age-based blocks, then randomly assign subjects within each block to Medication A or B.

Example 3: Taste Test

A coffee company wants to compare two new blends. Each of 25 tasters will taste both blends (in random order) and rate each. What design is this?

Each person = one block
Each taster receives both treatments. Individual taste preferences are controlled because the same person rates both blends.
Random order matters
Randomly assign which blend is tasted first to prevent order effects (e.g., palate fatigue).

Blinding and Placebos

In addition to choosing a design, researchers should consider blinding to reduce bias from expectations and the placebo effect.

Single-Blind
Either the subjects or the evaluators (but not both) do not know which treatment was assigned.
Double-Blind
Neither the subjects nor the evaluators know the treatment assignments. This is the gold standard.
Placebo
A dummy treatment (e.g., sugar pill) given to the control group so subjects cannot tell if they received the real treatment.

Common Mistakes

✕ Confusing Blocking with Stratifying
Blocking is for experiments; stratifying is for sampling. They both group by a variable, but blocking controls assignment within an experiment.
✕ No Random Assignment
Letting subjects choose their own group is not an experiment. Random assignment is essential for causal conclusions.
✕ Matched Pairs for 3+ Treatments
Matched pairs is only for exactly 2 treatments. For 3 or more, use CRD or RBD with larger blocks.

Question 1 of 5

A researcher wants to test three different diets on weight loss. She has 60 volunteers with no known differences that might affect results. Which design is most appropriate?

Question 2 of 5

A school wants to compare two teaching methods. Students vary greatly in prior math ability. The school creates groups of students with similar math scores, then randomly assigns methods within each group. This is an example of:

Question 3 of 5

In a matched pairs experiment, 20 runners each run a mile with Shoe A and Shoe B on different days, in random order. The response variable is mile time. Why is matched pairs appropriate here?

Question 4 of 5

A pharmaceutical company tests a new drug against a placebo. Neither the patients nor the doctors measuring outcomes know who received the drug. This is an example of:

Question 5 of 5

A researcher notices that sunlight exposure varies across her greenhouse. She wants to test 4 fertilizers on plant growth. She divides the greenhouse into sections with similar light levels and randomly assigns fertilizers within each section. What principle does this illustrate?