Selecting an Experimental Design - AP Statistics

SECTION 1

Historical Context & Motivation

For centuries, scientists drew conclusions from uncontrolled observations—comparing outcomes without systematically ruling out alternative explanations. Agricultural yields, medical treatments, and industrial processes were evaluated anecdotally, leaving results vulnerable to hidden biases that today we call confounding variables. The question of how to design an experiment so that observed differences can be attributed to a treatment, rather than to lurking factors, drove the development of modern experimental design. This intellectual thread runs from early agricultural field trials to the randomized clinical trials that underpin modern medicine and policy evaluation.

1747 • Lind's Scurvy Trial

James Lind divided twelve sailors into six groups of two and gave each group a different dietary supplement. Though small and not randomized, this is often cited as the first controlled clinical trial, demonstrating the power of deliberate comparison.

1926 • Fisher Formalizes Experimental Design

Ronald A. Fisher published the first edition of Statistical Methods for Research Workers and later The Design of Experiments (1935), introducing the principles of randomization, replication, and blocking at Rothamsted Experimental Station.

1948 • First Modern Randomized Controlled Trial

The British Medical Research Council conducted a randomized controlled trial of streptomycin for tuberculosis, widely recognized as the first trial to use random allocation to treatment groups, setting the gold standard for medical research.

1965 • Cox's Framework for Experimental Planning

D. R. Cox published Planning of Experiments, systematizing how researchers choose among completely randomized, randomized block, and matched-pairs designs based on the structure of known sources of variability.

2000s • A/B Testing at Scale

Technology companies popularized large-scale randomized experiments—A/B tests—on millions of users, applying Fisher's principles to web interfaces, algorithms, and digital products, demonstrating that experimental design principles scale from farm plots to software platforms.

The central question that unites these milestones is deceptively simple: How should experimental units be assigned to treatments so that the resulting data support valid causal conclusions? Different designs answer this question in different ways, each balancing simplicity, control of variability, and practical constraints. Understanding when to use a completely randomized design, a randomized block design, or a matched-pairs design is a core competency in AP Statistics and a skill that extends well beyond the exam.

SECTION 2

Core Principles of Experimental Design

Before choosing a specific layout for an experiment, you must internalize the three foundational principles that Fisher established. Every well-designed experiment incorporates comparison through control, random assignment, and replication. A fourth principle—blocking—enters when researchers have advance knowledge about a source of variability that can be accounted for structurally. These ideas are not mutually exclusive; rather, they layer upon one another to strengthen the evidence an experiment can provide.

Control / Comparison

Every experiment needs a baseline. A control group (which may receive a placebo or the current standard) allows you to isolate the effect of the explanatory variable by comparing treatment outcomes to baseline outcomes under otherwise identical conditions.

Random Assignment

Subjects are allocated to treatment groups using a chance mechanism—coin flips, random number generators, or similar devices. Randomization tends to equalize both known and unknown confounding variables across groups, justifying causal inference.

Replication

Each treatment must be applied to a sufficient number of experimental units so that the natural variability among units does not obscure the treatment effect. More replication yields greater statistical power to detect genuine differences.

Blocking

When a known source of variation exists (e.g., age, gender, location), grouping units into blocks of similar units and randomizing within each block reduces experimental error and increases sensitivity to the treatment effect.

✦

KEY TAKEAWAY

Think of designing an experiment like planning a fair race. Control ensures every runner starts at the same line. Randomization assigns lane positions by lottery so no runner gets an unfair wind advantage. Replication runs the race with enough competitors to see true speed differences. Blocking separates heats by skill level so an Olympic sprinter doesn't obscure the comparison among college athletes. Each principle removes a different threat to a fair conclusion.

SECTION 3

Visual Overview of Three Experimental Designs

The diagram below illustrates the structural differences among the three experimental designs you need to know for AP Statistics: the completely randomized design (CRD), the randomized block design (RBD), and the matched-pairs design. Pay attention to where the random assignment step occurs in each layout and how subjects flow from the pool of experimental units to the comparison of responses.

Panel A shows a completely randomized design where all subjects are randomly assigned to treatments without pre-grouping. Panel B illustrates a randomized block design where subjects are first sorted into blocks based on a known source of variation, then randomly assigned within each block. Panel C depicts a matched-pairs design, a special case of blocking where each block contains exactly two units (or one unit measured twice).

Notice that the key structural difference lies in what happens before randomization. In a CRD, nothing—subjects go directly to the randomization step. In an RBD, subjects are first sorted into homogeneous blocks. In matched pairs, subjects are explicitly paired so that within-pair differences reflect only the treatment effect plus random noise, not the blocking variable. The choice among these designs depends on whether you can identify a known source of variability and whether the practical constraints of the study permit blocking.

SECTION 4

How Each Design Works

Completely Randomized Design (CRD)

In a completely randomized design, every experimental unit has an equal probability of being assigned to any treatment group. This is the simplest design and is appropriate when the researcher has no prior information about sources of variation among units, or when units are fairly homogeneous. A random number generator or a table of random digits assigns each unit to a group, and the resulting treatment groups are compared directly.

The major advantage of the CRD is its simplicity: no blocking structure must be determined in advance, and the analysis (typically a two-sample t-test or one-way ANOVA) is straightforward. The disadvantage is that if a lurking variable creates substantial variation among units, the randomization may not perfectly balance groups—especially with small sample sizes—reducing the experiment's power to detect real effects.

Randomized Block Design (RBD)

A randomized block design first groups experimental units into blocks that are homogeneous with respect to some known source of variation, then randomly assigns treatments within each block. For example, if you are testing two fertilizers on crop yield and know that field position (sunny vs. shady) affects growth, you would create blocks of plots that share similar sunlight and randomize treatments within each block. By accounting for the block-to-block variation, the RBD reduces the experimental error and increases the power to detect the treatment effect.

⚠

AP EXAM TIP

On free-response questions, students frequently lose credit for failing to explain why they chose to block. Always state the blocking variable and explain that blocking reduces variability within groups, making it easier to detect a treatment difference. Simply saying "I will use a randomized block design" without justification is insufficient.

Matched-Pairs Design

A matched-pairs design is a special case of the randomized block design in which each block contains exactly two experimental units (or one unit measured under two conditions). The two most common forms are: (1) pairing two subjects who are similar on a key characteristic and randomly assigning one to each treatment, and (2) having each subject serve as their own control by receiving both treatments in a randomly determined order (a crossover design). The analysis focuses on the within-pair differences rather than on individual responses, which typically reduces variability dramatically.

Matched pairs are especially powerful when individual variability is large relative to the treatment effect. By looking at how each pair's responses differ, you effectively remove the pair-to-pair variation from the analysis. The cost is logistical: you must be able to identify appropriate pairs or ensure that carry-over effects do not contaminate a crossover design.

A university campus provides a setting for comparing sampling and study-design routes. The way people or groups enter a study determines its design. Sampling within groups, selecting whole groups and assigning treatments answer different questions.

Twelve schools are divided into urban and suburban blocks, each split into three schools per curriculum. Suppose 12 schools are blocked as urban or suburban, with three schools assigned to each curriculum in each block. Schools are the assignment units; measuring many students within a school does not create additional independent treatment assignments.

SECTION 5

Decision Framework: Choosing the Right Design

Selecting an experimental design is a decision that depends on the researcher's knowledge of the experimental units, the number of treatments, and practical constraints. The flowchart below provides a systematic decision framework. Note that the AP exam may present a scenario and ask you to identify the most appropriate design, or it may describe a design and ask you to justify it. Either way, you must articulate the reasoning behind the choice.

Start at the top: ask whether a known source of variation exists among your experimental units. If not, use a CRD. If yes, create blocks. Then ask whether each block can contain exactly two units (or each subject can serve as their own control)—if so, use matched pairs; otherwise, use a full randomized block design. In every case, random assignment to treatments occurs within the chosen structure.

Feature	CRD	RBD	Matched Pairs
Pre-grouping?	None	Blocks of ≥2 units based on a known variable	Pairs of 2 or 1 subject under 2 conditions
Randomization	All units randomly assigned to treatments	Within each block	Within each pair
Best when	Units are relatively homogeneous; no known confound	A known confound can be identified and used to create blocks	Individual-level variation is large; only 2 treatments
Analysis	Two-sample t-test or ANOVA	ANOVA with block as a factor	One-sample t-test on paired differences
AP Exam focus	Describe random assignment procedure	Identify blocking variable and justify	Explain how pairs are formed and why

Summary comparison of the three designs tested on the AP Statistics exam.

SECTION 6

Worked Example: Designing an Experiment

A nutritionist wants to determine whether a new protein supplement increases muscle mass more than a standard supplement over an eight-week period. She recruits 40 volunteers from a gym—20 are experienced weightlifters and 20 are beginners. She suspects that fitness level affects muscle gain independently of the supplement. Design an experiment to test the supplement's effectiveness.

Designing the Protein Supplement Experiment

1
Step 1 — Identify the Key Variables

The explanatory variable (factor) is the type of supplement: new protein supplement vs. standard supplement. The response variable is the change in muscle mass (in pounds) over eight weeks. The potential confounding variable is the fitness level of the volunteer (experienced vs. beginner), which is known before the experiment begins.

Factor: supplement type; Response: Δ muscle mass; Confound: fitness level
2
Step 2 — Select the Design

Because the researcher has identified a known source of variability (fitness level) that is expected to influence the response, a randomized block design is most appropriate. There are two natural blocks: experienced weightlifters and beginners. With two treatment conditions and blocks of size 20, each block can accommodate random assignment to both treatments.

Design: Randomized Block Design with two blocks (experienced, beginner)
3
Step 3 — Describe the Blocking and Random Assignment

Separate the 40 volunteers into two blocks: Block 1 = 20 experienced weightlifters, Block 2 = 20 beginners. Within Block 1, use a random number generator to assign 10 volunteers to the new supplement and 10 to the standard supplement. Repeat this process independently within Block 2. This ensures that each treatment group contains equal representation of both fitness levels, preventing fitness from confounding the treatment comparison.

Each block: 10 → new supplement, 10 → standard supplement (randomly assigned)
4
Step 4 — Describe Measurement and Comparison

Measure each participant's muscle mass at the start and end of the eight-week period. Calculate the change in muscle mass for each participant. Within each block, compare the mean change in the new-supplement group to the mean change in the standard-supplement group. If the new supplement is more effective, we expect a significantly larger mean change in the new-supplement group within each block.

Compare mean Δ muscle mass between treatment groups within each block
5
Step 5 — Justify the Design Choice

Why not a CRD? A completely randomized design would risk placing most experienced lifters in one treatment group by chance, especially with only 40 subjects. Blocking on fitness level removes fitness-level variability from the error term, increasing the experiment's ability to detect a true supplement effect. A matched-pairs design is not ideal here because we have two treatment groups and blocks of size 20 (not pairs of 2), and there is no natural one-to-one pairing among participants.

RBD justified: blocking removes known variability, increasing statistical power

SECTION 7

Strengths and Limitations of Each Design

No single design is universally superior; each involves trade-offs between simplicity, statistical efficiency, and practical feasibility. The table below summarizes the key strengths and limitations you should be prepared to discuss on the AP exam. When a free-response question asks you to justify a design, referencing both its strengths and the limitations of alternatives demonstrates a sophisticated understanding.

Design	Strengths	Limitations
Completely Randomized (CRD)	Simplest to implement; no advance knowledge of variability needed; flexible for any number of treatments; straightforward analysis	Uncontrolled variability inflates experimental error; less powerful when units are heterogeneous; with small n, groups may be unbalanced on key characteristics by chance
Randomized Block (RBD)	Accounts for known source of variation; reduces error variability; more powerful than CRD when blocks are truly homogeneous; can handle multiple treatments	Requires advance knowledge of a blocking variable; ineffective if blocks are poorly chosen (wrong variable or insufficient homogeneity); more complex analysis; loses degrees of freedom to blocks
Matched Pairs	Maximally controls individual-level variation; very powerful when subject-to-subject differences dominate; simple paired-difference analysis	Limited to exactly two treatments; finding matched pairs can be difficult or subjective; crossover designs risk carry-over effects; loss of one subject eliminates the entire pair

Comparative strengths and limitations of the three AP-relevant experimental designs.

✦

KEY TAKEAWAY

Think of design selection like choosing insulation for a house. A CRD is like a house with no insulation—heat escapes everywhere (variability is uncontrolled). An RBD adds insulation in the walls where you know heat escapes (blocking reduces known variability). Matched pairs are like adding insulation perfectly fitted to every crack (each pair eliminates individual-level noise). Better insulation costs more effort to install, but it preserves the 'heat' (treatment effect signal) far more effectively.

SECTION 8

Connections to Advanced Concepts

The three designs covered in AP Statistics—CRD, RBD, and matched pairs—are entry points into a much richer landscape of experimental design methodology studied in college-level courses and professional research. Understanding how these foundational designs connect to more advanced concepts deepens your appreciation of why design matters and helps you answer AP questions that probe the reasoning behind experimental structure.

AP-Level Concept	Advanced Extension	Connection
Completely Randomized Design	Factorial Experiments (2 × 2, 2 × 3, etc.)	CRDs can test multiple factors simultaneously, examining both main effects and interaction effects
Randomized Block Design	Latin Square and Split-Plot Designs	Blocks can be extended to control for two sources of variation simultaneously or to handle constraints where some factors must be applied at different levels of hierarchy
Matched-Pairs Design	Crossover Designs and Repeated Measures	When each subject receives both treatments in sequence, time-related effects and wash-out periods become design considerations
Random Assignment (all designs)	Randomization Tests and Permutation Inference	The act of randomization itself provides the mathematical justification for inference without distributional assumptions

How AP-level experimental designs extend into advanced statistical methodology.

For the AP exam, you do not need to implement factorial or Latin square designs, but understanding that blocking and randomization are scalable principles will help you reason about unfamiliar scenarios. The AP free-response section sometimes presents multi-factor experiments or unusual blocking structures, and students who understand the underlying logic—rather than just memorizing three design names—are best positioned to earn full credit.

📝

EXPERIMENT VS. OBSERVATIONAL STUDY

Remember: the defining feature of an experiment is the imposition of treatments by the researcher. If treatments are merely observed (not assigned), you have an observational study, and causal conclusions cannot be drawn regardless of the statistical method used. This distinction is fundamental and appears regularly on the AP exam.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher wants to test whether a new type of running shoe reduces 5K race times compared to a standard shoe. She has 30 runners available and knows that each runner's baseline fitness varies widely. She decides to have each runner complete one 5K in the new shoe and one 5K in the standard shoe, with the order randomized. Which experimental design is being used?

A. Completely randomized design
B. Randomized block design with 30 blocks
C. Matched-pairs design where each runner is their own control
D. Observational study with a natural experiment
E. Stratified random sampling design

Show answer (C)

C is correct. This is a matched-pairs design because each runner serves as their own control, receiving both treatments with the order randomized. Each runner constitutes a 'block' of size 2. A is incorrect because a CRD would randomly assign each runner to exactly one shoe type, not both. B is incorrect because while a matched-pairs design is technically a special case of an RBD, the standard AP terminology for blocks of size 2 (especially when subjects serve as their own controls) is 'matched-pairs design.' D is incorrect because the researcher is actively imposing treatments, making this an experiment. E is incorrect because stratified random sampling is a sampling method, not an experimental design—it pertains to how subjects are selected from a population, not how treatments are assigned.

Question 2: BASIC CALCULATION

A botanist wants to compare the effects of three different fertilizers on tomato plant growth. She has 60 tomato plants arranged in a greenhouse with 4 rows, where each row receives different amounts of sunlight. She creates blocks by row and randomly assigns fertilizers within each row. How many plants receive each fertilizer treatment within each block?

A. 5 plants per fertilizer per block
B. 15 plants per fertilizer per block
C. 20 plants per fertilizer per block
D. 4 plants per fertilizer per block
E. 10 plants per fertilizer per block

Show answer (A)

A is correct. With 60 plants in 4 blocks (rows), each block contains 60 ÷ 4 = 15 plants. With 3 fertilizer treatments, each treatment is applied to 15 ÷ 3 = 5 plants per block. B is incorrect because 15 is the total number of plants per block, not the number per treatment per block. C is incorrect because 20 = 60 ÷ 3 represents the total number of plants per treatment across all blocks, not within each block. D is incorrect because 4 is the number of blocks, not a meaningful count per treatment per block. E is incorrect because 10 does not arise from any correct division in this setup.

Question 3: INTERMEDIATE

A psychologist wants to test whether background music improves performance on a memory task. She has 50 participants and knows that age (young adults vs. older adults) is likely to affect memory performance. She is choosing between a completely randomized design and a randomized block design blocking on age group. Which of the following best explains why a randomized block design would be preferred?

A. Blocking on age ensures that older adults are not included in the experiment.
B. Blocking on age eliminates age as a variable entirely, making the groups identical.
C. Blocking on age reduces variability within treatment groups, increasing the ability to detect the effect of music on memory.
D. Blocking on age allows the researcher to generalize the results to the entire population of all ages.
E. Blocking on age means the researcher does not need to use random assignment within each block.

Show answer (C)

C is correct. By forming blocks of similar-aged participants, the variability within each block due to age is reduced. This means that within each block, differences between treatment groups are more likely to reflect the true effect of background music rather than natural age-related differences in memory ability, thereby increasing statistical power. A is incorrect because blocking includes all age groups; it organizes them, not excludes them. B is incorrect because blocking does not eliminate age—it controls for it by ensuring equal representation within treatment comparisons. D is incorrect because generalizability depends on how subjects were selected from the population, not on the experimental design structure. E is incorrect and dangerously wrong—random assignment within blocks is essential to establish causal inference. Blocking without randomization would reintroduce confounding.

Question 4: APPLIED

4 pts

A pharmaceutical company wants to test whether a new migraine medication reduces the frequency of migraines more effectively than the current standard medication. The company recruits 120 migraine sufferers. Researchers know from previous studies that migraine frequency is heavily influenced by whether a patient experiences chronic migraines (15+ per month) or episodic migraines (fewer than 15 per month). Among the 120 patients, 40 have chronic migraines and 80 have episodic migraines. (a) Identify an appropriate experimental design for this study and explain why it is preferred over a completely randomized design. (b) Describe in detail how the researchers should implement the design you identified, including how random assignment should be carried out. (c) Explain why random assignment is important even within a blocked design. (d) Suppose the results show that the new medication significantly reduces migraine frequency compared to the standard. Can the researchers conclude that the new medication caused the reduction? Explain.

Show answer

Scoring rubric (4 points): Part (a) — 1 point: A randomized block design is appropriate. The blocking variable is migraine type (chronic vs. episodic). This is preferred over a CRD because migraine type is a known source of variability in the response variable (migraine frequency). Blocking on migraine type groups similar patients together, reducing within-group variability and increasing the experiment's statistical power to detect a true medication effect. Without blocking, a CRD with 120 patients could, by chance, assign disproportionate numbers of chronic sufferers to one treatment group, biasing the comparison. Part (b) — 1 point: Create two blocks: Block 1 consists of the 40 chronic migraine patients, and Block 2 consists of the 80 episodic migraine patients. Within Block 1, use a random number generator to assign 20 patients to the new medication and 20 to the standard medication. Within Block 2, randomly assign 40 patients to the new medication and 40 to the standard medication. Administer the assigned medication to each patient for a specified period (e.g., 3 months) and record the frequency of migraines. Part (c) — 1 point: Random assignment within blocks is essential because, even after blocking on migraine type, other lurking variables (such as age, stress level, sleep quality, or other health conditions) may differ among patients within the same block. Random assignment tends to equalize these unknown confounding variables across treatment groups, so any observed difference in migraine frequency can be attributed to the medication rather than to these uncontrolled factors. Without randomization, the experiment cannot support a causal conclusion. Part (d) — 1 point: Yes, the researchers can conclude that the new medication caused the reduction in migraine frequency. Because this is a well-designed experiment with random assignment of treatments (within blocks), the differences between treatment groups can be attributed to the medication rather than to confounding variables. The combination of blocking (to control for migraine type) and random assignment (to balance other potential confounders) justifies a causal conclusion. However, the scope of inference is limited to the 120 volunteers in the study; generalizing to all migraine sufferers requires that the sample be representative of the broader population.

Question 5: CRITICAL THINKING

4 pts

A school district is evaluating two new math curricula (Curriculum A and Curriculum B) to determine which leads to higher student achievement on a standardized test. The district has 12 schools. Six schools are located in urban areas and six in suburban areas. Within each school, there are approximately 100 students in the relevant grade level. A district administrator proposes the following design: 'We will randomly assign 6 schools to Curriculum A and 6 schools to Curriculum B, then compare the average standardized test scores at the end of the year.' (a) What type of experimental design has the administrator proposed? Is this design appropriate given the structure of the schools? Explain. (b) Propose an improved experimental design. Clearly describe the blocking variable, the random assignment procedure, and the units of analysis. (c) A teacher suggests that instead of assigning by school, they should randomly assign individual students within each school to a curriculum. Discuss one advantage and one practical limitation of this approach. (d) Regardless of which design is used, explain why this experiment might have limitations in establishing causation that a laboratory experiment would not have.

Show answer

Scoring rubric (4 points): Part (a) — 1 point: The administrator has proposed a completely randomized design with schools as the experimental units. This design is not ideal because it ignores a known source of variability: school location (urban vs. suburban). Urban and suburban schools may differ systematically in student demographics, resources, and baseline achievement levels. With only 12 schools, random assignment alone may not adequately balance these differences, potentially confounding the curriculum comparison. Part (b) — 1 point: An improved design is a randomized block design blocking on school location. Create two blocks: Block 1 = 6 urban schools, Block 2 = 6 suburban schools. Within Block 1, randomly assign 3 urban schools to Curriculum A and 3 to Curriculum B. Within Block 2, randomly assign 3 suburban schools to Curriculum A and 3 to Curriculum B. The experimental unit is the school (since the curriculum is implemented school-wide), and the response variable is the school's mean standardized test score. This ensures that urban-suburban differences do not confound the curriculum comparison. Part (c) — 1 point: Advantage: Randomly assigning individual students within each school dramatically increases the sample size for each treatment (approximately 600 per curriculum instead of 6 schools), providing much greater statistical power and making it easier to detect even small differences in achievement. Practical limitation: It would be extremely difficult for a teacher to simultaneously implement two different curricula in the same school. Students in the same building might share materials, discuss lessons, or switch between classes, creating contamination between treatment groups. Teachers would also need to be trained in both curricula, introducing additional logistical and fidelity concerns. Part (d) — 1 point: Unlike a laboratory experiment where conditions can be tightly controlled, a school-based field experiment involves many uncontrollable variables: teacher quality and enthusiasm, student motivation, parental involvement, classroom disruptions, and fidelity of curriculum implementation. Even with random assignment, these factors introduce noise that may differ systematically between treatment groups (e.g., if one school's teachers are less enthusiastic about their assigned curriculum). Additionally, it is impossible to blind participants—teachers and students know which curriculum they are using—which may introduce placebo-like effects or behavioral changes unrelated to curriculum quality. These threats to internal validity are inherent to field experiments and are why experimental design principles (blocking, randomization, replication) are so important: they mitigate, though they cannot fully eliminate, these challenges.

SUMMARY

Summary & Review

Summary

Selecting an experimental design is the process of choosing the structural framework that determines how experimental units are assigned to treatments. The three designs tested on the AP Statistics exam are the completely randomized design (CRD), which randomly assigns all units to treatments without pre-grouping; the randomized block design (RBD), which first sorts units into homogeneous blocks based on a known source of variability and then randomizes within each block; and the matched-pairs design, a special case of blocking where each block contains exactly two units or one unit is measured under both conditions.

All three designs rest on the foundational principles of control (a comparison group ensures a baseline), random assignment (a chance mechanism balances known and unknown confounders, enabling causal conclusions), and replication (enough units per treatment to detect genuine effects). When choosing a design, ask: Is there a known source of variability? If yes, block on it. Can blocks have exactly two units? If yes, use matched pairs. If no known variability exists or blocking is impractical, the CRD remains a valid and powerful choice. On the AP exam, always justify your design by explaining what variability the design controls and why random assignment is necessary.