Introduction to Planning a Study

The need for rigorous study design grew out of centuries of scientific inquiry in which poorly collected data led to misleading or outright incorrect conclusions. In the early history of medicine, for example, physicians routinely prescribed treatments based on anecdotal observations and personal conviction rather than systematic evidence. Agriculture faced similar challenges: farmers debated the merits of different fertilizers without any controlled method for determining which truly improved crop yields. The formalization of statistical study planning emerged when researchers recognized that how data are collected is just as important as how data are analyzed. This section traces the key milestones that shaped the modern approach to planning studies, from early census efforts to the randomized controlled trial.

The central question that study planning addresses is deceptively simple: What kind of evidence do we need, and how should we collect it, to answer a specific question reliably? Without a clear answer to this question before data collection begins, even the most sophisticated analysis techniques cannot rescue a flawed study. The remainder of this lesson explores the principles, structures, and decisions that underpin effective study planning in statistics.

Planning a study requires a clear understanding of several foundational concepts that determine the type of study you conduct, the conclusions you can draw, and the population to which those conclusions apply. At the broadest level, every statistical investigation begins with a research question—a precisely stated inquiry that the study is designed to answer. The nature of that question dictates whether you need an observational study or an experiment. Understanding the distinction between these two broad categories—and the further subtypes within each—is arguably the single most important idea in the AP Statistics data-collection unit.

The diagram below provides a comprehensive visual map of the decision-making process that underlies planning a statistical study. Starting from a research question, the flowchart distinguishes the two main study types and highlights the key design elements associated with each. Understanding this branching structure is essential for the AP Statistics exam, where you will frequently be asked to identify the type of study, evaluate its design, and determine the scope of valid conclusions.

Notice that the diagram distinguishes between two types of randomness that often confuse students. Random selection (choosing who is in the study from the population) supports generalizability—the ability to extend conclusions to the broader population. Random assignment (deciding who receives which treatment within the study) supports causal inference—the ability to attribute differences in the response to the treatment itself. A study can have one, both, or neither of these features, and the scope of inference changes accordingly.

While the "Collecting Data" unit of AP Statistics is less equation-heavy than inference or probability, there is a rigorous logical framework that governs how study design maps onto the conclusions you may legitimately draw. The relationship between design features and inferential scope can be organized into a 2 × 2 scope-of-inference matrix, which is one of the most tested conceptual structures on the AP exam.

The matrix above is not a formula to memorize; it is a logical consequence of what randomness accomplishes. Random selection ensures that the sample reflects the population, so findings can be generalized outward. Random assignment balances both known and unknown confounding variables across treatment groups, so any observed difference in the response variable can be attributed to the treatment rather than to lurking variables. When a study lacks one or both forms of randomness, the corresponding inferential claim—generalizability or causation—is no longer supported.

Confounding: The Key Threat to Causal Inference

A confounding variable is a variable that is associated with both the explanatory variable and the response variable, creating an alternative explanation for the observed relationship. In an observational study comparing coffee drinkers to non-coffee drinkers on anxiety levels, for instance, stress could be a confounder: high-stress individuals may drink more coffee and experience more anxiety, making it unclear whether coffee itself drives the association. Random assignment in an experiment disrupts confounding by distributing all variables—measured or unmeasured—roughly equally across groups. This is why only well-designed experiments can support causal claims.

Within the two broad categories of observational studies and experiments, several subtypes appear regularly on the AP Statistics exam. Understanding the characteristics of each subtype helps you identify study designs in context and anticipate the strengths and limitations that follow.

A sample survey collects information from respondents at a single point in time, typically through questionnaires or interviews. It is perhaps the most common form of observational study and is subject to biases such as nonresponse and wording effects. A prospective study follows a group of subjects forward in time, recording exposures and outcomes as they develop; this approach is stronger than a retrospective design because it can track the temporal sequence of events. A retrospective study looks backward, mining existing records or asking subjects to recall past behavior—useful when the outcome has already occurred but susceptible to recall bias.

On the experimental side, a completely randomized design (CRD) is the simplest structure: all subjects are randomly allocated to treatment groups without any preliminary grouping. A randomized block design first organizes subjects into blocks based on a variable that is expected to affect the response (such as age group or gender), and then randomly assigns treatments within each block. This reduces variability and increases the precision of comparisons. A matched pairs design is a special case of blocking in which each block contains exactly two subjects who are matched on relevant characteristics, or a single subject who receives both treatments in random order.

The top shelf receives more light than the lower shelf. Place both fertilizer treatments on each shelf, control watering, soil and pot size, and measure height increase over four weeks.

For 30 seedlings, form two 15-plant shelf blocks and randomize both fertilizers within each shelf. The shown allocation assigns 7 new and 8 standard on top, 8 new and 7 standard below. Compare treatment growth within shelves so the light difference stays in the analysis.Open full diagram

Consider the following scenario, which mirrors the kind of prompt you would encounter on the AP Statistics free-response section.

No single study type is universally superior. The choice between observational and experimental designs depends on the research question, ethical constraints, available resources, and the population of interest. Recognizing the relative strengths and limitations of each approach is critical both for the AP exam and for evaluating real-world research.

The principles introduced in this lesson lay the groundwork for every inference procedure you will encounter later in AP Statistics. When you perform a two-sample t-test or construct a confidence interval for a difference in proportions, the validity of those procedures depends on how the data were collected. A well-designed experiment with random assignment allows you to test the null hypothesis that the treatment has no effect, while a well-designed sample survey allows you to construct confidence intervals that apply to the population. Conversely, when study design is flawed, no amount of inferential machinery can fix the resulting bias.

Looking beyond AP Statistics, advanced courses in biostatistics and econometrics formalize these ideas further. Quasi-experimental designs—such as regression discontinuity and difference-in-differences—attempt to approximate the benefits of randomization when true experiments are infeasible. Meta-analyses synthesize findings across many studies, weighting each study partly by the rigor of its design. The fundamental question remains the same one you are learning to ask now: Given how these data were collected, what conclusions are justified?