Evaluating Study Conclusions — I can interpret survey/experiment descriptions and evaluate whether conclusions are justified. - Math 3

SECTION 1

Historical Context & Motivation

Every day, you encounter claims backed by data: a news headline says a new study "proves" that eating breakfast improves test scores, or an advertisement claims that 9 out of 10 dentists recommend a specific toothpaste. But how do you know whether those conclusions are actually supported by the data? The ability to evaluate study conclusions is one of the most practical skills in statistics, and its development has a rich history tied to the rise of modern science.

1747 • First Clinical Trial

James Lind conducted one of the first controlled experiments on sailors with scurvy. He divided subjects into groups receiving different treatments and observed which group recovered, establishing the power of comparison groups.

1936 • Literary Digest Poll Failure

The Literary Digest magazine surveyed 2.4 million people and incorrectly predicted that Alf Landon would defeat Franklin Roosevelt. The sampling bias — surveying mostly wealthy Americans — led to a famously wrong conclusion, proving that a large sample does not guarantee accuracy.

1948 • Randomized Controlled Trials

The British Medical Research Council conducted the first randomized controlled trial (RCT) to test streptomycin for tuberculosis. Random assignment became the gold standard for establishing cause and effect.

2005–present • Replication Crisis

Researchers found that many published studies could not be replicated, sparking a worldwide conversation about statistical validity and the importance of critically evaluating conclusions rather than accepting them at face value.

These historical moments all point to the same fundamental question: Does the design of a study actually support the conclusion being drawn? In this lesson, you will learn the framework for answering that question with confidence.

SECTION 2

Core Principles & Definitions

Before you can evaluate whether a conclusion is justified, you need to understand the building blocks of statistical studies. Every study has a design, and that design determines what kinds of conclusions are valid. Let's break down the key terms and ideas.

Observational Study vs. Experiment

An observational study watches and records data without interfering. An experiment deliberately applies a treatment to subjects. Only experiments can establish causation.

Random Sampling

Random sampling means every individual in the population has a known chance of being selected. This allows you to generalize findings to the larger population.

Random Assignment

Random assignment places subjects into treatment or control groups by chance. This controls for confounding variables and allows researchers to claim cause and effect.

Confounding Variable

A confounding variable is an outside factor that influences both the explanatory and response variables. If not controlled, it makes it impossible to determine true cause and effect.

Scope of Inference

The scope of inference describes who the findings apply to and whether a causal claim can be made. It depends entirely on how the study was designed — specifically, whether random sampling and random assignment were used.

✦

KEY TAKEAWAY

Think of study design like a courtroom. Random sampling is like selecting a jury that represents the whole community — it lets you generalize. Random assignment is like a fair trial that eliminates bias — it lets you assign blame (causation). Without the right jury, your verdict only applies to the people in the room. Without a fair trial, you can't be sure the defendant is guilty.

SECTION 3

Visual Explanation — The Scope of Inference Chart

The relationship between random sampling, random assignment, and the type of conclusion you can draw is best understood through a two-by-two chart. This visual is the single most important tool for evaluating study conclusions. Study it carefully — it will be your go-to reference.

The scope of inference chart shows four possible scenarios. The top-left cell (both random sampling and random assignment) is the strongest design. The bottom-right cell (neither) gives the weakest conclusions. Most real-world studies fall somewhere in between.

When you read a study description on a test or in the real world, your first move should be to identify which cell of this chart the study falls into. Ask yourself two questions: Was the sample randomly selected from a population? and Were subjects randomly assigned to treatment groups? The answers determine everything about what the study can and cannot conclude.

SECTION 4

How Study Design Determines Valid Conclusions

The Two Key Questions Framework

Evaluating a study conclusion is a systematic process. You don't need formulas — you need a logical framework. Here is the step-by-step reasoning process you should follow every time you encounter a study and its conclusion.

QUESTION 1 — GENERALIZATION

Random Sampling → Can generalize to the population

If the study used a random sample from a defined population, the results can be extended beyond the study participants to that entire population. If the sample was a convenience sample (e.g., volunteers, one classroom), the results apply only to those specific subjects.

QUESTION 2 — CAUSATION

Random Assignment → Can claim cause and effect

If subjects were randomly assigned to treatment and control groups, confounding variables are balanced across groups. This lets the researcher attribute differences in outcomes to the treatment itself. Without random assignment, the conclusion must be limited to an association (correlation), not causation.

Common Conclusion Errors

The most frequent mistake in evaluating studies is claiming causation from an observational study. For example, if a survey finds that students who eat breakfast score higher on tests, you cannot conclude that eating breakfast causes higher scores. A confounding variable — perhaps family income — might explain both behaviors. Another common error is generalizing from a non-random sample. If you survey only students at one school, you cannot claim the results represent all teenagers nationwide.

💡

Remember This Phrase

"Correlation does not imply causation." This is the single most important sentence in introductory statistics. An observational study can show that two things are related, but only a well-designed experiment with random assignment can show that one causes the other.

SECTION 5

Types of Studies & Their Valid Conclusions

Not all studies are created equal. Understanding the different types of study designs helps you quickly identify what conclusions are valid. The diagram below shows the decision tree you can use to classify any study you encounter.

This decision tree classifies any study. Start at the top: does the researcher impose a treatment? If yes, it is an experiment. If no, it is an observational study. Then check for random assignment and random sampling to determine the valid scope of inference.

Study Type	Key Feature	Valid Conclusion
Randomized Experiment	Treatment imposed + random assignment to groups	Cause-and-effect (and generalization if random sampling is also used)
Non-Randomized Experiment	Treatment imposed but groups are pre-existing (e.g., period 1 vs. period 3)	Weak causal claim — confounders may not be balanced
Sample Survey	No treatment; data collected from a sample of a population	Association only; can generalize if random sample
Prospective Study	Follows subjects forward in time, recording outcomes	Association only — no causation
Retrospective Study	Looks backward at existing records or past data	Association only — no causation

Summary of study types and the conclusions each can support

SECTION 6

Worked Example — Evaluating a Study Conclusion

Let's walk through a complete example of reading a study description and deciding whether its conclusion is justified.

ℹ

Study Description

A researcher at a large university wanted to know whether listening to classical music while studying improves exam performance. She posted flyers around campus asking for volunteers. Sixty students signed up. She randomly assigned 30 students to study with classical music and 30 to study in silence. After one week, both groups took the same exam. The music group scored an average of 82%, while the silent group scored 76%. The researcher concluded: "Classical music causes higher exam scores for college students."

Evaluating the Conclusion

1
Step 1 — Identify the study type

The researcher imposed a treatment (listening to classical music vs. silence) on the subjects. Because a treatment was deliberately applied, this is an experiment, not an observational study.

Study type: Experiment
2
Step 2 — Check for random assignment

The description says the researcher "randomly assigned" 30 students to each group. This means confounding variables (like prior GPA, sleep habits, or study skills) should be roughly balanced between the two groups. Random assignment is present, so a causal conclusion about the treatment is supported.

Random assignment: Yes → Causation is supported
3
Step 3 — Check for random sampling

The students were volunteers who responded to flyers — this is a convenience sample, not a random sample of all college students. The volunteers may differ from the general student population (perhaps they are more motivated or more interested in music). Therefore, the results cannot be generalized beyond the 60 participants.

Random sampling: No → Cannot generalize to all college students
4
Step 4 — Evaluate the conclusion

The researcher claimed: "Classical music causes higher exam scores for college students." The causal language ("causes") is supported because of random assignment. However, the phrase "for college students" implies generalization to the entire population of college students, which is not justified because the sample was not randomly selected.

Conclusion is PARTIALLY justified
5
Step 5 — Write a corrected conclusion

A properly worded conclusion would narrow the scope: "For the 60 volunteer students in the study, listening to classical music while studying caused higher exam scores compared to studying in silence." This preserves the causal claim (from random assignment) while limiting the population (because there was no random sampling).

Corrected: causation claim limited to the study participants

SECTION 7

Strengths & Limitations of Different Study Designs

Each study design has trade-offs. Randomized experiments provide the strongest evidence but are not always ethical or practical. For example, you cannot randomly assign people to smoke for 20 years to study lung cancer. In such cases, observational studies are the only option. Understanding these trade-offs helps you evaluate why certain designs were chosen and what that means for the conclusions.

Feature	Randomized Experiment	Observational Study
Can establish causation?	Yes — random assignment controls for confounders	No — confounders remain uncontrolled
Can generalize?	Only if random sampling was also used	Only if random sampling was used
Ethical flexibility	Limited — cannot impose harmful treatments	More flexible — simply observes existing behaviors
Cost and time	Often expensive and time-consuming	Often cheaper and faster
Main weakness	May lack ecological validity (lab setting ≠ real world)	Cannot rule out confounding variables

Comparison of randomized experiments and observational studies

✦

KEY TAKEAWAY

Think of study design like a lock and key. The conclusion is the lock, and the study design is the key. A causal conclusion requires the specific key of random assignment. A generalization conclusion requires the specific key of random sampling. If you try to open a lock with the wrong key, it simply doesn't work — and neither does a conclusion that overreaches its study design.

SECTION 8

Connections to Advanced Statistical Reasoning

The principles you've learned in this lesson form the foundation for more advanced statistical concepts that you'll encounter in AP Statistics, college courses, and real-world data literacy. Understanding scope of inference now prepares you for the deeper reasoning ahead.

This Lesson	Advanced Extension
Identifying confounding variables	Multiple regression analysis — statistically controlling for confounders
Random assignment → causation	A/B testing in tech companies; clinical trials in medicine
Random sampling → generalization	Margin of error and confidence intervals for population parameters
Evaluating whether a conclusion is justified	Hypothesis testing (p-values, significance levels) — quantifying the strength of evidence
Correlation ≠ causation	Simpson's Paradox — where aggregated data reverses a trend seen in subgroups

How this lesson connects to future topics

In AP Statistics, you will learn to quantify how strong the evidence is using tools like p-values and confidence intervals. But even the most sophisticated statistical test cannot fix a flawed study design. If a study lacks random assignment, no amount of math can turn an association into a causal claim. The concepts from this lesson remain your first line of defense throughout your statistical career.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A study found that people who drink coffee daily tend to have lower rates of depression. A newspaper headline reads: "Coffee Prevents Depression." Is this headline justified? Explain your reasoning by identifying the study type and the valid scope of inference.

Show answer

The headline is not justified. The study is observational — researchers did not randomly assign people to drink or avoid coffee. Since there is no random assignment, we cannot claim causation. There may be confounding variables — for example, people who drink coffee may also be more social or have different lifestyles. A valid conclusion would be: "Daily coffee consumption is associated with lower rates of depression." The causal word "prevents" goes beyond what an observational study can support.

Question 2: BASIC CALCULATION

A school principal randomly selects 200 students from the entire school population and surveys them about their sleep habits. She finds that students who sleep more than 8 hours per night have higher GPAs. She concludes: "Getting more sleep is associated with higher GPAs among students at this school." Is this conclusion justified? Identify whether random sampling and random assignment were used.

Show answer

This conclusion is justified. The principal used random sampling (200 students randomly selected from the school population), so the results can be generalized to all students at the school. However, no treatment was imposed and no random assignment was used — this is an observational study. The conclusion correctly uses the word "associated" rather than a causal word like "causes" or "improves." The generalization to "students at this school" is appropriate because the random sample was drawn from that population. This is a well-worded conclusion that matches the study's design.

Question 3: INTERMEDIATE

Researchers recruited 100 volunteer adults and randomly assigned them to either a meditation group or a control group for 8 weeks. At the end, the meditation group reported significantly lower stress levels. The researchers conclude: "Meditation reduces stress in adults." Identify two problems with this conclusion and rewrite it to be properly justified.

Show answer

There are two problems. First, although random assignment was used (supporting causation), the subjects were volunteers — not a random sample of all adults. Therefore, generalizing to "adults" as a whole is not justified. Second, self-reported stress is a subjective measure, and participants in the meditation group knew they received the treatment, which could create a placebo effect. A properly worded conclusion would be: "For the 100 volunteer participants in the study, the meditation program caused a reduction in self-reported stress levels compared to the control group." This preserves the causal language (justified by random assignment) while limiting the population scope (because there was no random sampling).

Question 4: APPLIED

A tech company wants to know if changing the color of a "Buy Now" button from blue to green increases sales. They randomly select 10,000 users from their customer database and randomly assign half to see the blue button and half to see the green button. After one week, the green-button group had 15% more purchases. The company concludes: "Changing the button to green increases sales among our customers." Evaluate this conclusion.

Show answer

This conclusion is well justified. The company used both random sampling (10,000 users randomly selected from the customer database) and random assignment (half see blue, half see green). Random assignment supports the causal claim that the green button increased sales — confounders like user demographics should be balanced between groups. Random sampling from the customer database supports generalizing the result to their customer base. The conclusion appropriately says "among our customers" rather than "for all internet users," correctly limiting generalization to the sampled population. This study design — called an A/B test — falls in the top-left cell of the scope of inference chart, the strongest possible scenario.

Question 5: CRITICAL THINKING

Consider two studies about exercise and heart health. Study A: Researchers randomly sample 5,000 adults from the U.S. population and survey them about their exercise habits and heart health, finding that more exercise is associated with better heart health. Study B: Researchers recruit 200 volunteers and randomly assign half to a 12-week exercise program. The exercise group shows improved heart health markers. Which study better supports the claim "Exercise improves heart health for American adults"? Explain why neither study alone fully supports this claim, and describe what an ideal study would look like.

Show answer

Neither study alone fully supports the claim. Study A has random sampling from the U.S. population, so its findings can be generalized to American adults, but it is observational — exercise was not assigned, so confounding variables (like income, diet, or genetics) could explain the association. It supports "Exercise is associated with better heart health for American adults" but not causation. Study B uses random assignment, so it can establish that the exercise program caused improved heart health. However, the volunteer sample means the results apply only to those 200 participants, not all American adults. An ideal study would combine both features: randomly sample a large group of American adults AND randomly assign them to an exercise or control group. In practice, this is difficult because you cannot ethically force people to exercise or not exercise for extended periods, which is why researchers often rely on combining evidence from multiple imperfect studies.

SUMMARY

Lesson Summary

Summary

Evaluating study conclusions requires a systematic approach. First, determine whether the study is an experiment (treatment imposed) or an observational study (no treatment). Then ask two critical questions: Was random sampling used? If yes, the results can be generalized to the population. Was random assignment used? If yes, a cause-and-effect conclusion is supported. Without random assignment, the conclusion must be limited to an association. Without random sampling, the conclusion cannot be extended beyond the study participants.

Always watch for confounding variables in observational studies and for convenience sampling that limits generalization. Remember: correlation does not imply causation. The scope of inference chart — with random sampling on one axis and random assignment on the other — is your essential tool for matching any study's design to the conclusions it can validly support.