Inference and
Experiments
Learn how to draw causal conclusions from randomized experiments using hypothesis tests for the difference of two means and two proportions.
Why Experiments Matter
In statistics, the design of a study determines the scope of conclusions we can draw. Only a well-designed randomized experiment allows us to establish a cause-and-effect relationship between variables.
Anatomy of a Randomized Experiment
Key Terms in Experimental Inference
A listening study with fixed response cards and moving treatment badges
Random assignment determines which rearrangements belong in an experimental comparison. Independent groups and measurements linked to the same people require different treatment of the data.Keep responses fixed while labels change
Suppose eight responses are 6, 8, 9, 10, 11, 12, 14 and 18. Under a sharp no-effect null, all 70 assignments of four treatment labels preserve those responses. The observed difference is 1; 56 assignments have an absolute difference at least 1, giving 56/70 = 0.8.The Two-Sample \(t\)-Test for Means
When comparing two group means from an experiment, the test statistic is:
The Two-Proportion \(z\)-Test
When comparing two group proportions, we pool the proportions under \(H_0\) and use:
Experiments vs. Observational Studies
| Experiment | Observational Study | |
|---|---|---|
| Treatment | Imposed by researcher | Not imposed |
| Random Assignment | Yes | No |
| Confounding | Controlled by randomization | May be present |
| Conclusion | Causation | Association only |
| Generalization | Depends on random selection | Depends on random selection |
Checking Conditions for Inference
Step-by-Step Inference Procedure
Example 1: Comparing Means
A researcher randomly assigns 60 students to two groups: 30 use a new study app, 30 use traditional methods. After 4 weeks, test scores are recorded.
Example 1: Solution
Example 2: Comparing Proportions
A pharmaceutical company randomly assigns 200 patients to receive either a new drug (100 patients) or a placebo (100 patients). After treatment, 68 of the drug group recovered vs. 52 in the placebo group.
Example 2: Solution
Writing a Conclusion in Context
A strong AP conclusion has four components:
Type I and Type II Errors
Common Mistakes
Question 1 of 5
A researcher randomly assigns 50 volunteers to a new exercise program and 50 to a control group. After 8 weeks, the exercise group lost significantly more weight (\(p\)-value = 0.003). Which conclusion is most appropriate?
Question 2 of 5
In a two-proportion \(z\)-test for an experiment, why do we use a pooled proportion \(\hat{p}_c\) rather than the individual sample proportions?
Question 3 of 5
A two-sample \(t\)-test yields a \(p\)-value of 0.08 at \(\alpha = 0.05\). Which statement is correct?
Question 4 of 5
A study finds that coffee drinkers have lower rates of depression than non-coffee drinkers (\(p = 0.01\)). The data were collected from a large random sample survey. What is the strongest valid conclusion?
Question 5 of 5
In a randomized experiment, a researcher rejects \(H_0\) at \(\alpha = 0.05\). If \(H_0\) is actually true, what type of error has been made?