Comparing Groups — I can compare two groups using simulated variability and decide whether a difference is meaningful at my level. - Grade 11 Math

SECTION 1

Historical Context & Motivation

People have always compared groups — does this medicine work better than that one? Do students who study with music score differently from those who study in silence? The real challenge has never been seeing a difference; it's been deciding whether the difference actually means something. Even in a perfectly fair coin-flip experiment, you'll rarely get exactly 50 heads out of 100. Random variation is everywhere, and for centuries scientists struggled to separate genuine effects from noise.

The idea of using randomization to measure what chance alone can produce is surprisingly modern. Before computers, statisticians relied on mathematical formulas and printed tables. Today, we can simulate thousands of random re-assignments in seconds, giving us a powerful and intuitive way to answer the question: "Is this difference real, or could it just be luck?"

1710 • John Arbuthnot's Sex Ratio Test

Arbuthnot noticed that slightly more boys than girls were born in London every year for 82 consecutive years. He argued this couldn't be due to chance alone — one of the earliest recorded significance arguments.

1935 • Fisher's Randomization Test

R. A. Fisher formalized the idea of randomly reassigning group labels to see what differences could arise by chance. His "lady tasting tea" experiment became a famous illustration of this logic.

1969 • Permutation Tests by Computer

As computers became available, researchers could actually carry out thousands of random reshuffles instead of relying on approximate formulas, making simulation-based inference practical.

2000s • Simulation in the Statistics Classroom

Modern statistics curricula shifted toward teaching randomization and simulation before formal formulas. Tools like applets and coding environments put these methods in every student's hands.

The central question this lesson addresses is deceptively simple: when you compare two groups and see a difference, how do you know if it's meaningful or just the result of random variability? Simulation gives us a concrete, visual way to answer that.

SECTION 2

Core Principles & Definitions

Before diving into simulations, you need a clear vocabulary. The logic of comparing groups rests on a few foundational ideas that connect together like links in a chain. Each concept builds on the last, so take a moment to internalize them before moving on.

Observed Difference

The actual difference in a statistic (such as means or proportions) between the two groups in your real data. This is the number you're trying to evaluate.

Null Hypothesis

The default assumption that there is no real difference between the groups — any difference you see is just due to random chance in how individuals ended up in each group.

Simulated Distribution

A collection of differences generated by randomly reassigning group labels many times. This shows what chance alone can produce and serves as your yardstick.

Statistical Significance

A conclusion reached when the observed difference falls far from the center of the simulated distribution — meaning chance alone is unlikely to have produced it.

P-value

The proportion of simulated differences that are as extreme as (or more extreme than) the observed difference. A small p-value suggests the observed difference is unlikely under the null hypothesis.

✦

KEY TAKEAWAY

Think of it like a basketball free-throw contest. If your friend makes 8 out of 10 and you make 6 out of 10, is she really better, or could that 2-shot gap happen just from normal streaky shooting? A simulation is like replaying the contest thousands of times with equally skilled players to see how often a gap of 2 or more shows up by luck alone. If it almost never happens, the gap is probably real.

SECTION 3

Visual Explanation — The Randomization Process

The diagram below walks through the core simulation logic step by step. Imagine you have 20 students: 10 used a new study app and 10 did not. The app group averaged 82 on a test, while the control group averaged 76. The observed difference is 82 − 76 = 6 points. To test whether this 6-point gap is meaningful, you pool all 20 scores together, randomly deal them into two groups of 10, and compute the difference in means. Repeat this hundreds of times to build a picture of what chance alone can produce.

The five-step randomization process: collect data, pool scores, randomly reassign labels, compute the difference in means, and repeat to build a null distribution. The observed difference is then compared against this distribution.

Notice that Steps 3 and 4 form a loop — you repeat them many times. Each iteration gives one simulated difference, and the collection of all those differences forms the null distribution. This distribution is centered near zero because, under the null hypothesis, there's no true difference. If your observed difference of 6 lands far out in the tail, you have evidence that the study app really did have an effect.

SECTION 4

Mathematical Framework

The mathematics behind simulation-based inference is surprisingly straightforward. You don't need calculus or complex formulas — just a clear understanding of differences, counting, and proportions. Here are the key quantities you'll work with.

OBSERVED DIFFERENCE IN MEANS

D_obs = x̄₁ − x̄₂

where x̄₁ is the mean of Group 1 and x̄₂ is the mean of Group 2. This is the number you want to evaluate.

SIMULATED DIFFERENCE (ONE TRIAL)

D_sim = x̄_A − x̄_B

where A and B are the two groups formed by one random shuffle of the pooled data. Each shuffle produces a different D_sim.

ESTIMATED P-VALUE

p ≈ (number of simulations where |D_sim| ≥ |D_obs|) ÷ (total simulations)

This proportion tells you how often random chance alone produces a difference as extreme as what you observed. The absolute value signs make this a two-sided test — you're checking for extreme differences in either direction.

💡

What counts as "significant"?

A common threshold is p < 0.05, meaning the observed difference would occur fewer than 5 times out of 100 by chance alone. However, this cutoff is a convention, not a law of nature. In some fields a stricter threshold (like 0.01) is used, and in exploratory work a looser one may be acceptable. Always state and justify your threshold.

Notice that you don't need a formula for standard deviation or a t-distribution table. The simulation generates the reference distribution directly from the data. This is one of the great advantages of simulation-based methods: they rely on the logic of randomization rather than on assumptions about bell-shaped curves.

SECTION 5

Anatomy of a Null Distribution

The null distribution is the heart of this method. It is a histogram (or dot plot) of all the simulated differences you generated. Because each simulated difference comes from a random shuffle where group labels are meaningless, the distribution is centered near zero. Its spread tells you how much variability you'd expect from chance alone.

The purple histogram shows 1000 simulated differences in means under the null hypothesis. The distribution is roughly centered at zero. The dashed red line marks the observed difference of 6. The red-shaded bars to the right represent simulations that produced a difference ≥ 6, which is a small fraction of the total — suggesting the observed difference is statistically significant.

In the diagram above, the observed difference of 6 sits far to the right of the distribution's center. Very few of the 1000 simulated differences reached 6 or beyond. If, say, only 18 out of 1000 simulations produced a difference of 6 or more (in absolute value), then the estimated p-value would be 18 ÷ 1000 = 0.018. Because 0.018 < 0.05, you would conclude that the observed difference is statistically significant — the study app likely had a real effect.

ℹ

Reading the Null Distribution

If the observed difference lands near the center of the null distribution, it's consistent with chance — not significant. If it lands in the tails (far from zero), it's unusual under the null hypothesis — potentially significant. The farther into the tail, the smaller the p-value.

SECTION 6

Worked Example — Does a Warm-Up Improve Sprint Times?

A PE teacher wants to know whether a specific dynamic warm-up routine improves 100-meter sprint times. She randomly assigns 16 students to two groups: 8 do the warm-up, 8 do not. Sprint times (in seconds) are recorded below.

Warm-Up Group	No Warm-Up Group
13.2, 13.5, 12.8, 13.1, 13.0, 12.9, 13.3, 13.4	14.0, 13.8, 14.2, 13.7, 13.9, 14.1, 13.6, 14.3

Sprint times in seconds for 16 students

Simulation-Based Comparison

1
Step 1 — Compute Observed Means

Warm-Up mean = (13.2 + 13.5 + 12.8 + 13.1 + 13.0 + 12.9 + 13.3 + 13.4) ÷ 8 = 105.2 ÷ 8 = 13.15 seconds. No Warm-Up mean = (14.0 + 13.8 + 14.2 + 13.7 + 13.9 + 14.1 + 13.6 + 14.3) ÷ 8 = 111.6 ÷ 8 = 13.95 seconds.

x̄₁ = 13.15 s, x̄₂ = 13.95 s
2
Step 2 — Compute Observed Difference

D_obs = 13.15 − 13.95 = −0.80 seconds. The negative sign means the warm-up group was faster (lower time is better).

D_obs = −0.80 s
3
Step 3 — State the Null Hypothesis

H₀: The warm-up has no effect. Any difference in means is due to random assignment of students to groups. Under H₀, the group labels are interchangeable.
4
Step 4 — Simulate

Pool all 16 sprint times. Randomly assign 8 to "Group A" and 8 to "Group B." Compute D_sim = mean(A) − mean(B). Repeat 1000 times. Suppose the simulation produces the following summary: out of 1000 trials, 12 had D_sim ≤ −0.80 and 10 had D_sim ≥ 0.80.
5
Step 5 — Estimate the P-value

p ≈ (12 + 10) ÷ 1000 = 22 ÷ 1000 = 0.022. We use a two-sided count because we're interested in any large difference, not just one direction.

p ≈ 0.022
6
Step 6 — Draw a Conclusion

Since p = 0.022 < 0.05, the observed difference of 0.80 seconds is statistically significant at the 5% level. We have convincing evidence that the dynamic warm-up routine reduces sprint times compared to no warm-up.

Reject H₀ — the warm-up appears to improve sprint times.

SECTION 7

Strengths & Limitations of Simulation-Based Inference

Like every statistical method, simulation-based inference has advantages and trade-offs. Understanding these will help you know when to use it and when a different approach might be more appropriate.

Strengths	Limitations
No assumption of normal (bell-shaped) data needed — works for any shape of distribution.	Results vary slightly each time because they're based on random simulations; more trials reduce this variability.
Intuitive and visual — you can literally see what chance looks like.	Requires a computer or calculator for a large number of simulations; not practical by hand.
Works well with small sample sizes where traditional formulas may fail.	Still requires that data collection was done properly (random sampling or random assignment).
Easily adaptable to different statistics: means, medians, proportions, etc.	Statistical significance ≠ practical importance. A tiny difference can be "significant" with large samples.

✦

KEY TAKEAWAY

Simulation-based inference is like a universal adapter for statistics — it works in many situations where traditional formulas require strict assumptions. However, it still can't fix bad data. If your groups weren't randomly formed in the first place, no amount of shuffling will give you a trustworthy answer. Good design first, simulation second.

SECTION 8

Connection to Formal Hypothesis Testing

The simulation approach you've learned is conceptually identical to the formal hypothesis tests you'll encounter later, such as the two-sample t-test and the z-test for proportions. Those tests use mathematical formulas to calculate the same kind of p-value that you estimated through simulation. The logic is the same: assume no difference, measure how surprising your data is under that assumption, and make a decision.

Feature	Simulation-Based Test	Formal t-Test
How is the null distribution created?	By randomly reshuffling the data many times	By using a t-distribution formula
Assumptions about data shape	None — any distribution works	Assumes approximately normal populations (or large samples)
Tools needed	Computer or applet for simulations	Calculator, formula, or software
Ease of explanation	Very intuitive — "what could chance produce?"	Requires understanding of degrees of freedom and distribution theory
P-value	Estimated from simulation counts	Calculated exactly from the t-distribution

As you move into AP Statistics or college-level courses, you'll find that formal tests give more precise p-values and are standardized across studies. But the simulation approach is the foundation of the reasoning behind all of them. If you understand why shuffling labels tells you what chance can produce, you already understand the deepest idea in inferential statistics.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

In a randomization test, why do we shuffle the group labels rather than creating entirely new data? Explain the purpose of this step in your own words.

Show answer

We shuffle group labels because we want to see what differences would arise if group membership didn't matter — that is, if the null hypothesis were true. By keeping the actual data values but reassigning which group each value belongs to, we preserve the overall variability of the data while breaking any real association between group membership and outcome. This lets us build a distribution of differences that reflects chance alone, which serves as the benchmark against which we compare our observed difference.

Question 2: BASIC CALCULATION

A researcher runs a randomization test with 500 simulations. Of those, 35 produce a difference in means with an absolute value at least as large as the observed difference. Estimate the p-value. At the 5% significance level, would you reject the null hypothesis?

Show answer

The estimated p-value is 35 ÷ 500 = 0.07. Since 0.07 > 0.05, you would not reject the null hypothesis at the 5% significance level. This means the observed difference is not unusual enough to conclude that it's due to more than random chance. However, 0.07 is fairly close to 0.05, so the researcher might consider collecting more data or running more simulations for a more precise estimate.

Question 3: INTERMEDIATE

Two groups of students were tested on vocabulary retention. Group A (new method, n = 12) had a mean of 74.5, and Group B (traditional method, n = 12) had a mean of 70.2. A simulation of 1000 random shuffles produced the following results: 42 simulations had a difference ≥ 4.3, and 38 had a difference ≤ −4.3. Compute the two-sided p-value and state your conclusion at the 5% level.

Show answer

The observed difference is 74.5 − 70.2 = 4.3 points. For the two-sided p-value, count all simulations where |D_sim| ≥ 4.3. That's 42 + 38 = 80 simulations. The p-value is 80 ÷ 1000 = 0.08. Since 0.08 > 0.05, we fail to reject the null hypothesis. There is not sufficient evidence at the 5% level to conclude that the new method leads to different vocabulary retention than the traditional method. The difference of 4.3 points, while noticeable, falls within the range that chance alone could plausibly produce.

Question 4: APPLIED

A local coffee shop tests whether playing classical music increases the average tip per customer. Over two weeks, the shop alternates daily between classical music (8 days) and no music (8 days). Classical-music days averaged $3.40 in tips per customer, and no-music days averaged $2.95. A 2000-trial simulation found that only 28 out of 2000 shuffles produced a difference of $0.45 or more (in absolute value). Should the shop conclude that classical music increases tips? Discuss significance and any caveats.

Show answer

The observed difference is $3.40 − $2.95 = $0.45. The p-value is 28 ÷ 2000 = 0.014, which is below 0.05, so the result is statistically significant. There is convincing evidence that the type of music is associated with a difference in tipping. However, several caveats apply: the study alternated days rather than randomly assigning customers, so day-of-week effects, weather, or customer traffic patterns could be confounding variables. Also, statistical significance doesn't tell us about practical importance — the shop should consider whether the $0.45 increase is large enough to justify the cost and logistics of playing classical music daily.

Question 5: CRITICAL THINKING

Suppose a researcher performs a randomization test and gets a p-value of 0.03. She concludes that her treatment definitely caused the improvement. Identify at least two flaws in her reasoning, and explain what additional conditions are needed to claim causation.

Show answer

First, a p-value of 0.03 means the result is unlikely under the null hypothesis, but it doesn't guarantee the null is false — there's still a 3% chance of seeing such a result by luck alone. Saying the treatment "definitely" caused the improvement overstates the evidence. Second, statistical significance alone does not establish causation. Causation requires random assignment of subjects to treatment and control groups to rule out confounding variables. If the study was observational (no random assignment), the difference might be due to pre-existing differences between the groups. Third, a single study, no matter how small the p-value, should ideally be replicated. To claim causation, she needs (1) random assignment, (2) proper control of confounding variables, and (3) recognition that p-values quantify surprise, not certainty.

SUMMARY

Lesson Summary

Summary

When comparing two groups, the key challenge is distinguishing a meaningful difference from one that could arise by random variability. A randomization test addresses this by pooling all observations, randomly reassigning group labels many times, and computing the difference in a chosen statistic (like the mean) for each shuffle. The collection of these simulated differences forms the null distribution, which represents what chance alone can produce under the assumption that there is no real difference between the groups.

The p-value is estimated as the proportion of simulated differences that are as extreme as or more extreme than the observed difference. If the p-value is below a chosen threshold (commonly 0.05), the result is called statistically significant, and you have evidence that the difference is real. Remember: significance depends on proper data collection, does not prove causation on its own, and should always be interpreted alongside practical importance. This simulation-based approach provides the conceptual foundation for all formal hypothesis tests you'll encounter later in statistics.