01 / 21
?

Are My Results
Unexpected?

Learn how statisticians determine whether observed outcomes are surprising or simply the result of random chance.

AP Statistics

What Does "Unexpected" Mean in Statistics?

In everyday life, we constantly judge whether outcomes are surprising. If you flip a coin 10 times and get 10 heads in a row, you would probably be shocked. But if you got 6 heads, you might shrug it off.

Statistics gives us a formal framework to decide: is an observed result so far from what we expected that it is unlikely to have happened by chance alone?

Expected vs. Observed Outcomes

Expected Outcome
What a model or assumption predicts should happen. For a fair coin flipped 100 times, we expect about 50 heads.
Observed Outcome
What actually happens when we collect data. In our experiment, we might observe 62 heads out of 100 flips.

Variability and the Bell Curve

When we repeat an experiment many times, results cluster around the expected value but vary due to random chance. This spread of possible outcomes forms a distribution.

Expected (50) Observed (62) Distribution of outcomes if coin is fair 30 70

The Null Hypothesis \( H_0 \)

To decide whether a result is unexpected, we first need a baseline assumption to compare against. In statistics, this is called the null hypothesis (\( H_0 \)).

We also define an alternative hypothesis (\( H_a \)) that captures what we suspect might be true instead: for example, \( H_a: p \neq 0.5 \) (the coin is biased).

The \( p \)-value: Measuring Surprise

The \( p \)-value quantifies how surprising our observed data would be if the null hypothesis were true.

Small \( p \)-value
The observed result is rare under \( H_0 \). Strong evidence against the null hypothesis.
Large \( p \)-value
The observed result is common under \( H_0 \). Not enough evidence to reject it.

The Significance Level \( \alpha \)

Before collecting data, we choose a threshold called the significance level (\( \alpha \)) to decide what counts as "unexpected enough."

Reject Reject Fail to reject \( H_0 \) \( \alpha/2 \) \( \alpha/2 \) Two-tailed test with \( \alpha = 0.05 \)

Statistical vs. Practical Significance

A result can be statistically significant (unlikely by chance) without being practically significant (meaningful in the real world). These are different questions.

Statistical Significance Practical Significance
Question Is the result unlikely by chance? Is the result large enough to matter?
Measured by \( p \)-value Effect size, context
Example A drug lowers blood pressure by 0.5 mmHg (\( p = 0.01 \)) Is 0.5 mmHg clinically meaningful? Probably not.
Depends on Sample size, variability Domain knowledge, stakes

How to Determine if Results Are Unexpected

1
State the hypotheses. Write \( H_0 \) (no effect/difference) and \( H_a \) (the claim you are testing).
2
Choose \( \alpha \). Set your significance level before collecting data (usually \( \alpha = 0.05 \)).
3
Collect data and compute the test statistic. Summarize how far the observed result is from what \( H_0 \) predicts.
4
Find the \( p \)-value. Calculate the probability of seeing a result this extreme under \( H_0 \).
5
Make a decision. If \( p < \alpha \), reject \( H_0 \). Otherwise, fail to reject \( H_0 \).

Example 1: The Coin Flip Test

You flip a coin 100 times and get 62 heads. Is this coin fair?

1
Hypotheses: \( H_0: p = 0.5 \) (fair coin) vs. \( H_a: p \neq 0.5 \) (biased coin)
2
Significance level: \( \alpha = 0.05 \)
3
Test statistic: \( z = \dfrac{\hat{p} - p_0}{\sqrt{\frac{p_0(1-p_0)}{n}}} = \dfrac{0.62 - 0.50}{\sqrt{\frac{0.25}{100}}} = \dfrac{0.12}{0.05} = 2.4 \)
4
\( p \)-value: \( P(|Z| \geq 2.4) \approx 0.016 \)
5
Decision: Since \( 0.016 < 0.05 \), we reject \( H_0 \). The result is statistically unexpected.

Visualizing the Coin Flip Result

\( z = 2.4 \) \( z = 0 \) \( \frac{p}{2} = 0.008 \) \( \frac{p}{2} = 0.008 \) Standard normal distribution under \( H_0 \)

Example 2: A Not-So-Surprising Result

You flip the same coin 100 times and this time get 53 heads. Is this unexpected?

1
Hypotheses: \( H_0: p = 0.5 \) vs. \( H_a: p \neq 0.5 \)
2
Test statistic: \( z = \dfrac{0.53 - 0.50}{0.05} = 0.6 \)
3
\( p \)-value: \( P(|Z| \geq 0.6) \approx 0.549 \)
4
Decision: Since \( 0.549 > 0.05 \), we fail to reject \( H_0 \). This result is not unexpected: 53 heads is perfectly consistent with a fair coin.

Example 3: Sample Size Matters

Now suppose you flip the coin 1000 times and get 530 heads (\( \hat{p} = 0.53 \)). Same proportion as before: is it unexpected now?

1
Standard error: \( SE = \sqrt{\dfrac{0.5 \times 0.5}{1000}} = \sqrt{0.00025} \approx 0.0158 \)
2
Test statistic: \( z = \dfrac{0.53 - 0.50}{0.0158} \approx 1.90 \)
3
\( p \)-value: \( P(|Z| \geq 1.90) \approx 0.057 \)
4
Decision: At \( \alpha = 0.05 \), we barely fail to reject. But the same \( \hat{p} = 0.53 \) is much closer to significance with a larger sample.

Why Does This Logic Work?

1
Random processes have predictable patterns. When we know the probability model, we know the distribution of possible outcomes.
2
Most results cluster near the expected value. The Central Limit Theorem tells us that sample proportions and means are approximately normal for large samples.
3
Extreme results are rare under the null. If \( H_0 \) is true, landing far out in the tail is unlikely, so doing so is evidence against \( H_0 \).
4
The \( p \)-value quantifies "how rare." It converts the distance from the expected value into a probability we can compare against \( \alpha \).

Common Mistakes to Avoid

✕ "We proved \( H_0 \) true"
Failing to reject \( H_0 \) is not the same as proving it. We simply lack evidence against it.
✕ "\( p \)-value = probability \( H_0 \) is true"
The \( p \)-value is the probability of the data given \( H_0 \), not the probability of \( H_0 \) given the data.
✕ Choosing \( \alpha \) after seeing data
Setting \( \alpha \) after computing the \( p \)-value allows cherry-picking. Always choose \( \alpha \) before collecting data.

Summary: Key Concepts at a Glance

Concept Symbol What It Means
Null Hypothesis \( H_0 \) Baseline: nothing unusual is happening
Alternative Hypothesis \( H_a \) The claim you want to find evidence for
\( p \)-value \( p \) Probability of data this extreme under \( H_0 \)
Significance Level \( \alpha \) Threshold for "unexpected" (usually 0.05)
Reject \( H_0 \) \( p < \alpha \) Evidence suggests \( H_0 \) is false
Fail to Reject \( H_0 \) \( p \geq \alpha \) Not enough evidence to reject \( H_0 \)

Worked Example: Is This Die Fair?

Let us return to the die-rolling scenario from Section 3. You suspect a six-sided die may be loaded, so you roll it 120 times and record the following frequencies: Face 1 → 25, Face 2 → 15, Face 3 → 20, Face 4 → 30, Face 5 → 18, Face 6 → 12. Perform a chi-square goodness-of-fit test at the α = 0.05 significance level. Chi-Square Goodness-of-Fit: Fair Die Step 1 — State Hypotheses H₀: The die is fair; each face has probability p = 1/6. Hₐ: The die is not fair; at least one face has a probability different from 1/6. Step 2 — Check Conditions Assume the 120 rolls are independent (reasonable for a physical die). Each expected count is E = 120 × (1/6) = 20, which satisfies the requirement that all expected counts be at least 5. The conditions for a chi-square goodness-of-fit test are met. All expected counts = 20 ≥ 5 ✓ Step 3 — Compute the Chi-Square Statistic χ² = (25 − 20)²/20 + (15 − 20)²/20 + (20 − 20)²/20 + (30 − 20)²/20 + (18 − 20)²/20 + (12 − 20)²/20 = 25/20 + 25/20 + 0/20 + 100/20 + 4/20 + 64/20 = 1.25 + 1.25 + 0 + 5.00 + 0.20 + 3.20 = 10.90 χ² = 10.90 Step 4 — Determine Degrees of Freedom & P-Value df = k − 1 = 6 − 1 = 5. Using a calculator: χ²cdf(10.90, 1E99, 5) ≈ 0.0534. Alternatively, consulting a chi-square table, the critical value for df = 5 at α = 0.05 is 11.07. p-value ≈ 0.0534, df = 5 Step 5 — State the Conclusion in Context Since the p-value (≈ 0.0534) is greater than α = 0.05, we fail to reject H₀. There is not sufficient evidence at the 5% significance level to conclude that the die is unfair. While the deviations are noticeable—particularly for faces 4 and 6—they are not statistically significant by this standard. Note, however, that the result is borderline; a researcher using α = 0.10 would reject H₀. Fail to reject H₀ at α = 0.05. Insufficient evidence that the die is unfair.

Six observed die-face counts 25,15,20,30,18,12 sum to 120. The fair-model expected count is 20 per face.
Observed face counts total 120, while the fair-die model expects 20 per face; one fair run need not match those expectations exactly.
Six labeled die-face icons with exactly one through six pips.
Six labeled die-face icons with exactly one through six pips.

Question 1 of 5

A researcher tests whether a die is fair by rolling it 600 times. She observes 115 sixes. If the die is fair, the expected number of sixes is 100. What is the null hypothesis?

Question 2 of 5

A test yields a \( p \)-value of 0.03. At the \( \alpha = 0.05 \) significance level, what should you conclude?

Question 3 of 5

Which of the following is a correct interpretation of a \( p \)-value of 0.02?

Question 4 of 5

Two studies test the same hypothesis. Study A has \( n = 50 \) and Study B has \( n = 5000 \). Both observe the same sample proportion \( \hat{p} = 0.54 \) under \( H_0: p = 0.50 \). Which study is more likely to reject \( H_0 \)?

Question 5 of 5

A researcher fails to reject \( H_0 \) at the \( \alpha = 0.05 \) level. Which statement is most accurate?