Test Statistics & p-Values — Test Statistics and p-Values - College Statistics

SECTION 1

Historical Context & Motivation

Long before the modern framework of hypothesis testing existed, scientists grappled with a fundamental question: how can we distinguish genuine effects from random fluctuation? In the eighteenth and nineteenth centuries, astronomers and social scientists relied on informal judgment to decide whether an observed pattern was 'real,' but this approach was subjective and inconsistent. The development of test statistics and p-values provided a rigorous, quantitative answer to this question — one that would become the backbone of scientific inference across disciplines from medicine to economics to psychology.

1710 • Arbuthnot's Significance Argument

John Arbuthnot analyzed London birth records and argued that the consistent excess of male births was too improbable to be mere chance — an early precursor to significance testing.

1900 • Pearson's Chi-Squared Test

Karl Pearson introduced the chi-squared goodness-of-fit test, one of the first formal test statistics, enabling researchers to compare observed data against expected distributions.

1908 • Student's t-Distribution

William Sealy Gosset, publishing under the pseudonym 'Student,' developed the t-distribution for small-sample inference, revolutionizing quality control at the Guinness brewery and beyond.

1925 • Fisher Formalizes the p-Value

Ronald A. Fisher published 'Statistical Methods for Research Workers,' formally defining the p-value as a continuous measure of evidence against the null hypothesis and popularizing the α = 0.05 threshold.

1933 • Neyman–Pearson Framework

Jerzy Neyman and Egon Pearson introduced the concepts of Type I and Type II errors and the power of a test, establishing the decision-theoretic framework that complements Fisher's approach.

The central question that test statistics and p-values address is deceptively simple: if the null hypothesis were true, how surprising would our observed data be? A test statistic compresses the entire dataset into a single number that measures the discrepancy between observed results and what the null hypothesis predicts. The p-value then translates that number into a probability, providing a universal scale of evidence that researchers across all fields can interpret and compare.

SECTION 2

Core Principles & Definitions

Before diving into formulas, it is essential to understand the conceptual architecture of hypothesis testing. Every hypothesis test follows the same logical structure: we begin with a null hypothesis (H₀) that represents the default or status quo claim, and an alternative hypothesis (H₁ or Hₐ) that captures the effect or difference we suspect exists. The test statistic and p-value work together as the evidentiary machinery that determines whether our data provide sufficient grounds to reject H₀.

Test Statistic

A standardized numerical summary of the data that measures how far the observed sample result deviates from what H₀ predicts, expressed in units of standard error. Larger absolute values indicate greater discrepancy with the null.

p-Value

The probability of observing a test statistic as extreme as, or more extreme than, the one computed from the data, assuming H₀ is true. It quantifies the strength of evidence against the null — smaller p-values indicate stronger evidence.

Significance Level (α)

A pre-specified threshold (commonly 0.05) that sets the maximum tolerable probability of a Type I error — rejecting a true null hypothesis. If p ≤ α, the result is declared 'statistically significant.'

Sampling Distribution

The theoretical distribution of the test statistic across all possible samples of the same size drawn from the population described by H₀. This distribution is the reference against which the observed test statistic is compared.

Rejection Region

The set of values of the test statistic that lead to rejection of H₀. Its location (one tail or both tails) depends on whether the alternative hypothesis is one-sided or two-sided.

✦

KEY TAKEAWAY

Think of a hypothesis test like a criminal trial. The null hypothesis is 'the defendant is innocent' — this is the presumed truth until proven otherwise. The test statistic is the totality of the prosecution's evidence compressed into a single score. The p-value is how unlikely that evidence would be if the defendant truly were innocent. A very small p-value is like overwhelming evidence — it compels the jury (the researcher) to reject the presumption of innocence (H₀). Just as courts demand 'beyond reasonable doubt,' statisticians demand that p fall below α before rejecting H₀.

SECTION 3

Visual Explanation — Anatomy of a Hypothesis Test

The following diagram illustrates the complete logical flow of a hypothesis test, from formulating hypotheses to reaching a conclusion. Understanding this pipeline is essential before examining the mathematics; every formula and calculation serves a specific step in this process.

The hypothesis testing pipeline: beginning with hypothesis formulation (Step 1), the researcher sets a significance level α (Step 2), collects data (Step 3), computes the test statistic (Step 4), derives the p-value (Step 5), and compares p to α to reach a conclusion (Step 6).

Notice how the test statistic and p-value occupy the critical middle steps of the pipeline. The test statistic transforms raw data into a standardized scale, and the p-value translates that standardized score into a probability. This two-step transformation is what allows us to compare results across entirely different experimental contexts — a drug trial measuring blood pressure reduction and a psychology experiment measuring reaction times can both yield p-values on the same 0-to-1 scale, making the strength of evidence directly comparable.

SECTION 4

Mathematical Framework

The general structure of every test statistic follows a common template: it measures the distance between the observed sample statistic and the null-hypothesized parameter value, scaled by the variability one would expect due to sampling error. This scaling is what makes different test statistics comparable, regardless of the original units of measurement.

GENERAL TEST STATISTIC TEMPLATE

Test Statistic = (Sample Statistic − Null Parameter Value) / Standard Error

The numerator captures how far the observed result deviates from the null hypothesis. The denominator — the standard error — captures the expected variability of the sample statistic. A large test statistic means the observed deviation is many standard errors away from the null value.

Z-TEST STATISTIC (KNOWN σ)

z = (x̄ − μ₀) / (σ / √n)

Where x̄ is the sample mean, μ₀ is the hypothesized population mean, σ is the known population standard deviation, and n is the sample size. Under H₀, this statistic follows a standard normal distribution N(0, 1).

T-TEST STATISTIC (UNKNOWN σ)

t = (x̄ − μ₀) / (s / √n)

Here s replaces σ as the sample standard deviation. Because s is itself a random variable estimated from data, the resulting statistic follows a t-distribution with df = n − 1 degrees of freedom, which has heavier tails than the standard normal.

P-VALUE DEFINITION (TWO-SIDED)

p = P(|T| ≥ |t_obs| | H₀ is true)

For a two-sided test, the p-value is the probability of observing a test statistic at least as extreme in either direction. For a one-sided test (right-tailed), p = P(T ≥ t_obs | H₀). The p-value is always computed under the assumption that H₀ is true — it is a conditional probability.

⚠

Common Misconception

The p-value is not the probability that H₀ is true. It is the probability of obtaining results as extreme as the observed data, given that H₀ is true. This conditional direction matters enormously: P(data | H₀) ≠ P(H₀ | data). Conflating the two is known as the prosecutor's fallacy or the transposed conditional.

SECTION 5

Reference Distributions & Tail Areas

Every test statistic has a known sampling distribution under H₀, and the p-value is computed as a tail area under that distribution. The choice of reference distribution depends on the type of test, the parameter being tested, and the assumptions satisfied by the data. In introductory statistics, three distributions dominate: the standard normal (z), Student's t, and the chi-squared (χ²). Understanding how to locate the observed test statistic on the appropriate distribution and shade the corresponding tail area is the geometric essence of computing a p-value.

A standard normal (z) distribution curve showing a two-sided p-value as the sum of the two shaded tail areas beyond ±t_obs. The central unshaded region corresponds to test statistic values that would not lead to rejection of H₀.

Test	Test Statistic	Reference Distribution	Typical Use Case
One-sample z-test	z = (x̄ − μ₀) / (σ/√n)	N(0, 1)	Population σ known, large n
One-sample t-test	t = (x̄ − μ₀) / (s/√n)	t(n − 1)	Population σ unknown
Two-sample t-test	t = (x̄₁ − x̄₂) / SE_diff	t(df)	Comparing two means
Proportion z-test	z = (p̂ − p₀) / √(p₀(1−p₀)/n)	N(0, 1)	Testing a population proportion
χ² goodness-of-fit	χ² = Σ(O−E)²/E	χ²(k − 1)	Categorical frequency comparison

Common test statistics and their reference distributions

The key insight connecting all of these tests is that each test statistic converts an observed discrepancy into units of standard error. A z-score of 2.0 means the sample statistic fell two standard errors away from the null value — an event that would occur with only about 4.6% probability (two-tailed) under H₀. The p-value is simply the tail area beyond the observed test statistic on the appropriate reference distribution, making it a universal currency of statistical evidence.

SECTION 6

Worked Example — One-Sample z-Test

A university claims that the mean score on its statistics placement exam is μ = 72 points. A faculty member suspects students score higher than this and collects a random sample of n = 36 students, obtaining a sample mean of x̄ = 75.2. The population standard deviation is known to be σ = 9. We conduct a one-sided z-test at the α = 0.05 significance level.

One-Sample z-Test: Is the True Mean Greater Than 72?

1
Step 1 — State the Hypotheses

The null hypothesis is H₀: μ = 72 (the placement exam mean is 72). The alternative hypothesis is Hₐ: μ > 72 (students actually score higher). Because the alternative is directional, this is a right-tailed test.

H₀: μ = 72 | Hₐ: μ > 72
2
Step 2 — Compute the Standard Error

The standard error of the sample mean is SE = σ / √n = 9 / √36 = 9 / 6 = 1.5. This tells us that sample means from populations with μ = 72 and σ = 9 will typically vary by about 1.5 points from one sample to the next.

SE = 1.5
3
Step 3 — Calculate the Test Statistic

We now apply the z-test formula: z = (x̄ − μ₀) / SE = (75.2 − 72) / 1.5 = 3.2 / 1.5 ≈ 2.133. This means the sample mean fell approximately 2.13 standard errors above the hypothesized mean of 72.

z = 2.133
4
Step 4 — Find the p-Value

Because this is a right-tailed test, the p-value is P(Z ≥ 2.133). Using a standard normal table or software, P(Z ≥ 2.133) ≈ 0.0164. This means that if the true mean were 72, we would see a sample mean of 75.2 or higher only about 1.64% of the time.

p ≈ 0.0164
5
Step 5 — Make the Decision

We compare the p-value to the significance level: p = 0.0164 < α = 0.05. Since the p-value falls below our threshold, we reject H₀. There is statistically significant evidence at the 5% level that the true mean placement exam score exceeds 72 points.

Reject H₀ — evidence supports μ > 72
6
Step 6 — Interpret in Context

The data provide sufficient evidence to conclude that students score higher than the claimed average of 72 on the statistics placement exam. The observed difference of 3.2 points, while perhaps modest in practical terms, is statistically significant. A prudent researcher would also consider the effect size (Cohen's d = 3.2/9 ≈ 0.36, a small-to-medium effect) and whether the difference is educationally meaningful.

Statistically significant; Cohen's d ≈ 0.36

SECTION 7

Strengths, Limitations, and Common Pitfalls

Test statistics and p-values are extraordinarily powerful tools, but they are frequently misunderstood and misapplied. The American Statistical Association issued a formal statement in 2016 warning against mechanical use of p-value thresholds, emphasizing that no single number can capture the complexity of scientific evidence. Understanding both the strengths and limitations of p-values is essential for responsible statistical practice.

Strengths	Limitations
Provides a continuous measure of evidence on a universal 0-to-1 scale, enabling cross-study comparison.	A small p-value does not indicate a large or practically important effect — statistical significance ≠ practical significance.
Grounded in well-understood probability theory; derivations are transparent and reproducible.	Highly sensitive to sample size: with large n, even trivially small effects produce tiny p-values.
Applicable across virtually all parametric and many nonparametric settings — a truly general-purpose tool.	Depends on model assumptions (normality, independence, etc.); violations can invalidate the p-value.
Controls Type I error rate: when used correctly, only α proportion of true nulls are rejected over the long run.	Susceptible to p-hacking: multiple testing, selective reporting, and optional stopping inflate false positive rates.
Easy to compute with modern software; widely understood across disciplines.	Says nothing about the probability that H₀ or Hₐ is true — this requires Bayesian methods.

Strengths and limitations of p-value-based hypothesis testing

✦

KEY TAKEAWAY

Think of the p-value as a metal detector at the beach. A beep (small p-value) tells you something metallic is down there, but it does not tell you whether it is a gold ring or a bottle cap. The p-value detects the existence of an effect, not its magnitude or importance. You always need to pair statistical significance with effect size and domain expertise to make sound conclusions.

SECTION 8

Connection to Confidence Intervals & Bayesian Inference

While p-values remain the dominant tool in classical (frequentist) statistics, they are part of a broader ecosystem of inferential methods. Two closely related approaches — confidence intervals and Bayesian posterior probabilities — address some of the limitations of p-values and are increasingly favored in modern statistical practice. Understanding how these approaches relate to and extend hypothesis testing prepares you for advanced coursework in statistical inference.

Feature	p-Value (Frequentist)	Confidence Interval	Bayesian Posterior
What it answers	How surprising is the data if H₀ is true?	What range of parameter values is consistent with the data?	What is the probability distribution of the parameter given the data?
Effect size information	No — only measures evidence against H₀	Yes — directly estimates magnitude and precision	Yes — provides full posterior distribution
Requires prior beliefs?	No	No	Yes — prior distribution must be specified
Decision rule	Reject H₀ if p ≤ α	Reject H₀ if null value falls outside CI	Depends on loss function or credible intervals
Relationship to p-value	—	A 95% CI and a two-sided α = 0.05 test always agree	Can diverge from p-value, especially with informative priors

Comparing the p-value approach with confidence intervals and Bayesian inference

A particularly elegant connection exists between p-values and confidence intervals: for a two-sided test at significance level α, rejecting H₀ is mathematically equivalent to finding that the null parameter value falls outside the (1 − α) × 100% confidence interval. This duality means that confidence intervals carry strictly more information than p-values — they tell you not just whether to reject H₀, but also the plausible range of the true parameter. As you advance in statistics, you will find that reporting confidence intervals alongside (or instead of) p-values is considered best practice by many journals and professional organizations.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher obtains a p-value of 0.03 in a hypothesis test. Her colleague claims, 'There is only a 3% chance that the null hypothesis is true.' Explain why this interpretation is incorrect and provide the correct interpretation of the p-value.

Show answer

The colleague commits the transposed conditional fallacy. The p-value is P(data as extreme or more | H₀ true), not P(H₀ true | data). The correct interpretation is: if H₀ were true, there is a 3% probability of observing data as extreme as, or more extreme than, what was actually observed. The p-value measures the compatibility of the data with H₀, not the truth or falsehood of H₀. Computing P(H₀ true | data) would require Bayesian methods and a prior probability for H₀.

Question 2: BASIC CALCULATION

A manufacturer claims its light bulbs last μ = 1000 hours on average. A consumer group tests n = 49 bulbs and obtains x̄ = 985 hours. The population standard deviation is σ = 42 hours. Compute the z-test statistic and the two-sided p-value.

Show answer

First, compute the standard error: SE = σ/√n = 42/√49 = 42/7 = 6. Then the z-statistic: z = (x̄ − μ₀)/SE = (985 − 1000)/6 = −15/6 = −2.50. For a two-sided test, p = 2 × P(Z ≤ −2.50) = 2 × 0.0062 = 0.0124. At α = 0.05, we would reject H₀ because 0.0124 < 0.05, concluding there is statistically significant evidence that the true mean lifetime differs from 1000 hours.

Question 3: INTERMEDIATE

A nutritionist hypothesizes that a new diet reduces cholesterol. She samples n = 25 patients and measures their cholesterol reduction, finding x̄ = 12.4 mg/dL with s = 18.0 mg/dL. The null hypothesis is H₀: μ = 0 (no reduction). Compute the t-test statistic, state the degrees of freedom, and determine whether to reject H₀ at α = 0.05 using a one-sided test. The critical t-value for df = 24 at α = 0.05 (one-sided) is 1.711.

Show answer

The standard error is SE = s/√n = 18.0/√25 = 18.0/5 = 3.6. The test statistic is t = (x̄ − μ₀)/SE = (12.4 − 0)/3.6 ≈ 3.444. With df = n − 1 = 24, we compare t = 3.444 to the critical value t₀.₀₅,₂₄ = 1.711. Since 3.444 > 1.711, the test statistic falls in the rejection region. Equivalently, the one-sided p-value for t = 3.444 with 24 df is approximately 0.001, which is well below α = 0.05. We reject H₀ and conclude there is significant evidence that the diet produces a mean cholesterol reduction.

Question 4: APPLIED

A quality engineer tests whether the proportion of defective widgets on a production line exceeds the acceptable rate of 0.02. In a random sample of n = 500 widgets, she finds 18 defective. Set up the hypotheses, compute the z-test statistic for a proportion, find the p-value, and state the conclusion at α = 0.01.

Show answer

Hypotheses: H₀: p = 0.02, Hₐ: p > 0.02 (right-tailed). The sample proportion is p̂ = 18/500 = 0.036. Under H₀, the standard error is SE = √(p₀(1−p₀)/n) = √(0.02 × 0.98/500) = √(0.0000392) ≈ 0.00626. The test statistic is z = (p̂ − p₀)/SE = (0.036 − 0.02)/0.00626 ≈ 2.556. The p-value is P(Z ≥ 2.556) ≈ 0.0053. Since 0.0053 < 0.01 = α, we reject H₀. There is statistically significant evidence at the 1% level that the defective rate exceeds the acceptable threshold of 2%. The engineer should investigate the production process.

Question 5: CRITICAL THINKING

Study A tests a new drug with n = 30 patients and obtains p = 0.048. Study B tests the same drug with n = 3000 patients and obtains p = 0.00001. Both reject H₀ at α = 0.05. (a) Which study provides stronger evidence against H₀? (b) Does a smaller p-value necessarily imply a larger treatment effect? (c) If you were a health policy maker, what additional information beyond the p-values would you request before recommending the drug?

Show answer

(a) Study B provides stronger evidence against H₀ in the sense that p = 0.00001 indicates extreme incompatibility of the data with the null hypothesis. (b) No — a smaller p-value does not necessarily imply a larger effect. Study B's tiny p-value may partly reflect its enormous sample size; with n = 3000, even a clinically trivial effect becomes statistically detectable. (c) A policy maker should request: effect size estimates (e.g., mean difference and confidence intervals), clinical significance thresholds (what minimum benefit is meaningful to patients), information about side effects and safety, the study's design quality (randomization, blinding), reproducibility across independent studies, and cost-effectiveness data. The p-value alone is insufficient for making policy decisions — it must be contextualized within the broader landscape of clinical and practical evidence.

SUMMARY

Lesson Summary

Summary

A test statistic compresses the discrepancy between observed data and the null hypothesis into a single standardized number by dividing the difference (sample statistic minus null parameter) by the standard error. This statistic is then compared to its sampling distribution under H₀ — whether that is the standard normal, the t-distribution, or another reference distribution — to compute the p-value, which represents the probability of observing results as extreme as or more extreme than the data if H₀ were true.

When the p-value falls at or below the pre-specified significance level α, we reject H₀ in favor of the alternative hypothesis. However, a small p-value does not measure effect size or practical importance, nor does it give the probability that H₀ is true. Responsible inference requires pairing p-values with confidence intervals and effect size measures, understanding the assumptions underlying each test, and guarding against pitfalls like p-hacking and the transposed conditional fallacy.