p-Values & Significance Levels — Interpret p-values and significance levels - Biostatistics

SECTION 1

Historical Context & Motivation

Statistical inference has deep roots in the desire to separate genuine effects from random noise. Long before modern biostatistics, scientists wrestled with a fundamental question: when an observed result deviates from expectation, is that deviation meaningful or merely the product of chance? The formalization of the p-value and the significance level provided the quantitative machinery needed to answer this question with rigor. Understanding the intellectual lineage of these concepts illuminates why they are used—and sometimes misused—in contemporary biomedical research.

1710 • Arbuthnot's Sign Test

John Arbuthnot analyzed London christening records and computed the probability that male births would exceed female births in every year for 82 consecutive years by chance alone, finding it vanishingly small. This is often cited as the first published significance test.

1900 • Pearson's Chi-Square Test

Karl Pearson introduced the chi-square goodness-of-fit test, providing a systematic framework for comparing observed frequencies with theoretical expectations. This marked one of the earliest general-purpose significance tests.

1925 • Fisher's Framework

R. A. Fisher published Statistical Methods for Research Workers, formalized the p-value, and proposed α = 0.05 as a convenient threshold for significance—a convention that persists to this day.

1933 • Neyman–Pearson Decision Theory

Jerzy Neyman and Egon Pearson reframed hypothesis testing as a decision procedure between two hypotheses, introducing Type I error (α) and Type II error (β) rates. Their approach emphasized pre-specified significance levels and power.

2016–present • The Replication Crisis & ASA Statement

The American Statistical Association issued a formal statement on p-values, cautioning against rigid thresholds and mechanical interpretation. This reflected a broader replication crisis in psychology, medicine, and other fields, prompting calls for effect sizes, confidence intervals, and Bayesian alternatives.

The modern practice of hypothesis testing in biostatistics is thus a hybrid of Fisher's evidential approach and the Neyman–Pearson decision-theoretic framework. The central question this lesson addresses is: given a dataset and a null hypothesis, how do we compute and correctly interpret the p-value, and how does the pre-specified significance level α govern our conclusions?

SECTION 2

Core Principles & Definitions

Before diving into calculations, it is essential to establish a precise vocabulary. Hypothesis testing begins with two competing statements about a population parameter: the null hypothesis (H₀), which represents the default assumption of no effect or no difference, and the alternative hypothesis (H₁ or Hₐ), which captures the research claim. The entire inferential apparatus is built around quantifying how surprising the observed data are under H₀.

p-Value

The probability, assuming the null hypothesis is true, of observing a test statistic at least as extreme as the one computed from the sample. It measures the compatibility of the data with H₀.

Significance Level (α)

A pre-specified threshold—typically 0.05, 0.01, or 0.10—representing the maximum probability of a Type I error (rejecting H₀ when it is actually true) that the researcher is willing to tolerate.

Test Statistic

A standardized quantity (e.g., z, t, χ²) computed from sample data that summarizes how far the observed result lies from the value expected under H₀. The sampling distribution of this statistic under H₀ determines the p-value.

Decision Rule

If p ≤ α, we reject H₀ and conclude the result is statistically significant. If p > α, we fail to reject H₀. Note: failing to reject is not the same as accepting H₀.

✦

KEY TAKEAWAY

Think of the p-value as a measure of surprise. Imagine you suspect a coin is biased. You flip it 100 times and get 63 heads. The p-value answers: 'If the coin were perfectly fair, how likely would it be to see 63 or more heads?' A very small p-value means the data would be very surprising under the fair-coin assumption—analogous to finding extremely unlikely results under the null hypothesis. The significance level α is the threshold you set before the experiment to define how much surprise warrants changing your conclusion.

⚠

Critical Misconception

The p-value is not the probability that H₀ is true. It is the probability of the observed data (or more extreme) given that H₀ is true. Confusing P(data | H₀) with P(H₀ | data) is an instance of the prosecutor's fallacy and is one of the most persistent errors in applied statistics.

SECTION 3

Visual Explanation

A p-value is best understood visually as an area in the tail(s) of the sampling distribution under H₀. The diagram below illustrates a two-tailed z-test scenario. The bell curve represents the distribution of the test statistic when the null hypothesis is true. The shaded tail areas correspond to the p-value—the combined probability of observing a test statistic as extreme or more extreme than the one actually obtained.

The standard normal (z) distribution under H₀. The red shaded tails represent the p-value for a two-tailed test—each tail contributes p/2. The dashed vertical lines at ±z* mark the observed test statistic. If the total shaded area (p-value) falls below the pre-specified α, the result lies in the rejection region and we reject H₀.

Several features of this diagram merit emphasis. First, the p-value is always computed under the assumption that H₀ is true—the curve is centered at the null value (here, z = 0). Second, the further the observed test statistic falls into the tails, the smaller the p-value becomes, reflecting increasing incompatibility between the data and H₀. Third, for a one-tailed test, only one tail would be shaded, and the p-value would be half of what it is for a two-tailed test at the same test statistic value. The choice between one-tailed and two-tailed tests must be made before examining the data, based on the directionality of the research hypothesis.

SECTION 4

Mathematical Framework

The calculation of a p-value follows a consistent logical chain regardless of the specific test employed. First, we compute a test statistic that standardizes the difference between the observed sample result and the null-hypothesized value. Then, we determine the probability of obtaining a test statistic as extreme or more extreme under the null distribution. Below we formalize these steps for the most commonly encountered tests in biostatistics.

Z-TEST STATISTIC (ONE-SAMPLE MEAN)

z = (x̄ − μ₀) / (σ / √n)

where x̄ is the sample mean, μ₀ is the hypothesized population mean under H₀, σ is the known population standard deviation, and n is the sample size. The denominator σ/√n is the standard error of the mean.

T-TEST STATISTIC (ONE-SAMPLE, σ UNKNOWN)

t = (x̄ − μ₀) / (s / √n), df = n − 1

When the population standard deviation is unknown, we substitute the sample standard deviation s. The test statistic follows a t-distribution with n − 1 degrees of freedom (df), which has heavier tails than the standard normal—especially for small samples.

P-VALUE COMPUTATION (TWO-TAILED)

p = 2 × P(Z ≥ |z_obs|) or p = 2 × P(T ≥ |t_obs|)

For a two-tailed test, the p-value is twice the probability of the observed (or more extreme) statistic in one tail. For a one-tailed test, omit the factor of 2 and compute the area in the relevant tail only.

DECISION CRITERION

Reject H₀ ⟺ p ≤ α

The significance level α must be chosen before data analysis. Common choices are α = 0.05 (5% Type I error rate), α = 0.01 (1%), and α = 0.10 (10%). The choice depends on the scientific context: clinical trials with serious consequences often require α = 0.01 or lower, while exploratory studies may tolerate α = 0.10.

It is worth noting that the p-value is a continuous measure of evidence against H₀. A p-value of 0.049 and a p-value of 0.051 represent nearly identical levels of evidence, despite falling on opposite sides of the conventional α = 0.05 boundary. Responsible interpretation therefore reports the exact p-value and considers it alongside effect sizes, confidence intervals, and domain knowledge rather than relying solely on the binary significant/not-significant dichotomy.

SECTION 5

One-Tailed vs. Two-Tailed Tests & Error Types

The directionality of the alternative hypothesis determines whether a one-tailed or two-tailed test is appropriate, which in turn affects the computation and magnitude of the p-value. Additionally, every hypothesis test is susceptible to two kinds of errors. Understanding the interplay between tail direction, error types, and significance levels is essential for designing and interpreting studies in biostatistics.

Top: the decision matrix showing the four possible outcomes of a hypothesis test. A Type I error (false positive) occurs at rate α, while a Type II error (false negative) occurs at rate β. Bottom: visual comparison of rejection regions for two-tailed, right-tailed, and left-tailed tests, with the key relationship between one-tailed and two-tailed p-values.

Feature	Two-Tailed Test	One-Tailed Test
Alternative Hypothesis	H₁: μ ≠ μ₀	H₁: μ > μ₀ or H₁: μ < μ₀
Rejection Region	Both tails (each at α/2)	One tail only (entire α in that tail)
p-Value	2 × P(Z ≥ \|z\|)	P(Z ≥ z) or P(Z ≤ z)
Power	Lower for a given α and effect size	Higher—concentrates α in one direction
When to Use	No prior expectation about direction of effect	Strong theoretical or empirical reason to expect a specific direction

Comparison of two-tailed and one-tailed hypothesis tests

SECTION 6

Worked Example

A pharmaceutical researcher wants to determine whether a new antihypertensive drug reduces systolic blood pressure (SBP) below the established population mean of 130 mmHg. A random sample of n = 36 patients treated with the drug yields a sample mean of x̄ = 125 mmHg with a known population standard deviation of σ = 15 mmHg. Test at the α = 0.05 significance level whether the drug lowers SBP.

One-Sample z-Test for Mean SBP Reduction

1
Step 1 — State the Hypotheses

Because the researcher specifically predicts a decrease in SBP, this is a left-tailed test. H₀: μ = 130 (no reduction). H₁: μ < 130 (the drug reduces SBP). We set α = 0.05.

H₀: μ = 130, H₁: μ < 130, α = 0.05
2
Step 2 — Compute the Standard Error

The standard error of the mean is SE = σ / √n = 15 / √36 = 15 / 6 = 2.5 mmHg. This measures the expected variability of the sample mean around the population mean under repeated sampling.

SE = 2.5 mmHg
3
Step 3 — Calculate the Test Statistic

Substituting into the z-formula: z = (x̄ − μ₀) / SE = (125 − 130) / 2.5 = −5 / 2.5 = −2.00. The observed mean is exactly 2 standard errors below the hypothesized population mean.

z = −2.00
4
Step 4 — Find the p-Value

Because this is a left-tailed test, p = P(Z ≤ −2.00). Using a standard normal table or software, P(Z ≤ −2.00) = 0.0228. There is approximately a 2.28% chance of observing a sample mean of 125 or lower if the true population mean were 130.

p = 0.0228
5
Step 5 — Make the Decision

Compare the p-value to α: 0.0228 < 0.05, so we reject H₀. At the 5% significance level, there is sufficient evidence to conclude that the antihypertensive drug reduces systolic blood pressure below the population mean of 130 mmHg. Note that we should also report the effect size (a 5 mmHg reduction) and a 95% confidence interval for the mean (125 ± 1.645 × 2.5 ≈ [120.89, 129.11] for a one-sided bound, or equivalently, the upper one-sided 95% CI for μ is ≤ 129.11) to give clinical context to the statistical significance.

Reject H₀ — statistically significant at α = 0.05

SECTION 7

Strengths, Limitations & Common Misinterpretations

The p-value framework is ubiquitous in biostatistics, but it is neither infallible nor universally sufficient. Appreciating both its power and its pitfalls is a hallmark of statistical literacy. The table below contrasts what p-values do well with their well-documented limitations and the common misinterpretations that plague the biomedical literature.

Strengths	Limitations	Common Misinterpretations
Provides a standardized, objective measure of evidence against H₀	Does not measure the probability that H₀ is true or false	"p = 0.03 means there is a 3% chance H₀ is true" — WRONG
Allows comparison across different study designs and test statistics	Sensitive to sample size — large n can produce small p even for trivial effects	"p > 0.05 means no effect exists" — WRONG; absence of evidence ≠ evidence of absence
Simple decision rule: compare p to α for transparent conclusions	Does not convey the magnitude or clinical importance of the effect	"Smaller p means larger effect" — WRONG; p depends on n and variability too
Well-established regulatory acceptance (FDA, EMA) in clinical trials	Vulnerable to p-hacking: testing many hypotheses or subgroups inflates Type I error	"Statistical significance = clinical significance" — WRONG; always consider effect size

Strengths, limitations, and common misinterpretations of p-values

✦

KEY TAKEAWAY

Think of the p-value as a thermometer rather than a binary switch. A thermometer tells you the temperature, but it does not tell you whether you should wear a jacket—that depends on your comfort threshold, the wind, your activity level, and other factors. Similarly, the p-value quantifies one dimension of evidence (compatibility with H₀), but clinical decision-making requires integrating it with effect sizes, confidence intervals, prior knowledge, biological plausibility, and the costs of Type I versus Type II errors.

SECTION 8

Connection to Confidence Intervals & Bayesian Approaches

The p-value does not exist in isolation—it is deeply connected to other inferential tools. Two of the most important connections are with confidence intervals and Bayesian posterior probabilities. Understanding these connections reveals the strengths and limitations of each approach and points toward more nuanced inference in biostatistics.

Aspect	Frequentist p-Value	Confidence Interval	Bayesian Posterior
What it quantifies	P(data ≥ observed \| H₀ true)	Range of plausible parameter values at (1 − α) coverage	P(H₀ true \| data), given a prior
Effect size info	No — only direction of evidence	Yes — width and location show magnitude and precision	Yes — full distribution of plausible values
Requires a prior?	No	No	Yes — results depend on prior choice
Duality with p-value	—	A 95% CI excludes μ₀ ⟺ p < 0.05 (two-tailed)	No direct equivalence without specifying the prior
Best used when	Quick screening; regulatory hypothesis tests	Communicating precision and practical significance	Incorporating prior knowledge; adaptive trial designs

Comparison of three major inferential frameworks

A particularly elegant result is the duality between hypothesis tests and confidence intervals. For a two-tailed test at level α, rejecting H₀ is equivalent to the (1 − α) × 100% confidence interval for the parameter not containing the null value. This means that if a 95% confidence interval for the mean difference excludes zero, then a two-tailed test at α = 0.05 would reject H₀. Confidence intervals are generally more informative because they convey both the direction and the plausible magnitude of the effect, and the ASA statement encourages reporting them alongside or instead of standalone p-values.

💡

Looking Ahead

Advanced courses in biostatistics will cover multiple testing corrections (Bonferroni, Benjamini–Hochberg FDR), power analysis (choosing n to achieve desired Type II error control), and Bayesian hypothesis testing using Bayes factors. Each of these builds directly on the foundational concepts of p-values and significance levels covered in this lesson.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher reports p = 0.03 for a two-tailed test comparing mean cholesterol levels between a treatment group and a placebo group. A colleague says, 'This means there is only a 3% probability that the drug has no effect.' Explain precisely why this statement is incorrect, and provide the correct interpretation of p = 0.03.

Show answer

The colleague has committed the common error of interpreting the p-value as P(H₀ | data), when it actually represents P(data as extreme or more | H₀). The correct interpretation is: assuming the drug has no effect on cholesterol (H₀ is true), there is a 3% probability of observing a difference between groups at least as large as the one obtained. The p-value is a property of the data under a model, not the probability that the null hypothesis is true. To compute P(H₀ | data) would require Bayes' theorem and a prior probability for H₀, which the frequentist framework does not provide.

Question 2: BASIC CALCULATION

A clinical trial tests whether a new analgesic reduces pain scores below the population mean of μ₀ = 50 on a validated scale. A sample of n = 49 patients yields x̄ = 47 with a known σ = 14. Compute the z-test statistic and the p-value for a left-tailed test. At α = 0.05, do you reject H₀?

Show answer

First, compute the standard error: SE = σ / √n = 14 / √49 = 14 / 7 = 2.0. Next, the z-statistic: z = (47 − 50) / 2.0 = −3 / 2.0 = −1.50. For a left-tailed test, p = P(Z ≤ −1.50). From the standard normal table, P(Z ≤ −1.50) = 0.0668. Since 0.0668 > 0.05, we fail to reject H₀. At the 5% significance level, there is insufficient evidence to conclude that the analgesic reduces pain scores below the population mean, though the p-value is relatively close to the threshold, suggesting a larger sample might yield a different conclusion.

Question 3: INTERMEDIATE

A biostatistician conducts a two-tailed t-test to compare mean hemoglobin levels between two independent groups (n₁ = 20, n₂ = 25). The computed t-statistic is t = 2.45. Using df = 43 (Welch approximation), the software reports p = 0.018. (a) State the decision at α = 0.05. (b) State the decision at α = 0.01. (c) Would a 99% confidence interval for the difference in means include zero? Explain.

Show answer

(a) At α = 0.05, p = 0.018 < 0.05, so we reject H₀ and conclude the difference in mean hemoglobin is statistically significant. (b) At α = 0.01, p = 0.018 > 0.01, so we fail to reject H₀. The result is not significant at the 1% level. (c) A 99% confidence interval corresponds to α = 0.01 for a two-tailed test. Because we failed to reject H₀ at α = 0.01, the 99% CI for the difference in means must include zero. This illustrates the duality between hypothesis tests and confidence intervals: the null value lies inside the (1 − α) CI if and only if p > α.

Question 4: APPLIED

An epidemiological study with n = 10,000 participants finds that a dietary supplement reduces fasting blood glucose by 0.5 mg/dL compared to placebo (s = 12 mg/dL). The two-sample z-test yields p < 0.001. The study authors conclude the supplement is 'highly effective.' Critique this conclusion by considering both statistical and clinical significance. What additional information would you want?

Show answer

While p < 0.001 indicates the observed 0.5 mg/dL difference is very unlikely under H₀, the enormous sample size (n = 10,000 per group) gives the test extreme power to detect even trivially small differences. A 0.5 mg/dL reduction in fasting glucose is clinically negligible—normal fasting glucose ranges from 70 to 99 mg/dL, and clinically meaningful reductions typically start around 5–10 mg/dL. The claim of 'high effectiveness' conflates statistical significance with clinical significance. We would want to see the 95% confidence interval for the difference (approximately 0.5 ± 0.33, or [0.17, 0.83] mg/dL), which, although excluding zero, confirms the effect is tiny. We should also consider effect size measures (Cohen's d ≈ 0.5/12 ≈ 0.04, a negligible effect) and clinical thresholds for meaningful glucose reduction.

Question 5: CRITICAL THINKING

A researcher tests 20 independent hypotheses, each at α = 0.05, and finds that exactly one yields p < 0.05. She reports this single significant result in her paper. (a) Under the assumption that all 20 null hypotheses are true, what is the expected number of false positives? (b) Explain why this scenario illustrates the multiple comparisons problem. (c) If she applies a Bonferroni correction, what adjusted α should she use per test, and would the result with p = 0.04 survive?

Show answer

(a) If all 20 null hypotheses are true and each test is conducted at α = 0.05, the expected number of false positives is 20 × 0.05 = 1.0. Thus, finding exactly one 'significant' result is precisely what chance predicts. (b) This illustrates the multiple comparisons problem because the probability of at least one Type I error across 20 tests is 1 − (1 − 0.05)²⁰ = 1 − 0.95²⁰ ≈ 1 − 0.358 = 0.642, or about 64%. The family-wise error rate is far higher than the nominal 5% per-test rate, making selective reporting of significant results misleading. (c) The Bonferroni-corrected threshold is α_adj = 0.05 / 20 = 0.0025. The result with p = 0.04 would not survive this correction (0.04 > 0.0025), appropriately reflecting the inflated false-positive risk from testing multiple hypotheses.

SUMMARY

Lesson Summary

Summary

This lesson introduced two foundational concepts in biostatistical hypothesis testing. The p-value is the probability, under the assumption that the null hypothesis (H₀) is true, of observing a test statistic at least as extreme as the one calculated from the sample data. The significance level α is a pre-specified threshold representing the maximum tolerable Type I error rate (false positive rate). The decision rule is straightforward: if p ≤ α, we reject H₀; otherwise, we fail to reject H₀.

Critical nuances include: the p-value is not the probability that H₀ is true; statistical significance does not imply clinical significance; large samples can yield small p-values for trivially small effects; and multiple testing inflates the family-wise error rate, requiring corrections such as Bonferroni. For robust inference, always report effect sizes and confidence intervals alongside p-values, leveraging the duality that a (1 − α) CI excluding the null value is equivalent to p < α for a two-tailed test.