Power - College Statistics

SECTION 1

Historical Context & Motivation

The formal theory of hypothesis testing crystallized in the early twentieth century through a spirited intellectual debate between two camps: Ronald A. Fisher, who championed significance testing with p-values, and the partnership of Jerzy Neyman and Egon Pearson, who introduced the concept of an alternative hypothesis and, with it, the notion of statistical power. Fisher's framework asked only whether data were surprising under a null hypothesis; Neyman and Pearson argued that a complete decision procedure must also quantify the probability of detecting a real effect when one exists. This conceptual leap transformed statistics from a tool for measuring surprise into a rigorous framework for making decisions under uncertainty.

1925 • Fisher's Significance Testing

R. A. Fisher publishes Statistical Methods for Research Workers, formalizing the p-value and the concept of rejecting a null hypothesis at a fixed significance level.

1928 • Neyman–Pearson Lemma Foundations

Neyman and Pearson publish their first joint paper proposing a decision-theoretic framework that explicitly considers an alternative hypothesis, laying the groundwork for the Type I / Type II error dichotomy.

1933 • The Neyman–Pearson Lemma

Their landmark paper proves that the likelihood ratio test is the most powerful test of a simple null against a simple alternative for a given significance level, formally defining power as 1 − β.

1969 • Cohen's Power Analysis

Jacob Cohen publishes Statistical Power Analysis for the Behavioral Sciences, providing practical tables and effect-size conventions that made power analysis accessible to applied researchers across disciplines.

1990s–Present • Power in Reproducibility

Growing concerns about the replication crisis elevate power analysis from a methodological nicety to a prerequisite for funding and publication, with journals and agencies requiring prospective sample-size justifications.

The central question that power addresses is deceptively simple: If a real effect exists, how likely is our study to find it? Without answering this question before collecting data, researchers risk investing time and resources into studies that are statistically doomed to fail—a problem that remains at the heart of the modern replication crisis.

SECTION 2

Core Principles & Definitions

To understand power, one must first situate it within the broader Neyman–Pearson framework of hypothesis testing. Every test involves a null hypothesis (H₀) representing the status quo and an alternative hypothesis (H₁ or Hₐ) representing the effect the researcher hopes to detect. Two kinds of errors are possible: a Type I error (rejecting H₀ when it is true, with probability α) and a Type II error (failing to reject H₀ when H₁ is true, with probability β). Power is defined as 1 − β: the probability that the test correctly rejects a false null hypothesis.

Power = 1 − β

Power is the complement of the Type II error rate. A study with β = 0.20 has power = 0.80, meaning it has an 80% chance of detecting a real effect.

Four Determinants

Power depends on four quantities: the significance level (α), the sample size (n), the effect size (δ or d), and the population variability (σ). Changing any one alters power.

Prospective Design Tool

Power analysis is most valuable when performed before data collection (a priori) to determine the minimum sample size needed to detect a meaningful effect at a desired power level.

Effect Size as Input

Power cannot be computed without specifying the magnitude of the effect under H₁. Cohen's conventions (small d = 0.2, medium d = 0.5, large d = 0.8) offer rough benchmarks when prior data are unavailable.

Trade-off with α

Lowering α (making it harder to reject H₀) simultaneously increases β and reduces power. Researchers must balance the costs of Type I and Type II errors for their specific context.

✦

KEY TAKEAWAY

Think of statistical power like a metal detector's sensitivity. Setting α is like choosing how loud the beep must be before you dig (your threshold for acting). Power is the probability the detector actually beeps when there really is metal underground. A larger metal object (bigger effect size), a more sensitive detector (larger n), and a lower threshold for beeping (higher α) all increase the chance you find the buried treasure. Designing a study without a power analysis is like buying a metal detector without checking whether it can sense objects smaller than a car.

SECTION 3

Visual Explanation — Null and Alternative Distributions

The most intuitive way to visualize power is to plot the sampling distributions of the test statistic under both hypotheses on the same axis. The diagram below shows two normal curves: the null distribution centered at μ₀ and the alternative distribution centered at μ₁. The critical value z* (determined by α) divides the null distribution into rejection and non-rejection regions. Power is the area of the alternative distribution that falls in the rejection region—visually, the pink-shaded region to the right of the critical value.

The cyan curve represents the null distribution centered at μ₀. The pink curve represents the alternative distribution centered at μ₁. The dashed yellow line marks the critical value z*. Power is the pink-shaded area of the alternative distribution that falls in the rejection region (right of z*). The purple-shaded region under the alternative curve to the left of z* represents β, the probability of a Type II error.

Notice that as the distance between μ₀ and μ₁ increases (a larger effect size), the two curves separate, and more of the alternative distribution's area falls in the rejection region—thus power increases. Similarly, if we increase the sample size, both distributions become narrower (the standard error shrinks), which also pushes more of the alternative distribution past the critical value. Conversely, choosing a smaller α shifts z* to the right, reducing the rejection region and decreasing power. This visual trade-off is the geometric intuition behind every power calculation.

SECTION 4

Mathematical Framework

We derive the power formula for the most common scenario: a one-sample z-test for a population mean with known variance σ². Consider testing H₀: μ = μ₀ against H₁: μ = μ₁ (where μ₁ > μ₀) at significance level α. Under H₀, the test statistic Z = (X̄ − μ₀) / (σ / √n) follows a standard normal distribution. We reject H₀ when Z ≥ z_α, the upper-α critical value.

TYPE II ERROR PROBABILITY

β = P(Z < zα | μ = μ₁) = Φ(zα − (μ₁ − μ₀)/(σ/√n))

Where Φ is the standard normal CDF, z_α is the upper-α critical value, μ₁ − μ₀ is the true difference, and σ/√n is the standard error of the mean.

POWER

Power = 1 − β = 1 − Φ(zα − δ√n / σ)

Here δ = μ₁ − μ₀ is the raw effect size. Equivalently, letting d = δ/σ (Cohen's d), we write Power = 1 − Φ(z_α − d√n). This form clearly shows that power increases with both effect size d and sample size n.

SAMPLE SIZE FORMULA

n = ((zα + zβ) × σ / δ)²

Rearranging the power equation to solve for n yields this formula, where z_β is the upper-β critical value. For 80% power (β = 0.20), z_β = 0.842; for 90% power, z_β = 1.282.

The sample size formula is the workhorse of prospective power analysis. Notice that n is inversely proportional to δ²: halving the effect size you wish to detect quadruples the required sample size. This quadratic relationship explains why detecting small effects demands large studies. The formula also reveals that σ enters linearly in the numerator, so reducing measurement variability through better instruments or designs is a powerful (and often cheaper) alternative to simply recruiting more subjects.

ℹ

Two-Sided Tests

For a two-sided test (H₁: μ ≠ μ₀), replace z_α with z_α/2 in all formulas above. For example, at α = 0.05, use z_α/2 = 1.96 instead of z_α = 1.645. This reduces power compared to a one-sided test at the same α, because the critical value moves further from the center of the null distribution.

SECTION 5

Factors That Influence Power

Researchers often ask which lever to pull to increase power. The answer depends on which factors are under their control and which are inherent to the phenomenon being studied. The diagram below illustrates how each of the four key determinants shifts the balance between Type II error and power.

Each panel isolates one determinant of power while holding the other three constant. The bar heights and numeric labels show how power changes. Notice that sample size and effect size have the strongest influence, while variability operates in the inverse direction—greater variability reduces power.

Factor	Direction of Change	Effect on Power	Researcher Control?
Sample size (n)	↑ Increase	↑ Power increases	Yes — primary lever
Effect size (d)	↑ Increase	↑ Power increases	Usually fixed by nature, but can be maximized by studying conditions that maximize the treatment contrast
Significance level (α)	↑ Increase	↑ Power increases	Yes, but constrained by convention (α = 0.05) and the cost of false positives
Variability (σ)	↓ Decrease	↑ Power increases	Partially — via better measurement, homogeneous samples, or covariates

Summary of the four determinants of power and whether each is under researcher control.

SECTION 6

Worked Example — A Priori Power Analysis

A clinical researcher wants to test whether a new cognitive training program improves working memory scores compared to a known population mean of μ₀ = 100 (σ = 15). Previous pilot data suggest the training raises scores by about 5 points, so μ₁ = 105. Using a two-sided test at α = 0.05, how many participants are needed for 80% power?

Sample Size for Detecting a 5-Point Improvement

1
Step 1 — Identify the Given Values

We have μ₀ = 100, μ₁ = 105, σ = 15, α = 0.05 (two-sided), and desired power = 0.80 (so β = 0.20).

δ = μ₁ − μ₀ = 5, σ = 15, α = 0.05, β = 0.20
2
Step 2 — Compute Cohen's d

Cohen's d standardizes the effect size by dividing the raw difference by the population standard deviation: d = δ / σ = 5 / 15 = 0.333. This falls between Cohen's conventions for a small (0.2) and medium (0.5) effect.

d = 0.333 (small-to-medium effect)
3
Step 3 — Look Up Critical z-Values

For a two-sided test at α = 0.05, z_α/2 = 1.96. For 80% power (β = 0.20), z_β = 0.842.

z_α/2 = 1.96, z_β = 0.842
4
Step 4 — Apply the Sample Size Formula

n = ((z_α/2 + z_β) × σ / δ)² = ((1.96 + 0.842) × 15 / 5)² = (2.802 × 3)² = (8.406)² = 70.66.

n = 70.66
5
Step 5 — Round Up and Interpret

Since we cannot have a fractional participant, we round up to the next whole number. The researcher needs at least 71 participants to achieve 80% power for detecting a 5-point improvement with σ = 15 at the α = 0.05 significance level (two-sided). If the researcher wanted 90% power, replacing z_β = 1.282 would yield n = ((1.96 + 1.282) × 3)² = (9.726)² = 94.6 ≈ 95 participants.

n = 71 for 80% power; n = 95 for 90% power

SECTION 7

Strategies for Increasing Power & Common Pitfalls

In practice, researchers face budget and logistical constraints that limit sample size, making it essential to consider all available strategies for boosting power. At the same time, several common mistakes can lead to misleading power calculations or inappropriate interpretations.

Strategy / Pitfall	Description	Impact on Power
✓ Increase sample size	The most direct and universally applicable approach—collect more data to narrow the sampling distribution.	Strong positive; power scales with √n.
✓ Reduce variability	Use more precise instruments, restrict the population (e.g., narrow age range), or include covariates (ANCOVA) to absorb extraneous variance.	Moderate to strong; equivalent to increasing d.
✓ Use a one-sided test	When the direction of the effect is firmly established a priori, a one-sided test uses a less extreme critical value and increases power.	Modest positive; but must be justified before data collection.
✓ Use within-subjects designs	Paired or repeated-measures designs eliminate between-subject variability, often dramatically reducing the error term.	Strong positive when within-subject correlation is high.
✗ Post hoc power analysis	Computing power after obtaining a non-significant result using the observed effect size is circular and provides no useful information—it is a one-to-one function of the p-value.	Misleading; widely criticized in the literature (Hoenig & Heisey, 2001).
✗ Ignoring multiple comparisons	When testing multiple hypotheses, corrections like Bonferroni reduce per-comparison α, which lowers power unless sample size is increased accordingly.	Negative if not planned for; can render studies severely underpowered.

Legitimate strategies (✓) and common pitfalls (✗) related to statistical power.

✦

KEY TAKEAWAY

Power analysis is a design tool, not a post-mortem. Like checking your fuel gauge before a road trip rather than after you've run out of gas, a priori power analysis ensures your study has the resources to reach its destination. Post hoc power calculations, by contrast, simply re-express the p-value in a different metric and provide no additional insight.

SECTION 8

Connections to Advanced Theory

The concept of power extends naturally into several advanced areas of statistics and study design. Understanding these connections enriches one's appreciation of why power analysis occupies such a central role in modern research methodology.

Introductory Concept	Advanced Extension	Key Idea
Power for a z-test	Power for t-tests and ANOVA	When σ is unknown and estimated from data, the test statistic follows a t-distribution (or F-distribution). Power is computed using the non-central t (or F) distribution, where the non-centrality parameter λ = d√n encodes the signal-to-noise ratio.
Fixed-sample power analysis	Sequential / adaptive designs	Instead of fixing n in advance, sequential designs allow researchers to analyze data at interim looks and stop early for efficacy or futility, maintaining overall Type I error control while often reducing the expected sample size.
Cohen's conventions for d	Minimum detectable effect (MDE)	Rather than relying on generic benchmarks, researchers increasingly specify the smallest effect size of practical importance (the MDE) and design their study to detect it, connecting power analysis to substantive theory.
Frequentist power	Bayesian design analysis	Bayesian counterparts replace 'probability of rejecting H₀' with metrics like the probability that a credible interval excludes zero or the expected Bayes factor, offering a richer decision framework that can incorporate prior information.

Connections between introductory power concepts and advanced statistical methodology.

As you progress through advanced coursework, you will encounter the non-central distribution framework repeatedly—it is the unifying mathematical structure behind power calculations for virtually all parametric tests. Software packages such as G*Power, R's pwr package, and Python's statsmodels.stats.power module automate these calculations, but understanding the underlying logic ensures you can critically evaluate the assumptions built into any power analysis.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher conducts a two-sample t-test and obtains a p-value of 0.23, concluding that there is no significant difference. A colleague suggests computing 'post hoc power' to determine whether the study had enough power. Explain why this suggestion is problematic and what the researcher should have done instead.

Show answer

Post hoc (observed) power is a direct monotonic function of the p-value: when p > α, observed power is necessarily below 50%. Computing it after the fact provides no information beyond what the p-value already conveys—it simply re-expresses the same result in different units. This circularity has been extensively documented (Hoenig & Heisey, 2001). The researcher should have conducted an a priori power analysis before data collection, specifying the minimum effect size of practical interest and determining the sample size needed to detect it with adequate power (typically 0.80 or 0.90). If the study is already complete and non-significant, the appropriate follow-up is to report a confidence interval for the effect size, which communicates both the estimate and the precision of the study.

Question 2: BASIC CALCULATION

A one-sided z-test is planned at α = 0.05 to detect a mean difference of δ = 3 when σ = 12 and n = 64. What is the power of this test? (Use z_α = 1.645.)

Show answer

First, compute the standard error: SE = σ/√n = 12/√64 = 12/8 = 1.5. Next, compute the non-centrality parameter (or equivalently, the z-value shift): d√n/σ = δ/SE = 3/1.5 = 2.0. Now find β: β = Φ(z_α − δ/SE) = Φ(1.645 − 2.0) = Φ(−0.355) ≈ 0.3613. Therefore, Power = 1 − 0.3613 = 0.639, or approximately 64%. This is below the conventional 80% threshold, suggesting the study is underpowered for this effect size. The researcher would need to increase n to achieve adequate power.

Question 3: INTERMEDIATE

Using the same setup as Problem 2 (δ = 3, σ = 12, one-sided α = 0.05), how large must n be to achieve 80% power? Verify your answer by computing the power at that n.

Show answer

Apply the sample size formula: n = ((z_α + z_β) × σ / δ)² = ((1.645 + 0.842) × 12/3)² = (2.487 × 4)² = (9.948)² = 98.96. Round up to n = 99. Verification: SE = 12/√99 = 12/9.950 = 1.206. Power = 1 − Φ(1.645 − 3/1.206) = 1 − Φ(1.645 − 2.487) = 1 − Φ(−0.842) = 1 − 0.200 = 0.800. The formula correctly yields the sample size needed for exactly 80% power. Note that increasing from n = 64 to n = 99 (a 55% increase in sample size) raised power from 64% to 80%.

Question 4: APPLIED

A pharmaceutical company plans a two-sided test (α = 0.05) comparing a new drug's mean blood pressure reduction (μ₁) to a placebo (μ₀ = 0 mmHg reduction). Prior trials suggest the drug reduces BP by 4 mmHg with a pooled standard deviation of 10 mmHg. The company can afford to recruit at most 200 participants (100 per group). Using the two-sample z-test approximation n_{per group} = 2 × ((z_α/2 + z_β) / d)², does the company have sufficient power at 80%? What about at 90%?

Show answer

Cohen's d = δ/σ = 4/10 = 0.4. For 80% power: n_{per group} = 2 × ((1.96 + 0.842)/0.4)² = 2 × (2.802/0.4)² = 2 × (7.005)² = 2 × 49.07 = 98.14 ≈ 99 per group. Since the company can afford 100 per group, it narrowly exceeds the requirement for 80% power. For 90% power: n_{per group} = 2 × ((1.96 + 1.282)/0.4)² = 2 × (3.242/0.4)² = 2 × (8.105)² = 2 × 65.69 = 131.38 ≈ 132 per group. The company cannot afford 132 per group with a total budget of 200, so 80% power is achievable but 90% power is not within the budget. To reach 90% power, the company could consider reducing σ through stricter inclusion criteria or using a more precise measurement instrument.

Question 5: CRITICAL THINKING

A social scientist argues: 'Since power depends on the true effect size, and we never know the true effect size, power analysis is fundamentally unknowable and therefore useless.' Critically evaluate this claim. In your response, discuss at least two approaches researchers use to handle the unknown true effect size in practice, and explain why power analysis remains valuable despite this uncertainty.

Show answer

The claim correctly identifies that power is a function of the unknown population effect size, but it overstates the implication. First, researchers can use pilot studies or meta-analyses of prior research to obtain informed estimates of the likely effect size, treating power analysis as a sensitivity exercise across a plausible range rather than a single-point calculation. Second, the minimum detectable effect (MDE) approach reframes the question: instead of asking 'how much power do I have?', the researcher asks 'what is the smallest effect I can reliably detect with my available resources?' If that MDE exceeds what is practically meaningful, the study should be redesigned or abandoned. Third, power curves (plotting power as a function of effect size for a given n) allow transparent communication of a study's sensitivity profile without committing to a single assumed effect size. Power analysis remains valuable precisely because it forces researchers to make their assumptions explicit, plan resource allocation rationally, and avoid conducting studies that are doomed to be inconclusive—all of which mitigate the replication crisis far more effectively than ignoring the issue entirely.

SUMMARY

Summary — Statistical Power

Summary

Statistical power is the probability that a hypothesis test correctly rejects a false null hypothesis, formally defined as 1 − β where β is the Type II error rate. Power is determined by four factors: sample size (n), effect size (d), significance level (α), and population variability (σ). Increasing n or d raises power, increasing σ lowers it, and raising α increases power at the cost of more Type I errors.

The key formula for a one-sample z-test is n = ((zα + zβ) × σ / δ)², which reveals that the required sample size scales with the square of the inverse effect size. A priori power analysis is an essential component of responsible study design, while post hoc power analysis is circular and uninformative. A conventional target of 80% power balances the costs of Type I and Type II errors for most research contexts. Advanced extensions include non-central distributions for t-tests and ANOVA, sequential designs, and Bayesian design analysis.