Sample Size & Power — Sample size and power concepts...

SECTION 1

Historical Context & Motivation

Every business experiment—whether it is an A/B test on a website, a clinical trial for a pharmaceutical company, or a marketing campaign pilot—ultimately depends on a deceptively simple question: How many observations do we need? Collect too few data points and you risk failing to detect an effect that truly exists; collect too many and you waste time, money, and organizational attention. The formal statistical framework for answering this question revolves around two intertwined concepts: sample size and statistical power. These ideas did not emerge overnight; they were forged across decades of work in agriculture, manufacturing, and biostatistics before becoming cornerstones of modern business analytics.

1908

Student's t-Distribution

William Sealy Gosset, working at the Guinness Brewery, published the t-distribution under the pseudonym 'Student.' His work demonstrated that small samples behave differently from large ones, laying the groundwork for understanding how sample size affects inference.

1928

Neyman–Pearson Framework

Jerzy Neyman and Egon Pearson introduced the concepts of Type I and Type II errors, creating the formal hypothesis-testing framework that defines statistical power as 1 − β, where β is the probability of a Type II error.

1969

Cohen's Power Analysis

Jacob Cohen published 'Statistical Power Analysis for the Behavioral Sciences,' providing practical tables and formulas for computing required sample sizes. His work made power analysis accessible to researchers outside of pure statistics.

2000s

A/B Testing at Scale

Technology companies like Google, Amazon, and Microsoft began running thousands of online experiments simultaneously. Power analysis became a standard pre-experiment checklist item, ensuring that each test had enough users to yield actionable results.

The central question that links all of these milestones is fundamentally practical: How can we design experiments that reliably distinguish real effects from random noise, while using resources efficiently? In a business context, getting this balance wrong can mean either launching a product feature that has no real impact (a false positive) or abandoning a profitable idea because the test was too small to detect its benefit (a false negative). The remainder of this lesson builds your intuition for navigating that tradeoff.

SECTION 2

Core Principles & Definitions

Before diving into formulas, it is essential to internalize the four foundational pillars that govern the relationship between sample size and statistical power. These four elements are tightly interconnected: changing any one of them directly influences the others. In practice, a business analyst will typically fix three of these quantities and solve for the fourth—most often solving for the required sample size.

Significance Level (α)

The probability of rejecting the null hypothesis when it is actually true—a Type I error or false positive. Conventionally set at 0.05, meaning a 5% chance of claiming an effect exists when it does not.

Statistical Power (1 − β)

The probability of correctly rejecting the null hypothesis when a real effect exists. A Type II error (β) occurs when we fail to detect a true effect. Industry standard targets 80% power (β = 0.20).

Effect Size (d or Δ)

The magnitude of the difference or relationship you want to detect. Larger effects are easier to detect with fewer observations. In business, this is often the minimum detectable effect (MDE)—the smallest improvement worth acting on.

Sample Size (n)

The number of independent observations in each group of the experiment. Larger samples reduce sampling variability, making it easier to distinguish signal from noise. This is typically the quantity analysts solve for.

✦ KEY TAKEAWAY

Think of a power analysis like adjusting a telescope. The effect size is how faint the star is; the sample size is the diameter of the lens; power is the probability you will actually see the star; and alpha is the probability you mistake a speck of dust on the lens for a star. A bigger lens (more data) lets you see fainter stars (smaller effects) with greater confidence.

In most business experimentation workflows, the analyst begins by specifying an acceptable α (usually 0.05), a desired power level (usually 0.80 or 0.90), and the minimum detectable effect that would be economically meaningful. The sample size then falls out of a formula that ties these three inputs together. Understanding the direction of each relationship is crucial: holding everything else constant, decreasing α requires more data, increasing power requires more data, and detecting smaller effects requires more data.

SECTION 3

Visualizing Power: Two Distributions

The most intuitive way to understand statistical power is to visualize two overlapping distributions: one representing the world where the null hypothesis is true (no real effect), and one representing the world where the alternative hypothesis is true (a real effect of size Δ exists). The diagram below shows these distributions for a simple two-group comparison of means, illustrating where α, β, and power live on the same number line.

The violet curve represents the null distribution (no effect). The cyan curve represents the alternative distribution (true effect Δ). The dashed yellow line is the critical value. The red-shaded area (α) is the false-positive zone; the orange-shaded area (β) is the false-negative zone. Power is the remaining area under the cyan curve to the right of the critical value.

Notice that the two curves overlap considerably. The degree of overlap determines the probability of making errors. If you increase the sample size, both distributions become narrower (their standard errors shrink), which pulls them apart and reduces the overlap. This mechanically increases power without changing α. Alternatively, if the true effect size is larger, the alternative curve shifts further to the right, again reducing overlap. These geometric relationships are the visual foundation for every power calculation you will encounter in business experimentation.

SECTION 4

Mathematical Framework

The mathematical relationship between sample size, power, effect size, and significance level can be expressed through a straightforward formula derived from the overlap of the null and alternative distributions. For a two-sample z-test comparing means (with equal group sizes and known variance), the required sample size per group is given by the following expression.

SAMPLE SIZE PER GROUP (TWO-SAMPLE Z-TEST)

n = [(z₁₋α/₂ + z₁₋β)² × 2σ²] / Δ²

Where n = sample size per group, z₁₋α/₂ = critical z-value for the chosen significance level (1.96 for α = 0.05 two-tailed), z₁₋β = z-value corresponding to the desired power (0.842 for 80% power), σ² = population variance, and Δ = minimum detectable effect (difference in means).

Several critical insights emerge from examining the structure of this formula. First, notice that sample size is inversely proportional to Δ². This means that if you want to detect an effect half as large, you need four times as many observations—a quadrupling, not a doubling. This inverse-square relationship is one of the most important practical lessons in experimentation design, because stakeholders often underestimate how dramatically sample size requirements escalate when they request detection of smaller effects.

COHEN'S d (STANDARDIZED EFFECT SIZE)

d = Δ / σ

Cohen's d expresses the effect size in standard-deviation units, enabling comparison across different metrics. Cohen's conventional benchmarks: small = 0.2, medium = 0.5, large = 0.8. In business A/B tests, typical effect sizes are often small (d ≈ 0.05–0.20), demanding large samples.

POWER FUNCTION

Power = 1 − β = Φ(−z₁₋α/₂ + Δ√(n / 2σ²))

Where Φ is the cumulative distribution function of the standard normal distribution. This form shows that power is a monotonically increasing function of n: more data always increases power, but with diminishing returns.

💡 Business Implication

When a product manager asks to detect a 0.5% lift in conversion rate instead of a 1% lift, they are effectively asking for 4× the traffic and 4× the experiment duration. Always translate sample size requirements into resource costs (time, money, opportunity cost) so stakeholders can make informed tradeoff decisions.

SECTION 5

Power Curves & Sensitivity Analysis

Rather than computing a single sample-size number, experienced analysts construct power curves—graphs that plot statistical power on the vertical axis against sample size (or effect size) on the horizontal axis. A power curve answers the question, 'For a given sample size, what is the probability we will detect effects of various magnitudes?' This sensitivity perspective is far more informative than a single-point calculation because, in practice, you rarely know the true effect size in advance.

Three power curves at α = 0.05 for large (d = 0.8), medium (d = 0.5), and small (d = 0.2) effects. The dashed horizontal line marks the conventional 80% power target. Larger effects reach 80% power with far fewer observations. The small-effect curve may not reach 80% power even at n = 500 per group.

The S-shaped form of each power curve reveals an important practical pattern: power gains are steep at first but exhibit diminishing marginal returns as sample size grows. Moving from n = 50 to n = 150 per group might boost power from 40% to 80%, but moving from n = 500 to n = 600 might only push power from 95% to 97%. This diminishing-returns pattern means there is an economically optimal sample size beyond which additional data provides minimal statistical benefit relative to the cost of collecting it. Constructing power curves before launching an experiment allows the analyst to present stakeholders with a clear menu of tradeoffs—different sample sizes mapped to different probabilities of detecting effects of various magnitudes.

Required sample sizes per group for a two-sample z-test at α = 0.05, by effect size and power target.

Effect Size (d)	n per Group (80% Power)	n per Group (90% Power)	Business Example
Large (0.8)	≈ 26	≈ 34	Major UI redesign impact on conversion
Medium (0.5)	≈ 64	≈ 86	New pricing tier effect on revenue per user
Small (0.2)	≈ 394	≈ 526	Button color change impact on click-through

SECTION 6

Worked Example: E-Commerce A/B Test

Suppose you are an analyst at an e-commerce company. The product team wants to test whether a new checkout flow increases the average order value (AOV). Historical data shows that the current AOV is $85 with a standard deviation of $30. The product team considers a $5 increase in AOV (from $85 to $90) to be the minimum improvement worth implementing. You are asked to determine how many users should be assigned to each group (control and treatment) to achieve 80% power at a 5% significance level (two-tailed).

Computing Required Sample Size for an A/B Test

Step 1 — Identify the ParametersWe have: α = 0.05 (two-tailed), so z₁₋α/₂ = 1.96. Desired power = 0.80, so z₁₋β = 0.842. The minimum detectable effect Δ = $90 − $85 = $5. The standard deviation σ = $30.

z₁₋α/₂ = 1.96, z₁₋β = 0.842, Δ = 5, σ = 30

Step 2 — Compute Cohen's dThe standardized effect size d = Δ / σ = 5 / 30 ≈ 0.167. This is a small effect by Cohen's conventions, which means we should expect to need a substantial sample.

d ≈ 0.167 (small effect)

Step 3 — Plug into the Sample Size Formulan = [(z₁₋α/₂ + z₁₋β)² × 2σ²] / Δ². First compute the numerator components: (1.96 + 0.842)² = (2.802)² = 7.851. Then 2σ² = 2 × (30)² = 2 × 900 = 1,800. Numerator = 7.851 × 1,800 = 14,131.8. The denominator is Δ² = 5² = 25.

Numerator = 14,131.8, Denominator = 25

Step 4 — Calculate n and Round Upn = 14,131.8 / 25 = 565.3. Since sample sizes must be whole numbers and we always round up to maintain at least the target power, we need n = 566 users per group.

n = 566 users per group (1,132 total)

Step 5 — Translate to Business TermsIf the website receives approximately 2,000 checkout-eligible users per day and 100% of them are enrolled in the experiment, the test would require about 1,132 / 2,000 ≈ 0.57 days, or roughly one day of traffic. If only 50% of traffic is eligible (due to segmentation or geographic restrictions), the test would take about 1.1 days. This timeline estimate helps stakeholders plan launch schedules and manage expectations.

≈ 1 day at full enrollment, ≈ 1.1 days at 50% enrollment

⚠️ Reality Check

In practice, analysts often add a 10–20% buffer to the computed sample size to account for factors like non-compliance, attrition, or novelty effects that can dilute the observed treatment impact. For this example, a prudent analyst might target 625–680 users per group.

SECTION 7

Tradeoffs: Under-Powered vs. Over-Powered Tests

Business experiments exist on a spectrum between two failure modes. An under-powered test uses too few observations, making it likely that real effects go undetected. An over-powered test uses far more observations than necessary, wasting resources and time that could be allocated to testing other ideas. Understanding the costs of each failure mode helps analysts strike the right balance.

Comparison of risks from under-powered and over-powered experiments.

Dimension	Under-Powered Test	Over-Powered Test
Primary Risk	Failing to detect a real, profitable effect (Type II error)	Detecting statistically significant but trivially small effects
Resource Cost	Low upfront cost, but high opportunity cost from missed improvements	High direct cost—extended test duration, delayed decisions, diverted traffic
Decision Quality	Inconclusive results; 'no significant difference' may mask real effects	Clear results, but may flag effects too small to be operationally meaningful
Organizational Impact	Erodes trust in experimentation; teams conclude 'nothing ever works'	Reduces experimentation velocity; fewer ideas tested per quarter
When It Happens	Common when teams skip power analysis or set unrealistically small MDEs	Occurs when teams run tests far longer than needed or set very large MDEs

✦ KEY TAKEAWAY

Think of experimentation capacity like a manufacturing assembly line. Each test occupies a 'slot' of user traffic and calendar time. An over-powered test is like using a sledgehammer to hang a picture frame—it works, but you've blocked the production line unnecessarily. An under-powered test is like using a rubber mallet on a railroad spike—you'll swing many times and achieve nothing. Right-sizing your sample is about matching the tool to the job so your experimentation program can test the maximum number of high-quality ideas per unit of time.

SECTION 8

Connections to Advanced Experimentation

The introductory power framework covered in this lesson assumes a simple two-group comparison with a fixed sample size determined before the experiment begins. In practice, modern business experimentation extends these foundations in several important directions. Understanding where the basic framework ends and advanced methods begin prepares you for more sophisticated experimental designs.

How introductory power concepts extend to advanced experimentation methods.

Feature	Introductory Framework (This Lesson)	Advanced Extensions
Sample Size	Fixed before experiment starts	Sequential testing: sample size adjusts based on interim results
Number of Groups	Two groups (control vs. treatment)	Multi-arm bandits and factorial designs with many variants
Outcome Metric	Continuous (means comparison)	Proportions, count data, survival/time-to-event metrics
Multiple Testing	Single hypothesis	Bonferroni or FDR corrections when testing many hypotheses
Variance Reduction	Uses raw variance (σ²)	CUPED and stratified randomization reduce effective variance

Two extensions deserve special attention for business students. First, sequential testing (also called group-sequential design) allows analysts to monitor results as data accumulate and stop the experiment early if the evidence is overwhelming—either in favor of or against the treatment. This can dramatically reduce the average sample size needed, though it requires careful adjustment of the significance threshold to avoid inflating the false-positive rate. Second, variance reduction techniques such as CUPED (Controlled-experiment Using Pre-Experiment Data) leverage pre-experiment covariates to shrink the effective variance σ², which in turn reduces the required sample size. Major tech companies report 30–50% reductions in required sample size using these techniques, representing enormous gains in experimentation throughput.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

A colleague proposes running an A/B test with only 30 users per group to detect a 2% increase in conversion rate. Explain why this is problematic, using the concepts of statistical power and Type II error in your reasoning.

PROBLEM 2 — BASIC CALCULATION

An analyst wants to detect a difference of Δ = 10 units between two groups, with σ = 25 for each group. Using α = 0.05 (two-tailed, z₁₋α/₂ = 1.96) and 80% power (z₁₋β = 0.842), compute the required sample size per group using the formula: n = [(z₁₋α/₂ + z₁₋β)² × 2σ²] / Δ².

PROBLEM 3 — INTERMEDIATE

Using the same setup as Problem 2 (Δ = 10, σ = 25, α = 0.05 two-tailed), compute the required sample size per group if the analyst now wants 90% power instead of 80% power. The z-value for 90% power is z₁₋β = 1.282. By what factor does the sample size increase when moving from 80% to 90% power?

PROBLEM 4 — APPLIED

A subscription-based SaaS company wants to test whether a new onboarding email sequence reduces 30-day churn rate from the current baseline of 12% to 10% (a 2 percentage-point reduction). The company acquires 500 new subscribers per week. Using the approximation for proportions where σ² ≈ p(1 − p) and treating p ≈ 0.11 (midpoint), with α = 0.05 (two-tailed) and 80% power, estimate the required sample size per group and how many weeks the test will need to run.

PROBLEM 5 — CRITICAL THINKING

A product manager argues: 'We should just run every A/B test with 99% power and α = 0.01 to be absolutely sure of our results. Why would anyone accept less?' Construct a multi-dimensional counter-argument that addresses statistical, economic, and organizational considerations.

SUMMARY

Lesson Summary

Every well-designed business experiment begins with a power analysis that connects four interdependent quantities: the significance level (α), which controls the false-positive rate; statistical power (1 − β), the probability of detecting a real effect; the effect size (Δ or d), representing the minimum improvement worth detecting; and the sample size (n), the number of observations per group. The core formula n = [(z₁₋α/₂ + z₁₋β)² × 2σ²] / Δ² reveals that sample size scales with the inverse square of the effect size, making small-effect detection disproportionately expensive.

In practice, analysts use power curves to visualize how power changes across sample sizes and effect sizes, enabling stakeholders to make informed resource-allocation decisions. Both under-powered tests (which waste effort by failing to detect real effects) and over-powered tests (which waste resources by collecting unnecessary data) impose costs. The goal is to right-size each experiment so that the organization maximizes its experimentation throughput—the number of high-quality, actionable insights generated per unit of time and traffic.