CI for a Proportion — Compute CI for a proportion...

SECTION 1

Historical Context & Motivation

The problem of estimating population proportions from limited samples has occupied statisticians for centuries. When an epidemiologist reports that 12% of patients in a clinical trial experienced a side effect, the natural follow-up question is: how close is that 12% to the true, unknown population proportion? The confidence interval for a proportion provides a principled answer by constructing a range of plausible values for the population parameter, anchored in the mathematics of the normal approximation to the binomial distribution. This tool is indispensable in biostatistics, where decisions about drug efficacy, disease prevalence, and public health policy rest on proportions derived from finite samples.

1733

De Moivre's Normal Approximation

Abraham de Moivre demonstrated that the binomial distribution could be approximated by the normal curve for large sample sizes, laying the mathematical foundation for z-based inference on counts and proportions.

1812

Laplace's Central Limit Theorem

Pierre-Simon Laplace formalized the Central Limit Theorem, proving that the sampling distribution of a sum (and hence a proportion) converges to normality as the sample size grows—regardless of the underlying distribution.

1934

Clopper–Pearson Exact Interval

C.J. Clopper and E.S. Pearson published the exact confidence interval for a binomial proportion, highlighting the limitations of the normal approximation for small samples and motivating criteria for when the Wald z-interval is appropriate.

1998

Agresti–Coull Adjusted Interval

Alan Agresti and Brent Coull proposed the "add two successes and two failures" adjustment, offering a simple correction that dramatically improves coverage probability compared to the standard Wald interval for moderate sample sizes.

Despite the availability of exact and adjusted methods, the Wald z-interval remains the most widely taught and applied confidence interval for proportions in introductory and intermediate biostatistics courses. Understanding its derivation, assumptions, and limitations equips you to both use it correctly and recognize when more sophisticated alternatives are warranted.

SECTION 2

Core Principles & Definitions

Before computing a confidence interval for a proportion, it is essential to understand the statistical machinery that makes the construction valid. The interval rests on several interconnected ideas: the notion of a population proportion as a fixed but unknown parameter, the behavior of the sample proportion as a random variable, the Central Limit Theorem's guarantee of approximate normality, and the interpretation of the confidence level itself.

Population Proportion (p)

The true fraction of individuals in the entire population possessing a characteristic of interest. This is a fixed parameter—unknown but not random. For example, the true prevalence of hypertension among all adults in a country.

Sample Proportion (p̂)

Computed as p̂ = X / n, where X is the number of successes and n is the sample size. Because different samples yield different values of p̂, it is a random variable with its own sampling distribution.

Standard Error (SE)

The standard deviation of the sampling distribution of p̂, estimated as SE = √[p̂(1 − p̂) / n]. It quantifies sampling variability and shrinks as n increases.

Critical Value (z*)

The number of standard errors needed to capture the central (1 − α) × 100% of the standard normal distribution. For a 95% CI, z* = 1.96. This multiplier sets the width of the interval.

Confidence Level (1 − α)

If we repeated the sampling and CI construction process many times, (1 − α) × 100% of the intervals would contain the true p. It describes the long-run reliability of the procedure, not the probability that a single interval contains p.

✦ KEY TAKEAWAY

Think of a confidence interval as a fishing net cast into a lake. The true proportion p is a specific fish whose exact location you cannot see. A 95% confidence interval means that if you cast 100 nets using the same method, about 95 of them would contain the fish. The net's width (margin of error) depends on how variable your casts are (standard error) and how much of the lake you choose to cover (confidence level). Wider nets catch the fish more often but provide less precise information about where it actually is.

SECTION 3

Visual Explanation — The Sampling Distribution & the Interval

The confidence interval for a proportion derives its validity from the fact that the sampling distribution of p̂ is approximately normal when the sample size is sufficiently large. The diagram below illustrates how the standard normal distribution and the critical value z* combine to define the interval boundaries. The shaded central region represents the proportion of the distribution captured by the confidence level, and the tails represent the probability α that is split equally on both sides.

The bell curve represents the sampling distribution of p̂ under the normal approximation. The shaded central region captures 95% of the distribution's area, bounded by the critical values at p̂ ± z* × SE. The red tail regions each contain α/2 = 2.5% of the area, representing the probability of excluding the true proportion. The purple bracket below shows the resulting confidence interval width.

Notice that the confidence interval is symmetric about p̂. This symmetry follows directly from the normal approximation, which assumes the sampling distribution of p̂ is a symmetric bell curve. In practice, the true binomial distribution of p̂ is slightly skewed when p is far from 0.5 or when n is small, which is one reason the z-interval's coverage can deviate from the nominal level in those settings. The visual also underscores why increasing the sample size narrows the interval: a larger n shrinks SE, which compresses the bell curve horizontally, bringing the critical-value boundaries closer to the center.

SECTION 4

Mathematical Framework

The construction of the z-interval for a proportion begins with the sampling distribution of the sample proportion. Let X ~ Binomial(n, p), where X counts the number of successes in n independent Bernoulli trials each with success probability p. The sample proportion is p̂ = X / n. By the Central Limit Theorem, for large n, the standardized statistic Z = (p̂ − p) / √[p(1 − p) / n] is approximately standard normal. Because the true p is unknown, we substitute p̂ in the standard error, yielding the Wald confidence interval.

SAMPLE PROPORTION

p̂ = X / n

where X = number of successes (events of interest), and n = total sample size.

ESTIMATED STANDARD ERROR

SE(p̂) = √[ p̂(1 − p̂) / n ]

This replaces the true standard deviation √[p(1 − p)/n] with its sample-based estimate, which is necessary because p is unknown. The substitution is justified when n is large enough that the CLT holds.

WALD CONFIDENCE INTERVAL

p̂ ± z* × √[ p̂(1 − p̂) / n ]

Equivalently: ( p̂ − z* × SE , p̂ + z* × SE ). Here z* is the critical value from the standard normal distribution corresponding to the desired confidence level. Common values: z* = 1.645 (90%), z* = 1.960 (95%), z* = 2.576 (99%).

MARGIN OF ERROR

E = z* × √[ p̂(1 − p̂) / n ]

The margin of error E is the half-width of the confidence interval. The full interval is p̂ ± E. The margin of error decreases with √n, so quadrupling the sample size halves the margin of error.

⚠️ Validity Conditions

The normal approximation underlying the z-interval requires that both np̂ ≥ 10 and n(1 − p̂) ≥ 10. Some texts use a threshold of 5 instead of 10. Additionally, the observations must be independent, which is typically satisfied when sampling with replacement or when the sample is less than 10% of the population (the 10% condition).

SECTION 5

Factors Affecting Interval Width

Three primary factors govern the width of the confidence interval for a proportion: the sample size, the sample proportion, and the confidence level. Understanding how each factor influences the interval is critical for study design, particularly in biostatistical applications where sample size determination is often the first step in protocol development. The diagram below visualizes these relationships.

Top left: CI width decreases as n increases (curve flattens), showing diminishing returns. Top right: SE is maximized when p̂ = 0.5 and minimized near 0 or 1—the parabolic shape of p̂(1 − p̂). Bottom: three confidence intervals for the same data at 90%, 95%, and 99% levels, illustrating the precision–confidence tradeoff.

Summary of factors affecting CI width

Factor	Change	Effect on CI Width
Sample size (n)	Increase n	Narrower CI — width ∝ 1/√n
Sample proportion (p̂)	p̂ moves toward 0.5	Wider CI — SE is maximized at p̂ = 0.5
Confidence level (1 − α)	Increase from 95% to 99%	Wider CI — z* increases from 1.96 to 2.576

SECTION 6

Worked Example — Prevalence of Antibiotic Resistance

A hospital infection control team cultures 400 bacterial isolates from clinical specimens and finds that 92 of them are resistant to a first-line antibiotic. Construct a 95% confidence interval for the true proportion of antibiotic-resistant isolates in the hospital's patient population.

95% CI for Antibiotic Resistance Prevalence

Step 1 — Identify Given ValuesNumber of successes (resistant isolates): X = 92. Sample size: n = 400. Desired confidence level: 95%, so α = 0.05 and z* = 1.960.

X = 92, n = 400, z* = 1.960

Step 2 — Compute the Sample Proportionp̂ = X / n = 92 / 400 = 0.230. This is our point estimate of the true resistance proportion.

p̂ = 0.230

Step 3 — Verify Normal Approximation ConditionsCheck: np̂ = 400 × 0.230 = 92 ≥ 10 ✓. Check: n(1 − p̂) = 400 × 0.770 = 308 ≥ 10 ✓. Both conditions are comfortably satisfied, so the z-interval is appropriate.

Conditions met: np̂ = 92, n(1−p̂) = 308

Step 4 — Calculate the Standard ErrorSE = √[p̂(1 − p̂) / n] = √[0.230 × 0.770 / 400] = √[0.1771 / 400] = √0.00044275 = 0.02104.

SE ≈ 0.02104

Step 5 — Compute the Margin of ErrorE = z* × SE = 1.960 × 0.02104 = 0.04124.

E ≈ 0.0412

Step 6 — Construct the Confidence IntervalLower bound: p̂ − E = 0.230 − 0.0412 = 0.1888. Upper bound: p̂ + E = 0.230 + 0.0412 = 0.2712. Therefore, the 95% CI is (0.189, 0.271).

95% CI: (0.189, 0.271)

Step 7 — Interpret the ResultWe are 95% confident that the true proportion of antibiotic-resistant isolates in this hospital's population lies between 18.9% and 27.1%. If the hospital's resistance threshold for changing empiric therapy protocols is 25%, this interval suggests the true rate could plausibly be above or below that threshold, indicating further monitoring or a larger study may be needed.

SECTION 7

Strengths, Limitations & Common Pitfalls

Strengths and limitations of the Wald z-interval for a proportion

Strengths	Limitations
Computationally simple — requires only p̂, n, and z*.	Coverage probability can fall well below the nominal level when n is small or p is near 0 or 1.
Widely understood and reported in biomedical literature, facilitating communication.	Symmetric construction can produce bounds below 0 or above 1, which are meaningless for proportions.
Directly connects to hypothesis testing: the CI excludes p₀ if and only if the z-test rejects H₀: p = p₀.	Relies on the plug-in estimate of SE; when p̂ ≈ 0 or p̂ ≈ 1, SE is severely underestimated.
Easily extended to two-proportion comparisons and stratified analyses.	Assumes independent observations — violated in clustered designs (e.g., patients nested within hospitals).

⚠️ Common Pitfall — Misinterpreting Confidence

A frequent error is stating "there is a 95% probability that the true proportion lies within this interval." In frequentist statistics, the true proportion is a fixed constant — it either is or is not in the interval. The 95% refers to the long-run success rate of the procedure: if you repeated the study infinitely many times, 95% of the resulting intervals would capture the true p.

✦ KEY TAKEAWAY

The Wald z-interval is like a quick field measurement with a tape measure: fast, good enough for most routine applications when the object isn't too small or oddly shaped, but you'd switch to a precision caliper (exact or adjusted methods) when accuracy at the extremes matters. In biostatistics, the extremes correspond to rare diseases or nearly universal exposures—precisely the situations where the Wald interval's performance degrades.

SECTION 8

Connection to Advanced Methods

The Wald z-interval is the starting point in a family of confidence intervals for proportions, each offering different tradeoffs between simplicity and coverage accuracy. Understanding these alternatives contextualizes the z-interval and prepares you for more nuanced statistical practice in clinical research, epidemiology, and regulatory biostatistics.

Comparison of confidence interval methods for a binomial proportion

Method	Key Idea	When to Prefer
Wald (z-interval)	Normal approximation with plug-in SE: p̂ ± z*√[p̂(1−p̂)/n]	Large n, p̂ not extreme; quick calculations and pedagogical settings.
Wilson score interval	Inverts the score test rather than the Wald test; better coverage for small n.	Moderate samples (n = 20–100); p near 0 or 1; recommended by many modern guidelines.
Agresti–Coull	Adds z²/2 pseudo-successes and pseudo-failures before applying the Wald formula: p̃ = (X + 2)/(n + 4) for 95% CI.	Simple adjustment that nearly matches Wilson coverage; practical compromise.
Clopper–Pearson (exact)	Based directly on the binomial distribution; guarantees at least (1−α) coverage.	Small samples; regulatory submissions (FDA guidance); conservative inference.
Bayesian credible interval	Uses a Beta prior on p and computes the posterior interval after observing data.	When prior information is available; adaptive clinical trials; probabilistic interpretation is desired.

In practice, influential papers by Agresti and Coull (1998) and Brown, Cai, and DasGupta (2001) demonstrated through simulation that the Wald interval can have actual coverage probabilities oscillating significantly below the nominal 95% level, even for moderately large samples. This has led to recommendations in many biostatistics textbooks to prefer the Wilson score interval or the Agresti–Coull adjusted interval as default choices. Nonetheless, the Wald interval remains essential to master because it is the conceptual foundation on which these refinements are built, it appears in most published research, and its derivation directly illustrates the general logic of confidence interval construction via pivotal quantities.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

A researcher computes a 95% confidence interval for the proportion of patients who respond to a new therapy and obtains (0.42, 0.58). She states: "There is a 95% probability that the true response rate is between 42% and 58%." Explain why this statement is technically incorrect under the frequentist framework, and provide a correct interpretation.

PROBLEM 2 — BASIC CALCULATION

In a survey of 250 adults, 75 reported receiving a flu vaccine in the past year. Compute the 95% confidence interval for the true vaccination proportion. Use z* = 1.960.

PROBLEM 3 — INTERMEDIATE

A public health department wants to estimate the prevalence of diabetes in a county to within ±3 percentage points with 99% confidence. Using a planning value of p* = 0.10 (based on national estimates), determine the minimum sample size required.

PROBLEM 4 — APPLIED

A Phase III clinical trial enrolls 600 patients, randomized equally to drug (n₁ = 300) and placebo (n₂ = 300). In the drug arm, 198 patients achieve clinical remission; in the placebo arm, 132 do. Compute a 95% CI for each arm's remission proportion separately. Then, without performing a formal two-proportion test, use the two intervals to make a preliminary assessment of whether the drug appears effective.

PROBLEM 5 — CRITICAL THINKING

A colleague reports a 95% Wald CI of (−0.02, 0.08) for the proportion of adverse events in a small pilot study with n = 30 and X = 1. Identify at least two methodological problems with this interval and recommend a more appropriate method. Explain your reasoning.

SUMMARY

Lesson Summary

The confidence interval for a proportion (z-interval) uses the normal approximation to the binomial distribution to construct a range of plausible values for the unknown population proportion p. The formula p̂ ± z* × √[p̂(1 − p̂) / n] combines the sample proportion p̂ as a point estimate, the standard error to quantify sampling variability, and the critical value z* to set the desired confidence level. The validity of this interval depends on satisfying the conditions np̂ ≥ 10 and n(1 − p̂) ≥ 10 and on the independence of observations.

The margin of error E = z* × SE is the half-width of the interval and decreases proportionally to 1/√n, meaning that quadrupling the sample size halves the margin of error. The interval is widest when p̂ = 0.5 and narrows as p̂ approaches 0 or 1. While the Wald z-interval is computationally simple and widely used, its coverage probability can deviate from the nominal level for small samples or extreme proportions. In such cases, the Wilson score interval, Agresti–Coull adjustment, or Clopper–Pearson exact interval provide superior alternatives.