CIs from Asymptotic Normality — Confidence intervals from...

SECTION 1

Historical Context & Motivation

The problem of quantifying uncertainty in statistical estimation is as old as the discipline itself. Early statisticians working with small samples from well-understood distributions—such as the normal or binomial—could rely on exact pivotal quantities to form confidence intervals. However, the vast majority of models encountered in practice do not yield tractable finite-sample distributions for their estimators. This gap between theory and application motivated a fundamentally different approach: rather than deriving the exact distribution of an estimator, one could characterize its limiting distribution as the sample size grows, and use that approximation to construct interval estimates. The intellectual threads leading to this idea span more than two centuries of probability theory and mathematical statistics.

1810

Laplace's Central Limit Theorem

Pierre-Simon Laplace establishes that sums of independent random variables converge in distribution to a Gaussian, providing the foundational tool for asymptotic normality arguments.

1920s

Fisher's Maximum Likelihood Framework

Ronald A. Fisher formalizes maximum likelihood estimation (MLE) and conjectures that MLEs are asymptotically normal with variance given by the inverse Fisher information, laying the groundwork for large-sample inference.

1930s–40s

Cramér, Rao & Regularity Conditions

Harald Cramér and C. R. Rao rigorously establish the information inequality and the conditions under which MLEs achieve asymptotic efficiency, enabling formal justification of asymptotic confidence intervals.

1948

Wald's General Decision Theory

Abraham Wald provides a unified decision-theoretic framework in which asymptotic confidence procedures are analyzed alongside hypothesis tests, introducing the Wald test and its associated interval.

1970s–Present

Modern Extensions

Researchers develop delta-method intervals, bootstrap refinements, and profile-likelihood alternatives, extending asymptotic normality-based CIs to complex models including generalized linear models, survival analysis, and causal inference.

The central question this lesson addresses is: When we know an estimator is asymptotically normal, how do we translate that distributional result into a practical confidence interval, and what are the trade-offs involved? Understanding this construction is essential because the vast majority of confidence intervals encountered in applied statistics—from logistic regression coefficients to hazard ratios—rely on exactly this asymptotic machinery.

SECTION 2

Core Principles & Definitions

Before constructing any interval, we need a firm grasp on the foundational ideas that make asymptotic confidence intervals possible. These principles connect limiting distribution theory to the practical mechanics of interval estimation, and understanding each one is crucial for both correct application and for recognizing when the approach may fail.

Asymptotic Normality

An estimator θ̂ₙ is asymptotically normal if √n(θ̂ₙ − θ) converges in distribution to N(0, σ²(θ)) as n → ∞. This convergence is the engine that drives the entire interval construction.

Fisher Information

The Fisher information I(θ) = −E[∂²ℓ/∂θ²] measures the curvature of the log-likelihood. For MLEs under regularity conditions, the asymptotic variance is 1/I(θ), making the information directly determine interval width.

Slutsky's Theorem

If Xₙ →ᵈ X and Yₙ →ᵖ c (a constant), then XₙYₙ →ᵈ cX and Xₙ + Yₙ →ᵈ X + c. This theorem justifies replacing the unknown true variance with a consistent estimator without disrupting the limiting distribution.

Pivotal Quantity

A pivot is a function of data and the parameter whose distribution does not depend on unknown parameters. The quantity √n(θ̂ₙ − θ)/σ̂ₙ is an approximate pivot whose distribution is approximately N(0,1) for large n, enabling direct inversion into a confidence interval.

Coverage Probability

The coverage probability P(θ ∈ CI) equals the nominal level 1 − α only in the limit. For finite samples, the actual coverage may differ, and understanding this gap is essential for responsible statistical practice.

✦ KEY TAKEAWAY

Think of asymptotic normality as a universal adapter for statistical inference. Just as a universal power adapter lets you plug any device into any outlet worldwide, asymptotic normality lets you construct confidence intervals for virtually any well-behaved estimator—regardless of the original data distribution—by converting everything into the common currency of the standard normal. The 'plug' is the standardized statistic √n(θ̂ₙ − θ)/σ̂ₙ, and the 'outlet' is the N(0,1) quantile table.

SECTION 3

Visual Explanation — The Asymptotic Normal Approximation

The diagram below illustrates the central mechanism behind asymptotic confidence intervals. As the sample size n increases, the sampling distribution of the standardized estimator converges to the standard normal curve. The confidence interval is formed by inverting the probability statement that this standardized quantity falls between the critical values −z_α/2 and z_α/2.

The cyan bell curve represents the asymptotic N(0,1) density of the standardized estimator. The pink tails each capture α/2 probability. The amber arrow shows the resulting confidence interval obtained by inverting the probability statement P(−z_{α/2} ≤ √n(θ̂ₙ − θ)/σ̂ₙ ≤ z_{α/2}) ≈ 1 − α.

The visual makes explicit a key conceptual step that students often overlook: the confidence interval is not simply an assertion about where the parameter lies, but a direct algebraic inversion of a probability statement about the standardized pivot. Because the pivot is approximately N(0,1), the critical values come from the standard normal table, and solving for θ produces the familiar θ̂ₙ ± z_α/2 × σ̂ₙ/√n form. The quality of the interval depends on how well the finite-sample distribution of the pivot actually matches the Gaussian limit—a question we will revisit in later sections.

SECTION 4

Mathematical Framework

We now develop the formal machinery for constructing confidence intervals from asymptotic normality. The derivation proceeds in three stages: establishing the limiting distribution, forming an approximate pivot by estimating the asymptotic variance, and inverting the pivot to obtain an interval.

Stage 1: The Asymptotic Normality Result

ASYMPTOTIC DISTRIBUTION OF THE MLE

√n (θ̂ₙ − θ₀) →ᵈ N(0, 1/I(θ₀))

Here θ̂ₙ is the MLE, θ₀ is the true parameter value, n is the sample size, and I(θ₀) = E[−∂²ℓ(θ)/∂θ² | θ = θ₀] is the Fisher information per observation. This result holds under standard regularity conditions: the parameter space is open, the support of the density does not depend on θ, and the log-likelihood is thrice differentiable with bounded third derivative in a neighborhood of θ₀.

Stage 2: Forming an Approximate Pivot

The asymptotic variance 1/I(θ₀) depends on the unknown θ₀. By Slutsky's theorem, if we replace I(θ₀) with a consistent estimator—either the observed Fisher information Î(θ̂ₙ) = −∂²ℓ(θ)/∂θ² evaluated at θ̂ₙ, or the expected information I(θ̂ₙ)—the limiting distribution is unchanged.

APPROXIMATE PIVOT (WALD STATISTIC)

Zₙ = √(nÎ(θ̂ₙ)) × (θ̂ₙ − θ₀) →ᵈ N(0, 1)

Equivalently, Zₙ = (θ̂ₙ − θ₀) / ŝₑ(θ̂ₙ), where ŝₑ(θ̂ₙ) = 1/√(nÎ(θ̂ₙ)) is the estimated standard error of the MLE. This is the Wald statistic, the workhorse of asymptotic inference.

Stage 3: Inversion to a Confidence Interval

WALD CONFIDENCE INTERVAL

CI₁₋α = ( θ̂ₙ − z_{α/2} · ŝₑ(θ̂ₙ), θ̂ₙ + z_{α/2} · ŝₑ(θ̂ₙ) )

This is obtained by solving P(−z_α/2 ≤ Zₙ ≤ z_α/2) ≈ 1 − α for θ₀. Here z_α/2 = Φ⁻¹(1 − α/2) is the upper α/2 quantile of the standard normal. For a 95% CI, z_α/2 = 1.96.

📐 Multiparameter Extension

For a parameter vector θ ∈ ℝᵖ, the MLE satisfies √n(θ̂ₙ − θ₀) →ᵈ Nₚ(0, I(θ₀)⁻¹), where I(θ₀) is the p × p Fisher information matrix. A confidence region for a single component θⱼ uses ŝₑ(θ̂ⱼ) = √([I(θ̂ₙ)⁻¹]ⱼⱼ / n), producing the same Wald interval form. For joint confidence regions, one invokes the chi-squared limit: (θ̂ₙ − θ₀)ᵀ nI(θ̂ₙ)(θ̂ₙ − θ₀) →ᵈ χ²ₚ.

SECTION 5

Variance Estimation Strategies & The Delta Method

The width of the asymptotic confidence interval is governed entirely by the estimated standard error ŝₑ(θ̂ₙ). In practice, there are several strategies for obtaining this quantity, and the choice can substantially affect both the accuracy and the width of the resulting interval. Moreover, when the parameter of interest is a transformation g(θ) of the natural parameter, the delta method provides a principled way to propagate the asymptotic normality.

Three variance estimation strategies—expected information, observed information, and the sandwich estimator—all feed into the Wald confidence interval. The delta method extends the construction to smooth functions of the parameter.

The expected information approach requires computing E[−ℓ″(θ)] analytically and plugging in θ̂ₙ, which is straightforward for exponential family models but can be burdensome otherwise. The observed information Î(θ̂ₙ) = −ℓ″(θ̂ₙ) is computed directly from the data; Efron and Hinkley (1978) argued it often produces more accurate intervals because it conditions on the observed data rather than averaging over the sample space. The sandwich (robust) estimator V̂ = Â⁻¹B̂Â⁻¹ provides consistent variance estimates even when the assumed model is misspecified, at the cost of wider intervals. Here Â = n⁻¹∑ᵢ(−∂²ℓᵢ/∂θ²) and B̂ = n⁻¹∑ᵢ(∂ℓᵢ/∂θ)² are the empirical analogs of the expected Hessian and the outer-product-of-scores, respectively.

DELTA METHOD CI FOR g(θ)

CI₁₋α for g(θ) = g(θ̂ₙ) ± z_{α/2} × |g′(θ̂ₙ)| × ŝₑ(θ̂ₙ)

This follows from the chain rule applied to the asymptotic distribution: if √n(θ̂ₙ − θ₀) →ᵈ N(0, σ²), then √n(g(θ̂ₙ) − g(θ₀)) →ᵈ N(0, [g′(θ₀)]²σ²) by the delta method, provided g′(θ₀) ≠ 0. Common applications include CIs for odds ratios (g(θ) = eᶿ) and relative risks.

SECTION 6

Worked Example — Exponential Rate Parameter

Suppose we observe n = 100 independent realizations from an Exponential(λ) distribution, where λ > 0 is the rate parameter and the density is f(x; λ) = λe⁻ˡˣ for x > 0. The sample mean is x̄ = 4.2. We wish to construct a 95% asymptotic confidence interval for λ and also for the mean lifetime μ = 1/λ.

95% Asymptotic CI for Exponential Rate λ and Mean μ = 1/λ

Step 1 — Identify the MLEThe log-likelihood is ℓ(λ) = n ln(λ) − λ∑xᵢ = n ln(λ) − nλx̄. Setting ℓ′(λ) = n/λ − nx̄ = 0 yields λ̂ = 1/x̄.

λ̂ = 1/4.2 ≈ 0.2381

Step 2 — Compute the Fisher InformationThe second derivative is ℓ″(λ) = −n/λ². The Fisher information per observation is I(λ) = −E[ℓ″(λ)]/n = 1/λ². Evaluated at the MLE: I(λ̂) = 1/λ̂² = x̄² = (4.2)² = 17.64.

I(λ̂) = 17.64 per observation

Step 3 — Compute the Estimated Standard ErrorThe asymptotic standard error of λ̂ is ŝₑ(λ̂) = 1/√(nI(λ̂)) = 1/√(100 × 17.64) = 1/√1764 = 1/42 ≈ 0.02381. Note that for the Exponential family, ŝₑ(λ̂) = λ̂/√n = 0.2381/10 = 0.02381, confirming the calculation.

ŝₑ(λ̂) ≈ 0.02381

Step 4 — Construct the 95% CI for λUsing z₀.₀₂₅ = 1.96: CI = λ̂ ± 1.96 × ŝₑ(λ̂) = 0.2381 ± 1.96 × 0.02381 = 0.2381 ± 0.04667.

95% CI for λ: (0.1914, 0.2847)

Step 5 — Apply the Delta Method for μ = 1/λLet g(λ) = 1/λ, so g′(λ) = −1/λ². The estimated standard error for μ̂ = g(λ̂) is |g′(λ̂)| × ŝₑ(λ̂) = (1/λ̂²) × (λ̂/√n) = 1/(λ̂√n) = x̄/√n = 4.2/10 = 0.42. The 95% CI for μ = 1/λ is 4.2 ± 1.96 × 0.42 = 4.2 ± 0.8232.

95% CI for μ = 1/λ: (3.377, 5.023)

⚠️ Sanity Check

Notice that the CI for μ = 1/λ is not simply the reciprocal of the endpoints of the CI for λ. Inverting (0.1914, 0.2847) gives (3.513, 5.225), which differs from the delta-method interval (3.377, 5.023). Both are valid asymptotic intervals, but they approximate different things: the delta method interval is symmetric around μ̂, while the inverted interval respects the curvature of the transformation. For finite samples, the inverted interval often has better coverage because it accounts for skewness—this motivates variance-stabilizing transformations.

SECTION 7

Strengths, Limitations & Comparisons

Asymptotic normality-based confidence intervals are the most widely deployed interval estimation method in statistics, powering the standard output of virtually every regression package. However, their convenience comes with important caveats that every practitioner should understand. The table below summarizes the key trade-offs.

Strengths and limitations of asymptotic normality-based confidence intervals

Aspect	Strength	Limitation
Generality	Applies to any asymptotically normal estimator—MLEs, method of moments, M-estimators, etc.—without requiring the exact sampling distribution.	Requires regularity conditions (smooth likelihood, open parameter space, identifiability) that may not hold for all models.
Simplicity	The θ̂ ± z × ŝₑ formula is easy to compute and communicate. It requires only a point estimate and a standard error.	Symmetric by construction; performs poorly for skewed parameters (e.g., variance components, odds ratios on their natural scale).
Finite-sample coverage	Coverage converges to the nominal level as n → ∞ at rate O(n⁻¹/²), and can be improved by Bartlett correction or bootstrap calibration.	For moderate n, actual coverage may deviate substantially from the nominal level, especially near boundary parameters or with high-dimensional nuisance parameters.
Variance estimation	Multiple consistent estimators available (expected, observed, sandwich), providing flexibility for model diagnostics.	Standard errors can be badly estimated in small samples; the sandwich estimator, while robust, can be unstable when n is small relative to the number of parameters.
Boundary parameters	Works well when the true parameter is in the interior of the parameter space.	Fails when the true parameter is on or near the boundary (e.g., testing σ² = 0), where the asymptotic distribution is not normal but a mixture involving point masses.

✦ KEY TAKEAWAY

Asymptotic CIs are like GPS navigation: remarkably accurate and convenient in typical conditions (well-paved roads, clear signal), but potentially misleading in edge cases (tunnels, canyons, unmarked roads). Just as a savvy driver knows when to distrust the GPS—perhaps consulting a paper map or asking a local—a careful statistician should always assess whether the asymptotic approximation is trustworthy for the problem at hand, using diagnostics like sample size relative to the number of parameters, skewness of the estimator, and proximity to boundary values.

SECTION 8

Connection to Advanced Theory — Alternatives to Wald Intervals

The Wald interval derived from asymptotic normality is not the only way to exploit large-sample theory for interval estimation. Two important alternatives—the score (Rao) interval and the likelihood ratio interval—invert different test statistics that are also asymptotically χ² under the null. These alternatives often exhibit superior finite-sample performance, particularly for discrete data and parameters near boundaries.

Comparison of three asymptotic confidence interval methods

Property	Wald Interval	Score (Rao) Interval	Likelihood Ratio Interval
Pivotal statistic	(θ̂ − θ₀)² / Var̂(θ̂)	U(θ₀)² / I(θ₀), where U is the score function	2[ℓ(θ̂) − ℓ(θ₀)]
Evaluation point	All quantities at θ̂ (the MLE)	Score and information at θ₀ (the null)	Log-likelihood at both θ̂ and θ₀
Symmetry	Always symmetric about θ̂	Can be asymmetric	Typically asymmetric; respects likelihood curvature
Requires MLE?	Yes	Not necessarily; can be computed from score alone	Yes
Finite-sample coverage	Often liberal (under-covers)	Generally better than Wald, especially for proportions	Often best among the three; transformation-invariant
Computational cost	Lowest: closed-form	Moderate: requires solving an equation in θ₀	Highest: requires optimization for each boundary point

As a concrete example, consider the Wald interval for a binomial proportion p with x successes in n trials. The Wald interval p̂ ± z_α/2√(p̂(1−p̂)/n) famously has erratic coverage, especially when p is near 0 or 1. The score interval—equivalent to the Wilson interval—adds a correction that 'shrinks' the estimate toward 1/2 and produces markedly better coverage. This is a vivid reminder that even though all three methods share the same asymptotic justification, their finite-sample behavior can differ dramatically. In graduate-level work, the likelihood ratio interval is frequently preferred for its parametrization invariance: the CI for g(θ) obtained by inverting the LR test is exactly {g(θ) : θ ∈ CI_LR}, which is not true for Wald intervals in general.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why a Wald confidence interval for a parameter θ always produces a symmetric interval about the MLE θ̂ₙ, and describe a situation in which this symmetry is undesirable. What alternative asymptotic method addresses this limitation?

PROBLEM 2 — BASIC CALCULATION

Suppose X₁, …, X₆₄ are i.i.d. Poisson(λ). The observed sample mean is x̄ = 3.5. Construct an approximate 90% confidence interval for λ using asymptotic normality of the MLE.

PROBLEM 3 — INTERMEDIATE

Let X₁, …, Xₙ be i.i.d. Bernoulli(p). (a) Derive the Fisher information I(p) per observation. (b) Write the Wald 95% CI for p. (c) Apply the delta method to construct a 95% CI for the log-odds η = ln(p/(1−p)). (d) With n = 200 and p̂ = 0.12, compute both intervals numerically.

PROBLEM 4 — APPLIED

A reliability engineer models component lifetimes as Weibull with shape k = 2 (known) and unknown scale parameter λ. The density is f(x; λ) = (k/λ)(x/λ)^{k−1} exp(−(x/λ)^k). After testing n = 50 components, the MLE is λ̂ = 1200 hours and the observed Fisher information per observation is estimated as Î(λ̂) = 3.47 × 10⁻⁶. Construct a 95% CI for λ and for the mean lifetime E[X] = λΓ(1 + 1/k).

PROBLEM 5 — CRITICAL THINKING

Consider a family of distributions parameterized by θ ∈ (0, ∞) with Fisher information I(θ) = θ⁻² per observation. (a) Write the Wald 95% CI for θ based on n observations. (b) Find a variance-stabilizing transformation g(θ) such that the asymptotic variance of g(θ̂ₙ) does not depend on θ. (c) Construct a 95% CI for g(θ) and back-transform to obtain a CI for θ. (d) Compare the two intervals for θ̂ = 0.5, n = 25, and explain which you would recommend and why.

SUMMARY

Summary — Confidence Intervals from Asymptotic Normality

Confidence intervals from asymptotic normality exploit the fact that many estimators—most notably maximum likelihood estimators—satisfy √n(θ̂ₙ − θ₀) →ᵈ N(0, 1/I(θ₀)) under regularity conditions. By replacing the unknown Fisher information with a consistent estimate (via observed information, expected information, or a sandwich estimator) and applying Slutsky's theorem, we form the approximate pivot Zₙ = (θ̂ₙ − θ₀)/ŝₑ(θ̂ₙ) ∼̇ N(0,1), which inverts to the Wald confidence interval θ̂ₙ ± z_{α/2} × ŝₑ(θ̂ₙ).

The delta method extends this construction to smooth functions g(θ), and variance-stabilizing transformations can improve finite-sample coverage probability by making the normal approximation more accurate. While Wald intervals are simple and ubiquitous, alternatives such as the score interval and the likelihood ratio interval often provide superior coverage in finite samples and should be preferred when computational cost permits—particularly for parameters near boundaries or for discrete data.