Consistency & Efficiency — Consistency and efficiency...

SECTION 1

Historical Context & Motivation

The foundations of modern estimation theory rest on two deceptively simple questions: does an estimator eventually give us the right answer as we gather more data, and among all estimators that do, which one wastes the least information? These questions about consistency and efficiency have shaped statistical inference since the early twentieth century, forming the bedrock criteria by which we judge the quality of point estimators. The pursuit of these properties drove some of the most influential minds in probability and statistics to develop frameworks that remain central to graduate-level study today.

1920s

Fisher's Theory of Estimation

R.A. Fisher introduced the concepts of consistency, efficiency, and sufficiency as formal desiderata for estimators, laying the groundwork for parametric inference in his landmark 1922 paper.

1945

Cramér–Rao Inequality

Harald Cramér and C.R. Rao independently established a lower bound on the variance of unbiased estimators, providing a precise benchmark against which efficiency could be measured. This result formalized the notion that there is a fundamental limit to how precisely we can estimate a parameter.

1953

Le Cam's Asymptotic Framework

Lucien Le Cam developed a rigorous asymptotic theory that unified consistency, efficiency, and superefficiency within a single mathematical framework, showing that no regular estimator can beat the maximum likelihood estimator asymptotically.

1970s–80s

Robust Estimation and Tradeoffs

Peter Huber and others explored scenarios where classical efficiency is sacrificed for robustness, revealing deep tradeoffs between these estimator properties and expanding the conceptual landscape beyond purely parametric settings.

The central tension that motivated this line of inquiry can be stated concisely: given a parametric model and a growing sample, how should we construct an estimator that both converges to the true parameter value and squeezes as much information as possible from the data? Consistency addresses the first concern—convergence—while efficiency addresses the second—optimality of variance. Together, they define the gold standard for point estimation in statistical inference.

SECTION 2

Core Principles & Definitions

Before diving into formal definitions, it is essential to understand that consistency and efficiency are fundamentally asymptotic properties—they describe the behavior of estimators as the sample size n grows without bound. An estimator might behave poorly for small samples yet be perfectly consistent; conversely, an estimator could be unbiased in finite samples but fail to be efficient. The interplay between finite-sample and asymptotic behavior is one of the subtleties that makes these concepts rich and nuanced.

Consistency

An estimator θ̂ₙ is consistent for θ if it converges in probability to θ as n → ∞. Formally, for every ε > 0, P(|θ̂ₙ − θ| > ε) → 0. This guarantees that with enough data, the estimator gets arbitrarily close to the truth.

Efficiency (Asymptotic)

An estimator is asymptotically efficient if its asymptotic variance equals the Cramér–Rao lower bound. It extracts the maximum possible information from the data, leaving no room for improvement among regular estimators.

Cramér–Rao Lower Bound

For any unbiased estimator, the variance satisfies Var(θ̂) ≥ 1/I(θ), where I(θ) is the Fisher information. This bound serves as the yardstick against which we measure efficiency.

Fisher Information

The quantity I(θ) = E[(∂/∂θ log f(X; θ))²] measures how much information a single observation carries about the parameter θ. Larger Fisher information implies that smaller variances are achievable.

Relative Efficiency

When comparing two estimators, the asymptotic relative efficiency (ARE) is the ratio of their asymptotic variances. An ARE of 0.64 means the less efficient estimator needs about 56% more data to match the precision of the better one.

✦ KEY TAKEAWAY

Think of consistency as your GPS eventually guiding you to the correct destination regardless of the route—given enough time (data), you arrive at the truth. Efficiency is about fuel economy: among all GPS routes that reach the destination, the efficient one wastes the least fuel (variance). A consistent but inefficient estimator is like a GPS that always gets you there but takes scenic detours—you arrive eventually, but a more efficient estimator arrives with the same tank of gas in less time.

SECTION 3

Visual Explanation — Convergence of Estimators

The diagram shows the sampling distribution of a consistent estimator θ̂ₙ for four increasing sample sizes. As n grows from 10 to 1000, the density collapses around the true parameter value θ₀, illustrating convergence in probability. The spread (variance) decreases monotonically, and the probability mass outside any fixed ε-neighborhood of θ₀ vanishes.

The visual above captures the essence of consistency: as data accumulate, the sampling distribution of θ̂ₙ becomes increasingly concentrated around the true parameter θ₀. Notice that consistency does not require the estimator to be unbiased at any finite sample size—it only requires that the distribution degenerates to a point mass at θ₀ in the limit. The rate at which this concentration occurs is intimately linked to the estimator's variance, and this is precisely where efficiency enters the picture. Two estimators may both be consistent—both converging to θ₀—yet one may concentrate much more rapidly than the other, requiring fewer observations to achieve a given level of precision.

SECTION 4

Mathematical Framework

Formal Definition of Consistency

An estimator θ̂ₙ based on a sample of size n is said to be weakly consistent (or simply consistent) for a parameter θ if it converges in probability to θ. This is the most commonly used notion of consistency in parametric inference.

CONSISTENCY (CONVERGENCE IN PROBABILITY)

θ̂ₙ →ᵖ θ ⟺ ∀ε > 0, lim_{n→∞} P(|θ̂ₙ − θ| > ε) = 0

θ̂ₙ is the estimator based on n observations; θ is the true parameter value; ε is an arbitrary positive tolerance; →ᵖ denotes convergence in probability.

A stronger form, strong consistency, requires almost sure convergence: P(lim_{n→∞} θ̂ₙ = θ) = 1. Strong consistency implies weak consistency but not vice versa. In most practical settings, weak consistency suffices. A convenient sufficient condition for consistency is that the bias and variance both vanish asymptotically.

SUFFICIENT CONDITIONS FOR CONSISTENCY

lim_{n→∞} Bias(θ̂ₙ) = 0 and lim_{n→∞} Var(θ̂ₙ) = 0 ⟹ θ̂ₙ →ᵖ θ

This follows directly from Chebyshev's inequality: P(|θ̂ₙ − θ| > ε) ≤ MSE(θ̂ₙ)/ε² = [Var(θ̂ₙ) + Bias²(θ̂ₙ)]/ε² → 0.

The Cramér–Rao Lower Bound and Efficiency

The concept of efficiency is anchored to the Cramér–Rao lower bound (CRLB), which establishes the smallest variance that any unbiased estimator can achieve. For a single-parameter model with i.i.d. observations, the Fisher information for n observations is n × I(θ), where I(θ) is the information from a single observation.

CRAMÉR–RAO LOWER BOUND

Var(θ̂ₙ) ≥ 1 / [n × I(θ)]

I(θ) = E[(∂/∂θ log f(X; θ))²] = −E[∂²/∂θ² log f(X; θ)] is the Fisher information per observation, assuming regularity conditions (interchange of differentiation and integration).

ASYMPTOTIC EFFICIENCY

e(θ̂ₙ) = [1 / (n × I(θ))] / Var_asy(θ̂ₙ)

An estimator is asymptotically efficient when e(θ̂ₙ) = 1, i.e., its asymptotic variance equals the CRLB. The MLE achieves this under standard regularity conditions.

⚠️ Regularity Conditions

The Cramér–Rao bound and asymptotic efficiency of the MLE rely on regularity conditions: the parameter space is open, the support of f(x; θ) does not depend on θ, we can interchange differentiation and integration, and I(θ) is finite and positive. When these fail (e.g., uniform distribution on [0, θ]), the MLE may converge faster than √n, and the CRLB does not apply.

SECTION 5

Comparing Estimator Efficiency

To make the concept of efficiency concrete, consider estimating the mean μ of a normal distribution N(μ, σ²). The sample mean X̄ₙ and the sample median are both consistent estimators of μ, but they differ dramatically in their asymptotic variances. The asymptotic relative efficiency (ARE) provides a direct numerical comparison, telling us how many additional observations the inferior estimator requires to match the performance of the superior one.

The horizontal cyan line at σ² represents the asymptotic variance of the sample mean (multiplied by n for visualization), which equals the Cramér–Rao lower bound. The dashed pink line at πσ²/2 ≈ 1.571σ² represents the asymptotic variance of the sample median × n. The amber bars highlight the efficiency gap: the sample median wastes approximately 36.3% of the information available in normal data.

Efficiency comparison of location estimators under the normal model

Estimator	Asymptotic Variance	ARE relative to MLE	Extra obs. for same precision
Sample Mean (X̄ₙ)	σ² / n	1.000	0%
Sample Median	πσ² / (2n)	0.637	≈ 57%
Mid-Range	Not O(1/n) for normal	0 (inconsistent for normal)	N/A
10%-Trimmed Mean	≈ 1.06σ² / n	≈ 0.943	≈ 6%

The table reveals a crucial insight: under the normal model, the sample mean is the most efficient estimator of the population mean, achieving the Cramér–Rao bound exactly. The sample median, while consistent, pays a substantial efficiency cost—it would require roughly 57% more data to match the precision of the sample mean. However, this ranking is model-dependent: under heavy-tailed distributions like the Cauchy, the sample mean is not even consistent, while the sample median remains both consistent and relatively efficient.

SECTION 6

Worked Example — Verifying Consistency and Computing Efficiency

Let X₁, X₂, …, Xₙ be i.i.d. from an Exponential(λ) distribution with density f(x; λ) = λe^(−λx) for x > 0. We wish to verify that the MLE λ̂ₙ = 1/X̄ₙ is consistent for λ and to compute its asymptotic efficiency.

Consistency and Efficiency of the Exponential MLE

Step 1 — Find the MLEThe log-likelihood is ℓ(λ) = n log λ − λ ∑Xᵢ. Setting ∂ℓ/∂λ = n/λ − ∑Xᵢ = 0 gives λ̂ₙ = n / ∑Xᵢ = 1/X̄ₙ.

λ̂ₙ = 1/X̄ₙ

Step 2 — Verify Consistency via the Continuous Mapping TheoremBy the weak law of large numbers, X̄ₙ →ᵖ E[X₁] = 1/λ. Since the function g(x) = 1/x is continuous on (0, ∞), the continuous mapping theorem gives λ̂ₙ = g(X̄ₙ) →ᵖ g(1/λ) = λ.

λ̂ₙ is consistent for λ ✓

Step 3 — Compute the Fisher InformationThe score function is ∂/∂λ log f(X; λ) = 1/λ − X. The Fisher information per observation is I(λ) = E[(1/λ − X)²] = Var(X) = 1/λ² (since E[X] = 1/λ, the score has mean zero, and E[(1/λ − X)²] = Var(X) = 1/λ²). Alternatively, −∂²/∂λ² log f = 1/λ², confirming I(λ) = 1/λ².

I(λ) = 1/λ²

Step 4 — Derive the Asymptotic Variance of the MLEBy the delta method applied to λ̂ₙ = g(X̄ₙ) with g(x) = 1/x, we have √n(λ̂ₙ − λ) →ᵈ N(0, [g'(1/λ)]² × Var(X₁)). Here g'(x) = −1/x², so g'(1/λ) = −λ². Thus the asymptotic variance is λ⁴ × (1/λ²) / n = λ²/n.

Var_asy(λ̂ₙ) = λ²/n

Step 5 — Verify EfficiencyThe Cramér–Rao lower bound for n observations is 1/[n × I(λ)] = 1/(n/λ²) = λ²/n. Comparing: Var_asy(λ̂ₙ) = λ²/n = CRLB. Therefore, the asymptotic efficiency is e(λ̂ₙ) = (λ²/n)/(λ²/n) = 1.

e(λ̂ₙ) = 1 — the MLE is asymptotically efficient ✓

💡 Why this matters

The exponential example illustrates a general phenomenon: under regularity conditions, the MLE is always consistent and asymptotically efficient. This is a consequence of the deep connection between the MLE, the score function, and Fisher information, formalized in the classical asymptotic theory of maximum likelihood.

SECTION 7

Strengths, Limitations, and Tradeoffs

While consistency and efficiency are powerful criteria, they are not without limitations. The conceptual separation between finite-sample properties and asymptotic guarantees means that a consistent, efficient estimator may still perform poorly for the sample sizes encountered in practice. Moreover, the pursuit of efficiency under one model can lead to fragile estimators that break down under model misspecification. The table below summarizes the key strengths and limitations of using consistency and efficiency as guiding principles in estimation.

Strengths and limitations of key estimation properties

Property	Strengths	Limitations
Consistency	Guarantees convergence to truth; filters out fundamentally flawed estimators; applies broadly to parametric and nonparametric settings	Says nothing about finite-sample accuracy; rate of convergence may be slow; many trivially different estimators are all consistent
Efficiency	Provides a meaningful optimality criterion; directly linked to sample size requirements; enables comparison across estimators via ARE	Requires regularity conditions; sensitive to model misspecification; the CRLB may not be tight in non-regular families
MLE (achieves both)	Consistent and asymptotically efficient under regularity; invariant under reparametrization; broadly applicable	May be biased in finite samples; can be computationally expensive; not robust to outliers or model misspecification

⚖️ EFFICIENCY vs. ROBUSTNESS

There is an unavoidable tension between efficiency and robustness. A Formula 1 car is maximally efficient on a smooth track but breaks apart on gravel; a rally car sacrifices some track speed for the ability to handle any terrain. Similarly, the MLE is the Formula 1 car of estimation—optimal under the assumed model but potentially disastrous when the model is wrong. Robust alternatives like M-estimators or trimmed means are the rally cars: they sacrifice a small amount of efficiency under the ideal model in exchange for stability across a range of data-generating processes.

SECTION 8

Connection to Advanced Theory

The concepts of consistency and efficiency sit at the foundation of a much larger theoretical edifice. At the graduate level, these ideas extend naturally into several advanced domains, including Le Cam's theory of local asymptotic normality, higher-order efficiency, and semiparametric efficiency bounds.

From basic to advanced: how consistency and efficiency extend

Concept Covered Here	Advanced Extension	Key Idea
Consistency (convergence in probability)	Uniform consistency	Convergence holds uniformly over compact subsets of the parameter space, important for M-estimation
Cramér–Rao bound (fixed θ)	Local Asymptotic Minimax (LAM) bound	Le Cam showed that no regular estimator can beat the MLE's asymptotic risk under local alternatives
Asymptotic efficiency (first-order)	Higher-order efficiency	Second-order terms (Bartlett corrections, bias corrections) distinguish estimators with the same first-order behavior
Parametric efficiency	Semiparametric efficiency (Hájek–Le Cam)	Efficiency bounds in models where nuisance parameters are infinite-dimensional, central to modern causal inference

One particularly important advanced result is the Hájek–Le Cam convolution theorem, which states that for any regular estimator θ̂ₙ, its limiting distribution can be decomposed as the convolution of the optimal normal distribution N(0, I(θ)⁻¹) with some additional noise distribution. An estimator is efficient if and only if this additional noise component is degenerate (i.e., zero). This provides a deep structural understanding of why the MLE is special: it achieves the tightest possible asymptotic distribution, with no wasted variability.

🔬 Superefficiency Paradox

Can an estimator do better than the CRLB? Surprisingly, yes—at a single point. Hodges' estimator achieves a variance of zero at θ = 0 while remaining √n-consistent elsewhere. However, Le Cam showed that the set of parameter values at which a superefficient estimator beats the CRLB must have Lebesgue measure zero. This result, combined with the convolution theorem, firmly establishes that the MLE's efficiency is essentially unimprovable in a meaningful sense.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

An estimator θ̂ₙ is biased for every finite n, with Bias(θ̂ₙ) = 3/n. Is this estimator necessarily consistent? Explain your reasoning, including what additional condition would guarantee consistency.

PROBLEM 2 — BASIC CALCULATION

Let X₁, …, Xₙ be i.i.d. N(μ, σ²) with σ² known. Compute the Fisher information I(μ) per observation and verify that X̄ₙ attains the Cramér–Rao lower bound.

PROBLEM 3 — INTERMEDIATE

Consider estimating p in a Bernoulli(p) model. The sample proportion p̂ₙ = (1/n)∑Xᵢ is the MLE. Compute its asymptotic relative efficiency compared to the estimator T_n = (∑Xᵢ + 1)/(n + 2), which is a Bayesian estimator under a Beta(1,1) prior. Interpret the result.

PROBLEM 4 — APPLIED

A pharmaceutical company is estimating the mean clearance rate λ of a drug, modeled as Exponential(λ). They have n = 50 patients and compute both the MLE λ̂ = 1/X̄ and an alternative estimator λ* = (n−1)/(n·X̄). Determine which estimator is unbiased, verify both are consistent, and compute the ARE of λ* relative to λ̂. Which should the company report, and why?

PROBLEM 5 — CRITICAL THINKING

Prove that if θ̂ₙ is an asymptotically efficient estimator of θ under a regular parametric model, and g(·) is a differentiable function with g'(θ) ≠ 0, then g(θ̂ₙ) is an asymptotically efficient estimator of g(θ). Discuss why this result is sometimes called the 'invariance of efficiency under reparametrization' and identify where the regularity assumptions are essential.

SUMMARY

Summary & Review

Consistency ensures that an estimator θ̂ₙ converges in probability to the true parameter θ as the sample size grows, providing the minimal guarantee that our inference improves with more data. A sufficient condition is that both the bias and variance vanish asymptotically. The Cramér–Rao lower bound establishes a floor on the variance of unbiased estimators, depending on the Fisher information I(θ). An estimator that achieves this bound asymptotically is called asymptotically efficient, and the maximum likelihood estimator is the canonical example, achieving efficiency under standard regularity conditions.

The asymptotic relative efficiency (ARE) allows direct comparison of two consistent estimators by taking the ratio of their asymptotic variances, translating theoretical differences into practical sample-size requirements. While consistency and efficiency are the gold standard for parametric point estimation, they must be balanced against robustness and finite-sample performance. Advanced extensions include Le Cam's local asymptotic normality, the convolution theorem, and semiparametric efficiency bounds that generalize these ideas to infinite-dimensional nuisance parameter settings.