Asymptotic Normality of MLE

SECTION 1

Historical Context & Motivation

The quest for a unified theory of estimation drove much of twentieth-century statistics. At the heart of this effort lies the maximum likelihood estimator (MLE), a method that finds the parameter value making observed data most probable under a given model. While the MLE was known to produce sensible point estimates, statisticians needed to understand its distributional behavior in large samples to build confidence intervals, perform hypothesis tests, and compare competing estimators on rigorous theoretical grounds.

The key breakthrough was the realization that, under mild regularity conditions, the MLE converges in distribution to a normal random variable centered at the true parameter and with variance equal to the inverse of the Fisher information. This result — the asymptotic normality of the MLE — provides a universal recipe for inference whenever the sample size is large enough, regardless of the original distribution generating the data.

1912

Fisher's Early Work

Ronald A. Fisher, still an undergraduate at Cambridge, introduces the concept of maximum likelihood in a paper on the method of fitting frequency curves. He argues for likelihood as a principled measure of support for parameter values.

1922

Fisher's Foundational Paper

Fisher publishes 'On the Mathematical Foundations of Theoretical Statistics,' formally defining the MLE and asserting its asymptotic efficiency and normality. He introduces Fisher information as the key quantity governing estimator precision.

1948

Cramér's Rigorous Proof

Harald Cramér provides one of the first fully rigorous proofs of MLE asymptotic normality in his landmark textbook, formalizing the regularity conditions required and establishing the Cramér–Rao lower bound.

1956

Le Cam's Asymptotic Framework

Lucien Le Cam develops the theory of local asymptotic normality (LAN), providing an elegant abstract framework that subsumes MLE asymptotic normality as a special case and extends to semiparametric and nonparametric settings.

1970s–

Modern Extensions

Researchers extend asymptotic normality results to multiparameter models, misspecified models (Huber–White sandwich estimator), dependent data, and high-dimensional settings, making MLE theory a cornerstone of modern statistical methodology.

The central question that asymptotic normality answers is deceptively simple: if I compute the MLE from a large sample, what can I say about how close it is to the true parameter, and how should I quantify my uncertainty? The answer turns out to be remarkably elegant — the MLE behaves like a draw from a normal distribution whose variance shrinks at rate 1/n, with the exact variance determined by the curvature of the log-likelihood function.

SECTION 2

Core Principles & Definitions

Before stating the asymptotic normality theorem precisely, we must establish the key ingredients that make the result work. The theorem rests on a set of regularity conditions that ensure the log-likelihood function is sufficiently smooth and that the model is well-behaved. Understanding these conditions is essential because they delineate the boundary between settings where the MLE is trustworthy and those where it may fail catastrophically.

Consistency of the MLE

The MLE θ̂ₙ converges in probability to the true parameter θ₀ as n → ∞. This is a prerequisite: asymptotic normality describes the fluctuations of a consistent estimator around its target.

Fisher Information

The quantity I(θ) = E[−∂²ℓ(θ)/∂θ²] measures the curvature of the log-likelihood. Higher curvature means the data carry more information about θ, yielding a smaller asymptotic variance for the MLE.

Regularity Conditions

The parameter space Θ must be open, the support of the density must not depend on θ, the log-likelihood must be thrice differentiable, and Fisher information must be finite and positive. These ensure a valid Taylor expansion of the score function.

Score Function

The score S(θ) = ∂ℓ(θ)/∂θ has mean zero and variance I(θ) under the true model. The MLE solves S(θ̂ₙ) = 0, so the theorem hinges on a Taylor expansion of S around the true parameter.

Cramér–Rao Lower Bound

No unbiased estimator can have variance below 1/(nI(θ₀)). The MLE achieves this bound asymptotically, making it asymptotically efficient — the best possible estimator in the large-sample limit.

✦ KEY TAKEAWAY

Think of the log-likelihood surface as a landscape. The MLE sits at the peak of this landscape. Fisher information measures how sharply the peak rises — a narrow, pointed peak (high information) means the MLE is tightly concentrated around the true value, just as a precisely machined lens focuses light to a sharp point. A broad, flat peak (low information) means the data barely distinguish nearby parameter values, yielding a more diffuse estimator.

SECTION 3

Visual Explanation

The following diagram illustrates the core idea of asymptotic normality. As the sample size increases, the sampling distribution of the MLE — initially shaped by the idiosyncrasies of the underlying model — converges to a Gaussian bell curve centered at the true parameter θ₀. The spread of this bell curve is governed by 1/(nI(θ₀)), so the distribution becomes increasingly concentrated around the truth.

As sample size n increases from 10 (red) to 50 (amber) to 200 (cyan), the sampling distribution of the MLE θ̂ₙ becomes progressively more concentrated and Gaussian around the true value θ₀. The vertical dashed line marks θ₀.

Notice that the red curve (n = 10) is relatively wide and may exhibit slight asymmetry depending on the underlying model. By n = 50 (amber), the distribution has tightened considerably, and by n = 200 (cyan), it is nearly indistinguishable from a true Gaussian. This visual convergence is precisely what the asymptotic normality theorem guarantees: √n(θ̂ₙ − θ₀) converges in distribution to N(0, I(θ₀)⁻¹), so the width of the distribution shrinks proportionally to 1/√n.

SECTION 4

Mathematical Framework

We now develop the formal statement and proof sketch of the asymptotic normality result. Let X₁, X₂, …, Xₙ be i.i.d. random variables with density f(x; θ₀), where θ₀ ∈ Θ ⊂ ℝ is the true parameter value and Θ is an open set. Define the log-likelihood as ℓₙ(θ) = Σᵢ₌₁ⁿ log f(Xᵢ; θ) and the score function for a single observation as s(X; θ) = ∂ log f(X; θ)/∂θ.

FISHER INFORMATION

I(θ) = E_θ[ s(X; θ)² ] = −E_θ[ ∂²log f(X; θ) / ∂θ² ]

I(θ) is the Fisher information for a single observation. The equality of the two expressions holds under the regularity condition that permits differentiation under the integral sign. The total information from n observations is nI(θ).

ASYMPTOTIC NORMALITY THEOREM

√n (θ̂ₙ − θ₀) →ᵈ N(0, I(θ₀)⁻¹) as n → ∞

Here θ̂ₙ is the MLE, θ₀ is the true parameter, →ᵈ denotes convergence in distribution, and I(θ₀)⁻¹ is the inverse Fisher information. Equivalently, for large n, θ̂ₙ ≈ N(θ₀, 1/(nI(θ₀))).

Proof Sketch via Taylor Expansion

The proof proceeds by a first-order Taylor expansion of the score equation around the true parameter. Since the MLE satisfies the score equation ℓₙ′(θ̂ₙ) = 0, we expand around θ₀ to obtain:

TAYLOR EXPANSION OF SCORE

0 = ℓₙ′(θ̂ₙ) = ℓₙ′(θ₀) + ℓₙ″(θ*)(θ̂ₙ − θ₀)

Here θ* lies between θ̂ₙ and θ₀ by the mean value theorem. Rearranging: √n(θ̂ₙ − θ₀) = −ℓₙ′(θ₀)/√n ÷ ℓₙ″(θ*)/n. The numerator converges to N(0, I(θ₀)) by the CLT, and the denominator converges in probability to −I(θ₀) by the WLLN. Slutsky's theorem delivers the result.

The key steps deserve emphasis. First, the central limit theorem (CLT) applies to n⁻¹/²ℓₙ′(θ₀) = n⁻¹/² Σᵢ s(Xᵢ; θ₀) because the individual scores are i.i.d. with mean zero and variance I(θ₀). Second, the weak law of large numbers (WLLN) ensures that n⁻¹ℓₙ″(θ*) converges in probability to E[∂²log f(X; θ₀)/∂θ²] = −I(θ₀), provided θ* → θ₀ in probability (which follows from consistency). Finally, Slutsky's theorem combines these two convergence results to yield the ratio's limit distribution.

APPROXIMATE CONFIDENCE INTERVAL

θ̂ₙ ± z_{α/2} / √(n · I(θ̂ₙ))

A (1 − α)×100% confidence interval for θ₀ using the asymptotic normal approximation. Here z_{α/2} is the standard normal critical value. Fisher information I(θ₀) is replaced by its estimate I(θ̂ₙ), which is valid by the continuous mapping theorem.

SECTION 5

Regularity Conditions in Detail

The asymptotic normality theorem is not a universal truth — it fails when its regularity conditions are violated. Understanding these conditions both strengthens your ability to apply the theorem and alerts you to the settings where alternative methods are required. Below, we present the standard conditions in a structured format, followed by a diagram showing how violations manifest.

Standard regularity conditions for asymptotic normality of the MLE

Condition	Formal Statement	Why It Matters
R1: Identifiability	θ ≠ θ′ ⟹ f(·; θ) ≠ f(·; θ′) a.e.	Ensures the true parameter is uniquely determined. Without this, the MLE may not converge to θ₀.
R2: Common support	The set {x : f(x; θ) > 0} does not depend on θ.	Permits differentiation under the integral sign. Violated by Uniform(0, θ), where the support boundary moves with θ.
R3: Interior parameter	θ₀ is an interior point of the open parameter space Θ.	The Taylor expansion requires a neighborhood around θ₀. Boundary parameters invalidate the expansion.
R4: Smoothness	log f(x; θ) is thrice continuously differentiable in θ for almost all x.	Enables the Taylor expansion of the score function and control of the remainder term.
R5: Finite, positive information	0 < I(θ₀) < ∞	Zero information makes the limiting variance infinite; infinite information is pathological and indicates a non-regular model.
R6: Domination	∃ H(x) with E[H(X)] < ∞ such that \|∂³ log f(x; θ)/∂θ³\| ≤ H(x) for all θ near θ₀.	Controls the remainder in the Taylor expansion, ensuring it vanishes asymptotically.

Left: a regular model (e.g., Normal) has a smooth, quadratic log-likelihood peak, enabling the Taylor expansion that yields asymptotic normality. Right: a non-regular model (e.g., Uniform(0, θ)) has a cusp at the MLE — the support boundary depends on θ, violating condition R2. The MLE converges at rate n (not √n) and has an exponential, not Gaussian, limiting distribution.

⚠️ Classic Counterexample

For X₁, …, Xₙ ~ Uniform(0, θ), the MLE is θ̂ₙ = X₍ₙ₎ (the sample maximum). The support {x : 0 < x < θ} depends on θ, violating condition R2. Here n(θ − θ̂ₙ) converges to an Exponential(1/θ) distribution — a convergence rate of n rather than √n, and a non-normal limit. This dramatic departure underscores why checking regularity conditions is not merely a formality.

SECTION 6

Worked Example: Exponential Distribution

Consider n i.i.d. observations X₁, …, Xₙ from an Exponential(λ) distribution with density f(x; λ) = λe⁻ˡˣ for x > 0 and λ > 0. We will find the MLE, compute the Fisher information, state the asymptotic distribution, and construct a large-sample confidence interval.

Asymptotic Normality for Exponential MLE

Step 1 — Write the Log-LikelihoodThe log-likelihood is ℓₙ(λ) = Σᵢ₌₁ⁿ log(λe⁻ˡˣⁱ) = n·log(λ) − λ·Σᵢ₌₁ⁿ Xᵢ. Denote X̄ = (1/n)Σᵢ Xᵢ, so ℓₙ(λ) = n·log(λ) − nλX̄.

Step 2 — Find the MLETaking the derivative and setting it to zero: ℓₙ′(λ) = n/λ − nX̄ = 0. Solving yields λ̂ₙ = 1/X̄.

λ̂ₙ = 1/X̄

Step 3 — Compute Fisher InformationFor a single observation, log f(X; λ) = log(λ) − λX. The second derivative is ∂²log f/∂λ² = −1/λ². Therefore I(λ) = −E[−1/λ²] = 1/λ². Since E[X] = 1/λ under Exponential(λ), this is finite and positive for all λ > 0.

I(λ) = 1/λ²

Step 4 — State the Asymptotic DistributionThe Exponential family satisfies all regularity conditions (the support (0, ∞) is independent of λ, the log-density is infinitely differentiable in λ, and Fisher information is finite and positive). By the asymptotic normality theorem:

√n(λ̂ₙ − λ₀) →ᵈ N(0, λ₀²), equivalently λ̂ₙ ≈ N(λ₀, λ₀²/n) for large n.

Step 5 — Construct a 95% Confidence IntervalReplacing λ₀ by the estimated λ̂ₙ in the asymptotic variance, the standard error is SE(λ̂ₙ) = λ̂ₙ/√n = 1/(X̄√n). For a 95% CI, use z₀.₀₂₅ = 1.96. Suppose we observe n = 100 with X̄ = 2.5, giving λ̂ = 0.40. Then SE = 0.40/√100 = 0.04.

95% CI: 0.40 ± 1.96 × 0.04 = (0.322, 0.478)

💡 Verification Tip

You can verify the asymptotic variance formula by noting that Var(λ̂ₙ) = Var(1/X̄). By the delta method applied to X̄ with g(μ) = 1/μ, Var(g(X̄)) ≈ [g′(μ)]² × Var(X̄) = (1/μ²)² × (σ²/n) = (1/λ⁻²)² × (λ⁻²/n) = λ²/n, which matches 1/(nI(λ)) = λ²/n.

SECTION 7

Strengths, Limitations & Comparisons

Asymptotic normality of the MLE is one of the most powerful results in statistics, but like any asymptotic theory, it has boundaries of applicability. The following table organizes its key strengths alongside its limitations and common pitfalls that practitioners encounter.

Strengths vs. limitations of MLE asymptotic normality

Strengths	Limitations
Universal recipe: once I(θ) is computed, inference follows automatically — confidence intervals, Wald tests, and likelihood-ratio tests all leverage the normal approximation.	Requires sufficient sample size; the approximation can be poor for small n, especially when the true sampling distribution is highly skewed.
Asymptotic efficiency: MLE achieves the Cramér–Rao lower bound, so no regular estimator can do better asymptotically.	Regularity conditions must be verified; failure (non-regular models) leads to wrong rates and wrong limiting distributions.
Invariance: for a reparametrization η = g(θ), the MLE of η is g(θ̂ₙ), and the delta method extends asymptotic normality to η̂ₙ.	Multiparameter extensions require computing and inverting the Fisher information matrix, which can be computationally demanding for high-dimensional models.
Model-agnostic: the result applies to exponential families, GLMs, mixture models, and many more, provided regularity holds.	Under model misspecification, the MLE converges to the KL-closest parameter value, not necessarily the true parameter, and the asymptotic variance becomes the sandwich form (not simply I(θ)⁻¹).
Provides a natural pivot: the Wald statistic W = (θ̂ₙ − θ₀)/SE has a standard normal limit, enabling simple hypothesis tests.	The Wald confidence interval can have poor coverage when the likelihood is asymmetric; the likelihood-ratio or score interval often performs better in finite samples.

✦ KEY TAKEAWAY

Asymptotic normality is like a GPS system for estimation: it provides an extremely useful approximate position (the MLE) with a stated margin of error (the standard error). But just as GPS accuracy degrades in a dense urban canyon or indoors, the normal approximation degrades when the sample is small, the model is non-regular, or the parameter sits near a boundary. A skilled statistician, like a skilled navigator, knows when to trust the approximation and when to seek alternatives such as the bootstrap, profile likelihood, or exact methods.

SECTION 8

Connections to Advanced Theory

The asymptotic normality of the MLE is not an isolated result — it is a gateway into a rich landscape of advanced statistical theory. Understanding its connections to related concepts deepens your appreciation of its role and guides you toward more sophisticated tools when the standard result falls short.

Advanced extensions and related results

Extension / Related Concept	Relationship to MLE Asymptotic Normality	When You Need It
Delta Method	If √n(θ̂ₙ − θ₀) →ᵈ N(0, σ²) and g is differentiable with g′(θ₀) ≠ 0, then √n(g(θ̂ₙ) − g(θ₀)) →ᵈ N(0, [g′(θ₀)]²σ²). Extends normality to transformations of the MLE.	Inference on reparametrized quantities, e.g., odds ratios, hazard rates, or variance components.
Multiparameter MLE	√n(θ̂ₙ − θ₀) →ᵈ Nₚ(0, I(θ₀)⁻¹), where I(θ₀) is the p × p Fisher information matrix. The proof extends via multivariate Taylor expansion.	Any model with two or more parameters: regression coefficients, mixture proportions, covariance parameters.
Likelihood Ratio Test	Under H₀: θ = θ₀, Wilks' theorem gives 2[ℓₙ(θ̂ₙ) − ℓₙ(θ₀)] →ᵈ χ²₁. This follows directly from the asymptotic normality of the MLE and the quadratic approximation of the log-likelihood.	Hypothesis testing, especially for composite null hypotheses and nested model comparisons.
Sandwich (Robust) Variance	Under misspecification, √n(θ̂ₙ − θ) →ᵈ N(0, I(θ)⁻¹ J(θ) I(θ)⁻¹), where J = Var(score) and I = −E(Hessian). The sandwich replaces I⁻¹ when model and truth diverge.	Robust inference in quasi-likelihood and GEE frameworks, or whenever model correctness is uncertain.
Le Cam's LAN Theory	Local Asymptotic Normality provides a framework in which the entire likelihood ratio process converges to a Gaussian shift experiment. MLE asymptotic normality is a corollary of this deep structural result.	Proving asymptotic optimality of procedures, semiparametric efficiency bounds, and understanding the limits of inference.

Looking forward, the asymptotic normality of the MLE serves as the theoretical foundation for much of modern parametric and semiparametric inference. The Bernstein–von Mises theorem in Bayesian statistics demonstrates that, under regularity, the posterior distribution also converges to the same normal limit — establishing a deep asymptotic agreement between frequentist and Bayesian paradigms. In high-dimensional settings where the parameter dimension p grows with n, classical results break down and are replaced by theories requiring conditions on the ratio p/n, such as those developed by Portnoy, Huber, and more recently in the high-dimensional statistics literature.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain in your own words why the asymptotic normality theorem does not apply to the MLE of θ in a Uniform(0, θ) model. Which specific regularity condition is violated, and what is the actual limiting distribution of the MLE in that case?

PROBLEM 2 — BASIC CALCULATION

Let X₁, …, Xₙ be i.i.d. Bernoulli(p) random variables. Find the MLE of p, compute the Fisher information I(p), and state the asymptotic distribution of √n(p̂ − p₀).

PROBLEM 3 — INTERMEDIATE

Let X₁, …, Xₙ be i.i.d. Poisson(λ) random variables. (a) Derive the MLE λ̂ₙ. (b) Compute I(λ). (c) Use the delta method to find the asymptotic distribution of √n(√λ̂ₙ − √λ₀), where the parameter of interest is the square root of the rate.

PROBLEM 4 — APPLIED

A reliability engineer models component lifetimes as Exponential(λ) and observes n = 50 components with sample mean lifetime X̄ = 1200 hours. (a) Find the MLE of the failure rate λ. (b) Construct an approximate 95% confidence interval for λ using asymptotic normality. (c) Using the invariance property, construct a 95% CI for the mean lifetime μ = 1/λ.

PROBLEM 5 — CRITICAL THINKING

Consider the two-parameter model X₁, …, Xₙ i.i.d. N(μ, σ²) with both μ and σ² unknown. (a) Write the Fisher information matrix I(μ, σ²). (b) Verify that the off-diagonal elements are zero and interpret this. (c) The MLE of σ² is σ̂² = (1/n)Σ(Xᵢ − X̄)². Using the multivariate asymptotic normality result, what is the asymptotic joint distribution of (μ̂, σ̂²)? (d) Discuss why the Wald-based CI for σ² might perform poorly for small n, and suggest an alternative.

SUMMARY

Summary

The asymptotic normality of the MLE states that under standard regularity conditions, the maximum likelihood estimator θ̂ₙ satisfies √n(θ̂ₙ − θ₀) →ᵈ N(0, I(θ₀)⁻¹), where Fisher information I(θ₀) quantifies the curvature of the log-likelihood. The proof rests on a Taylor expansion of the score function combined with the central limit theorem, the weak law of large numbers, and Slutsky's theorem. The result confers asymptotic efficiency upon the MLE: it achieves the Cramér–Rao lower bound in the large-sample limit, meaning no regular estimator can have smaller asymptotic variance.

Practically, this theorem enables the construction of Wald confidence intervals and hypothesis tests for a vast range of parametric models. However, practitioners must verify that regularity conditions hold — in particular, that the support does not depend on the parameter and that the true parameter is interior to the parameter space. When these conditions fail, as in the Uniform(0, θ) example, both the convergence rate and the limiting distribution change dramatically. Extensions to multiparameter models, the delta method, misspecified models (sandwich variance), and Le Cam's LAN theory make asymptotic normality one of the most far-reaching and practically impactful results in the statistician's toolkit.