Fisher Information & Cramér-Rao Bound

SECTION 1

Historical Context & Motivation

Throughout the early twentieth century, statisticians grappled with a deceptively simple question: given a probabilistic model for observed data, how precisely can we hope to estimate an unknown parameter? The intuition that sharper likelihood functions should yield more precise estimators was widely felt, but formalizing this intuition required entirely new mathematical machinery. Ronald Aylmer Fisher, working at the Rothamsted Experimental Station in England, introduced the concept of Fisher information as a way to quantify the amount of information that an observable random variable carries about an unknown parameter upon which its probability distribution depends. This quantity provided a rigorous foundation for comparing the efficiency of competing estimators and established the theoretical backbone for the modern theory of point estimation.

The subsequent work by Harald Cramér and C. R. Rao independently established a universal lower bound on the variance of any unbiased estimator — the celebrated Cramér–Rao lower bound. This result transformed statistical inference by providing an absolute benchmark: no matter how clever an estimation procedure might be, if it is unbiased, its variance cannot fall below the reciprocal of the Fisher information. The interplay between Fisher information and the Cramér–Rao bound continues to pervade modern statistics, machine learning, information geometry, and quantum estimation theory.

1920–1922

Fisher's Foundational Papers

R. A. Fisher publishes 'A Mathematical Examination of the Methods of Determining the Accuracy of an Observation' and introduces the concept of sufficient statistics and the score function, laying the groundwork for information-based inference.

1925

Theory of Statistical Estimation

Fisher formally defines the information contained in a sample about a parameter, connecting the curvature of the log-likelihood to estimator variance, and introduces the concept of efficiency.

1945–1946

Cramér & Rao Prove the Lower Bound

Harald Cramér (Sweden) and C. R. Rao (India) independently derive the inequality now known as the Cramér–Rao lower bound, establishing a universal variance floor for unbiased estimators.

1950s–1960s

Information Geometry & Extensions

The Fisher information matrix is recognized as a Riemannian metric on the space of probability distributions, initiating the field of information geometry. Multi-parameter extensions and connections to the Rao–Blackwell theorem are developed.

2000s–present

Modern Applications

Fisher information becomes central to natural gradient methods in deep learning, quantum estimation bounds (quantum Fisher information), and high-dimensional statistics where efficiency benchmarks guide regularized estimation.

The central question that motivated these developments remains profoundly relevant: Given a statistical model, what is the best precision we can ever achieve when estimating its parameters, and which estimators attain that precision? This lesson develops the answers from first principles.

SECTION 2

Core Principles & Definitions

Before diving into the mathematics, it is essential to understand the conceptual pillars upon which Fisher information and the Cramér–Rao bound rest. These ideas connect the geometry of probability distributions, the behavior of the likelihood function, and the fundamental limits of statistical estimation. The following core principles provide the conceptual roadmap for everything that follows.

The Score Function

The score function is the gradient of the log-likelihood with respect to the parameter θ. It captures how sensitive the likelihood is to changes in θ. Under regularity conditions, the expected value of the score is zero: E[s(X; θ)] = 0.

Fisher Information as Curvature

The Fisher information I(θ) equals the variance of the score function. Equivalently, it is the expected negative second derivative of the log-likelihood — the average curvature of the log-likelihood surface around θ. Sharper curvature means more information.

Unbiased Estimation

An estimator T(X) is unbiased for θ if E[T(X)] = θ for all θ in the parameter space. The Cramér–Rao bound applies directly to unbiased estimators and can be extended to biased ones through a bias-correction term.

Cramér–Rao Lower Bound (CRLB)

For any unbiased estimator T of θ, the variance satisfies Var(T) ≥ 1/I(θ). This is the Cramér–Rao lower bound. No unbiased estimator can have variance smaller than the reciprocal of the Fisher information.

Efficiency

An unbiased estimator that achieves the CRLB with equality is called efficient. Efficiency measures how close an estimator comes to this theoretical optimum. The ratio 1/[nI(θ)·Var(T)] defines relative efficiency for n i.i.d. observations.

✦ KEY TAKEAWAY

Think of the log-likelihood function as a landscape with a peak at the true parameter value. Fisher information measures how sharply peaked that landscape is. A narrow, steep peak (high Fisher information) means data are very informative — like a GPS signal with high precision. A broad, flat peak (low Fisher information) means data are noisy and the parameter is hard to pin down — like trying to locate yourself with a weak, scattered signal. The Cramér–Rao bound tells you that no unbiased estimator can do better than what the sharpness of the peak allows.

SECTION 3

Visual Explanation — Log-Likelihood Curvature and Fisher Information

The following diagram illustrates the fundamental geometric intuition behind Fisher information. Two log-likelihood functions are shown: one with high curvature (high Fisher information) and one with low curvature (low Fisher information). The sharper the log-likelihood peak around the true parameter value θ₀, the more precisely we can estimate θ, and the tighter the Cramér–Rao lower bound becomes.

The solid cyan curve represents a log-likelihood with high curvature (high Fisher information), producing a narrow, sharp peak around the true value θ₀ and yielding low estimator variance. The dashed pink curve represents a flatter log-likelihood (low Fisher information), giving a wider spread and higher estimator variance. The Cramér–Rao bound directly links this curvature to the minimum achievable variance.

In the diagram above, both log-likelihood curves are maximized at the true parameter value θ₀, but they differ in their curvature. The high-information curve (solid cyan) drops off steeply on either side, indicating that the data strongly discriminate between nearby parameter values. The low-information curve (dashed pink) descends gently, meaning many parameter values near θ₀ are almost equally compatible with the data. Fisher information, I(θ), quantifies this distinction by measuring the expected value of the squared score or, equivalently, the expected negative second derivative of ℓ(θ). Higher curvature translates directly into a tighter lower bound on estimator variance via the Cramér–Rao inequality.

SECTION 4

Mathematical Framework

We now develop the formal mathematical definitions and the derivation of the Cramér–Rao bound. Let X = (X₁, …, Xₙ) be a random sample from a distribution with probability density (or mass) function f(x; θ), where θ ∈ Θ ⊆ ℝ is the unknown parameter. We assume the standard regularity conditions hold: the support of f does not depend on θ, the density is twice differentiable with respect to θ, and interchange of differentiation and integration is valid.

The Score Function

SCORE FUNCTION

s(x; θ) = ∂/∂θ ln f(x; θ)

The score function s(x; θ) is the partial derivative of the log-likelihood with respect to θ. A key property: E_θ[s(X; θ)] = 0 for all θ, which follows from differentiating ∫f(x; θ)dx = 1 under the integral sign.

Fisher Information — Single Observation

FISHER INFORMATION (VARIANCE FORM)

I(θ) = E_θ[ (∂/∂θ ln f(X; θ))² ] = Var_θ[s(X; θ)]

Since E[s] = 0, the Fisher information equals both the second moment and the variance of the score. This form emphasizes that I(θ) measures the variability of the score — how much the gradient of the log-likelihood fluctuates across samples.

FISHER INFORMATION (CURVATURE FORM)

I(θ) = −E_θ[ ∂²/∂θ² ln f(X; θ) ]

Under regularity conditions, the Fisher information also equals the expected negative second derivative of the log-likelihood. This curvature interpretation is often more convenient computationally. Greater expected curvature means the log-likelihood is more sharply peaked around θ.

Fisher Information for n i.i.d. Observations

INFORMATION ADDITIVITY

I_n(θ) = n · I(θ)

For n independent and identically distributed observations, the total Fisher information is simply n times the single-observation Fisher information. This additivity property follows from the independence of the score contributions and is one of the most practically useful features of Fisher information.

Derivation of the Cramér–Rao Lower Bound

Let T(X) be an unbiased estimator of θ, so E_θ[T(X)] = θ for all θ. Differentiating both sides with respect to θ under the integral sign yields: ∫ T(x) · [∂/∂θ f(x; θ)] dx = 1. Rewriting ∂f/∂θ = f · s, we obtain E_θ[T(X) · s(X; θ)] = 1. Since E[s] = 0, this is equivalent to Cov(T, s) = 1. Applying the Cauchy–Schwarz inequality to this covariance: [Cov(T, s)]² ≤ Var(T) · Var(s), which gives 1 ≤ Var(T) · I(θ). Rearranging produces the celebrated result.

CRAMÉR–RAO LOWER BOUND

Var_θ(T) ≥ 1 / I_n(θ) = 1 / [n · I(θ)]

For any unbiased estimator T of θ based on n i.i.d. observations, the variance of T is bounded below by the reciprocal of the total Fisher information. Equality holds if and only if T − θ is proportional to the score function, i.e., T(X) = θ + a(θ) · Σs(Xᵢ; θ) for some function a(θ).

📐 Biased Estimator Extension

If T is a biased estimator with E[T] = θ + b(θ), the bound generalizes to Var(T) ≥ [1 + b′(θ)]² / I_n(θ), where b′(θ) is the derivative of the bias function. This extension is important because many practical estimators (e.g., regularized or shrinkage estimators) are intentionally biased to reduce mean squared error.

SECTION 5

Fisher Information for Common Distributions

To build intuition and provide practical reference, we now compute the Fisher information for several important parametric families. These results are foundational — they appear repeatedly in hypothesis testing, confidence interval construction, and efficiency comparisons. The table below summarizes the key results, followed by a detailed visual diagram showing how Fisher information varies as a function of the parameter for the Normal and Bernoulli families.

Fisher information and Cramér–Rao bounds for common one-parameter families

Distribution	Parameter θ	I(θ) per observation	CRLB: 1/[nI(θ)]
Bernoulli(p)	p ∈ (0, 1)	1 / [p(1 − p)]	p(1 − p) / n
Poisson(λ)	λ > 0	1 / λ	λ / n
Normal(μ, σ²) (σ² known)	μ ∈ ℝ	1 / σ²	σ² / n
Normal(μ, σ²) (μ known)	σ² > 0	1 / (2σ⁴)	2σ⁴ / n
Exponential(λ)	λ > 0 (rate)	1 / λ²	λ² / n
Uniform(0, θ)	θ > 0	Not regular (support depends on θ)	CRLB does not apply

Top left: For the Bernoulli distribution, Fisher information I(p) = 1/[p(1−p)] is U-shaped, reaching its minimum at p = 0.5 (hardest to estimate) and diverging near p = 0 or 1. Top right: For the Exponential, I(λ) = 1/λ² is a monotonically decreasing function — small rates carry more information. Bottom: The CRLB for the Bernoulli sample proportion shrinks as sample size n increases, with the bound tightest near p = 0 or 1 and loosest at p = 0.5.

Several patterns emerge from these results. First, note that the Uniform(0, θ) distribution fails the regularity conditions because its support {x : 0 ≤ x ≤ θ} depends on θ, so the standard CRLB does not apply — and indeed, the maximum order statistic X_(n) achieves a variance of order O(1/n²), which is faster than any regular parametric rate. Second, for the Bernoulli case, the Fisher information is smallest when p = 0.5, reflecting the intuition that a fair coin is the hardest case for estimating the bias. Third, the sample mean X̄ achieves the CRLB for Normal(μ, σ²) with known σ², and the sample proportion p̂ = X̄ achieves it for the Bernoulli — both are efficient estimators.

SECTION 6

Worked Example — Exponential Distribution

Suppose X₁, X₂, …, Xₙ are i.i.d. Exponential(λ) random variables with density f(x; λ) = λe^−λx for x > 0. We wish to compute the Fisher information for λ, derive the CRLB, identify the maximum likelihood estimator (MLE), and check whether it achieves the bound.

Fisher Information & CRLB for Exponential(λ)

Step 1 — Write the Log-LikelihoodThe log-likelihood for a single observation is ℓ(λ; x) = ln(λe^−λx) = ln λ − λx. For n i.i.d. observations: ℓ(λ; x₁, …, xₙ) = n ln λ − λΣxᵢ.

ℓ(λ) = n ln λ − λΣxᵢ

Step 2 — Compute the Score FunctionFor a single observation, differentiate with respect to λ: s(x; λ) = ∂ℓ/∂λ = 1/λ − x. The score captures how fast the log-likelihood changes as we vary λ.

s(x; λ) = 1/λ − x

Step 3 — Compute Fisher Information (Variance Form)Since E[X] = 1/λ for the Exponential distribution, E[s] = 1/λ − E[X] = 1/λ − 1/λ = 0 (confirming the score has mean zero). The Fisher information is I(λ) = Var(s) = Var(1/λ − X) = Var(X) = 1/λ². Alternatively, the second derivative is ∂²ℓ/∂λ² = −1/λ², so −E[∂²ℓ/∂λ²] = 1/λ², confirming the result.

I(λ) = 1/λ²

Step 4 — Derive the Cramér–Rao Lower BoundFor n i.i.d. observations, the total Fisher information is Iₙ(λ) = n/λ². The CRLB for any unbiased estimator T of λ is therefore Var(T) ≥ 1/Iₙ(λ) = λ²/n.

CRLB = λ²/n

Step 5 — Find the MLE and Check EfficiencySetting the total score to zero: n/λ − Σxᵢ = 0, giving λ̂ = n/Σxᵢ = 1/X̄. To check: Var(X̄) = σ²/n = (1/λ²)/n, so Var(1/X̄) requires the delta method. Using g(μ) = 1/μ, g′(μ) = −1/μ², and μ = E[X̄] = 1/λ, we get Var(λ̂) ≈ [g′(1/λ)]² · Var(X̄) = λ⁴ · (1/λ²)/n = λ²/n. This matches the CRLB exactly (asymptotically), so the MLE is asymptotically efficient.

MLE: λ̂ = 1/X̄, asymptotic Var(λ̂) = λ²/n = CRLB ✓

⚠️ Note on Exact vs. Asymptotic Efficiency

The MLE λ̂ = 1/X̄ is biased in finite samples (Jensen's inequality ensures E[1/X̄] ≠ 1/E[X̄]). However, the bias vanishes as n → ∞. For the exact unbiased estimator of λ, one can show that T = (n − 1)/ΣXᵢ is unbiased, and its exact variance exceeds the CRLB, demonstrating that the CRLB is not always attainable for finite n in this family.

SECTION 7

Strengths, Limitations, and Practical Considerations

The Cramér–Rao bound is one of the most elegant results in mathematical statistics, but its practical utility comes with important caveats. Understanding both its power and its limitations is essential for applying it correctly in research and applications.

Strengths and limitations of the Fisher information and Cramér–Rao framework

Strengths	Limitations
Provides a universal benchmark: any unbiased estimator's variance can be compared to 1/I(θ).	Requires regularity conditions (differentiable log-likelihood, support independent of θ). Fails for Uniform(0, θ), order statistics.
Computationally straightforward — often only requires differentiating the log-density twice.	The bound may not be tight: for many models no unbiased estimator achieves it exactly. It is attainable iff the model belongs to the exponential family.
Extends naturally to multi-parameter settings via the Fisher information matrix, yielding matrix inequalities on covariance matrices.	Applies only to unbiased estimators in its basic form. Biased estimators (e.g., James–Stein) can have lower MSE than the CRLB on variance.
Foundational for asymptotic theory: MLE variance converges to 1/[nI(θ)], connecting finite-sample bounds to large-sample behavior.	In high-dimensional or nonparametric settings, parametric information bounds may be overly optimistic and minimax bounds are more appropriate.

✦ KEY TAKEAWAY

The Cramér–Rao bound is like a speed limit on a highway: it tells you the theoretical maximum precision (minimum variance) any unbiased estimator can achieve. Some estimators drive right at the limit (efficient estimators in exponential families), while others fall short. And just as speed limits don't apply to trains on separate tracks, the CRLB doesn't constrain biased estimators or nonparametric procedures, which play by different rules. Always check whether the regularity conditions hold before invoking the bound.

SECTION 8

Connections to Advanced Theory

Fisher information and the Cramér–Rao bound serve as launching points for several advanced areas of statistical theory. Understanding these connections clarifies why this material is so foundational and reveals the richer structure that emerges in more sophisticated settings.

Advanced topics connected to Fisher information

Concept	Relationship to Fisher Information / CRLB
Asymptotic Normality of MLE	Under regularity conditions, √n(θ̂ − θ) →ᵈ N(0, 1/I(θ)). The MLE is asymptotically efficient, attaining the CRLB in the limit. Fisher information governs the asymptotic variance.
Rao–Blackwell & Lehmann–Scheffé	The Rao–Blackwell theorem improves estimators by conditioning on sufficient statistics. Combined with completeness (Lehmann–Scheffé), it yields the UMVUE, which achieves the CRLB when it is attainable (exponential families).
Fisher Information Matrix	For multi-parameter models θ = (θ₁, …, θₖ), the information matrix [I(θ)]ᵢⱼ = E[−∂²ℓ/∂θᵢ∂θⱼ] generalizes I(θ). The CRLB becomes a matrix inequality: Cov(T) ≥ I(θ)⁻¹ in the Loewner order.
Information Geometry	The Fisher information matrix defines a Riemannian metric on the space of probability distributions, making statistical models into curved manifolds. Geodesic distances correspond to statistical distinguishability, and natural gradient descent exploits this geometry in machine learning.
Quantum Fisher Information	In quantum estimation theory, the quantum Cramér–Rao bound extends the classical result to quantum states. The quantum Fisher information provides the ultimate precision limit for parameter estimation in quantum mechanics, central to quantum metrology.

Perhaps the most important forward-looking connection is to the asymptotic efficiency of maximum likelihood estimation. The MLE theorem guarantees that under regularity conditions, the MLE θ̂ satisfies √n(θ̂ − θ₀) →ᵈ N(0, [I(θ₀)]⁻¹), meaning its asymptotic variance matches the CRLB exactly. This deep result — connecting a computable estimation procedure to a fundamental information-theoretic limit — is one of the crown jewels of classical statistics. Furthermore, the observed Fisher information (evaluated at θ̂ rather than the true θ₀) provides the basis for standard errors and Wald-type confidence intervals used throughout applied statistics.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain in your own words why Fisher information is defined as the variance of the score function rather than, say, the mean of the score. What would happen if the mean of the score were nonzero?

PROBLEM 2 — BASIC CALCULATION

Let X₁, …, X₂₅ be i.i.d. Bernoulli(p) random variables with p = 0.3. Compute the Fisher information I(p) for a single observation, the total Fisher information I₂₅(p), and the Cramér–Rao lower bound for any unbiased estimator of p.

PROBLEM 3 — INTERMEDIATE

Let X₁, …, Xₙ be i.i.d. Normal(μ, σ²) where both μ and σ² are unknown. Compute the 2 × 2 Fisher information matrix I(μ, σ²). Verify that the off-diagonal entries are zero and interpret this result.

PROBLEM 4 — APPLIED

A quality engineer models the time-to-failure of a component as Exponential(λ). From n = 40 units, she observes a total failure time of ΣXᵢ = 2000 hours. Compute the MLE of λ, an approximate 95% confidence interval using the observed Fisher information, and assess whether the MLE is achieving the CRLB.

PROBLEM 5 — CRITICAL THINKING

Consider the Uniform(0, θ) distribution with density f(x; θ) = 1/θ for 0 < x < θ. Explain why the Cramér–Rao bound does not apply here. The MLE is θ̂ = X₍ₙ₎ (the maximum order statistic), whose exact variance is nθ²/[(n+1)²(n+2)]. Show that this variance is O(1/n²), far smaller than any rate achievable under regularity. What does this tell us about the relationship between regularity conditions and estimation precision?

SUMMARY

Summary — Fisher Information & Cramér–Rao Bound

The Fisher information I(θ) quantifies the amount of information that an observable random variable carries about an unknown parameter θ. Defined as the variance of the score function s(X; θ) = ∂ ln f/∂θ, or equivalently as the expected negative curvature of the log-likelihood, it provides a scalar measure of how sharply the log-likelihood peaks around the true parameter value. For n i.i.d. observations, the total Fisher information is Iₙ(θ) = nI(θ), reflecting the additive nature of independent information.

The Cramér–Rao lower bound establishes that for any unbiased estimator T, the variance satisfies Var(T) ≥ 1/[nI(θ)]. This bound, derived via the Cauchy–Schwarz inequality, provides an absolute efficiency benchmark. Estimators that achieve this bound with equality are called efficient, and such efficiency is attainable precisely in exponential family models. The MLE is guaranteed to be asymptotically efficient under regularity conditions, making the CRLB the cornerstone for constructing confidence intervals, comparing estimators, and understanding the fundamental limits of statistical inference.