Historical Context & Motivation
Statistical inference confronts a deceptively simple question: given a sample of observed data, what can we say about the mechanism that generated it? Before the twentieth century, methods for answering this question were largely ad hoc—least squares for regression, the method of moments for distribution fitting—but no unified principle connected them. The introduction of the likelihood function provided exactly that unifying framework, reorienting the entire enterprise of parametric inference around a single, elegant object that measures the plausibility of a parameter value in light of observed data.
The need for such a framework grew acute as scientists in fields ranging from genetics to astronomy demanded principled methods for combining observations. Early contributions by Carl Friedrich Gauss and Pierre-Simon Laplace laid important groundwork through their work on inverse probability, but these approaches relied on prior distributions in ways that many found unsatisfying. It was R. A. Fisher who, starting in the 1920s, distilled the concept of likelihood into a formal principle that avoided the need for priors altogether, creating a cornerstone of frequentist inference and opening the door to maximum likelihood estimation, likelihood ratio tests, and the modern theory of efficient estimation.
The central question this lesson addresses is: how do we formalize the relationship between observed data and unknown parameters, and how do the likelihood, log-likelihood, and score function work together to produce optimal estimators and powerful tests?
Core Principles & Definitions
Understanding likelihood-based inference requires internalizing a subtle but crucial reversal of perspective. In probability, we fix a model (parameter values) and ask about the distribution of data. In likelihood, we fix the data and ask which parameter values are most compatible with what we actually observed. This reversal is not merely philosophical—it reshapes the entire mathematical apparatus we deploy. The following core principles constitute the foundation upon which maximum likelihood estimation, likelihood ratio tests, and score-based inference are built.
Likelihood Function L(θ)
Log-Likelihood ℓ(θ)
Score Function S(θ)
Fisher Information I(θ)
Likelihood Principle
Visual Explanation — The Likelihood Landscape
The following diagram illustrates the fundamental objects of likelihood inference for a simple one-parameter problem. We observe a sample and then view the likelihood, log-likelihood, and score function as functions of the parameter θ. Notice how the peak of the likelihood corresponds to the peak of the log-likelihood, and the score function crosses zero precisely at that peak. The width of the peak encodes how much the data constrain the parameter, which is formalized by the Fisher information.
Several features in the diagram merit careful attention. First, note that L(θ) and ℓ(θ) achieve their maxima at exactly the same θ̂—this is guaranteed because the natural logarithm is a strictly increasing function, so it preserves the ordering of function values. Second, the curvature of the log-likelihood near the peak determines how 'informative' the data are: a sharply peaked log-likelihood means the data strongly favor a narrow range of parameter values, and the Fisher information quantifies this curvature. Third, the score function S(θ) provides a natural mechanism for finding the MLE—simply solve S(θ) = 0—and its variance at the true parameter value equals the Fisher information, linking estimation precision directly to the shape of the likelihood surface.
Mathematical Framework
We now formalize the three central objects—likelihood, log-likelihood, and score—for a parametric model. Suppose we observe an independent and identically distributed (i.i.d.) sample X₁, X₂, …, Xₙ from a distribution with probability density (or mass) function f(x | θ), where θ ∈ Θ ⊆ ℝ is an unknown scalar parameter. The formalism extends naturally to vector-valued parameters, but the scalar case illuminates the core ideas most clearly.
The power of this framework becomes apparent when we connect these objects. The MLE θ̂ is obtained by solving S(θ̂) = 0. Under regularity conditions and as n → ∞, the MLE is consistent (θ̂ → θ₀ in probability), asymptotically normal (√n(θ̂ − θ₀) → N(0, 1/I(θ₀))), and asymptotically efficient (it achieves the Cramér–Rao lower bound). The score function thus serves as both a computational tool for finding estimators and a theoretical vehicle for establishing their optimality properties.
Score Function Properties & Fisher Information
The score function is far more than a mere derivative—it is an object with deep probabilistic structure that underpins the entire asymptotic theory of likelihood-based inference. In this section we examine its key properties in detail and connect them to the geometry of the Fisher information. The diagram below provides a visual roadmap of how the score, Fisher information, and the Cramér–Rao bound relate to one another in the estimation pipeline.
Proof Sketch: E[S(θ₀)] = 0
Under regularity conditions, we can differentiate under the integral sign. Start from the identity ∫ f(x | θ) dx = 1, which holds for all θ. Differentiating both sides with respect to θ yields ∫ (∂f/∂θ) dx = 0. Now observe that ∂ ln f / ∂θ = (1/f)(∂f/∂θ), so ∂f/∂θ = f · (∂ ln f / ∂θ). Substituting, we obtain ∫ (∂ ln f / ∂θ) · f dx = E[∂ ln f / ∂θ] = E[S(θ)] = 0. This elegant result states that the score function, evaluated at the true parameter, is an unbiased estimating equation—its expectation is zero. This is why solving S(θ̂) = 0 produces consistent estimators.
Information Identity
A second differentiation of ∫ f dx = 1 reveals the information identity. Differentiating E[S(θ)] = 0 with respect to θ gives E[∂S/∂θ] + E[S(θ) · S(θ)] = 0 (using the product rule inside the expectation). Since E[S(θ)] = 0, we have Var[S(θ)] = E[S(θ)²] = −E[∂²ℓ/∂θ²]. This identity is computationally valuable because it allows us to compute Fisher information via the expected negative second derivative of the log-likelihood, which is often easier to evaluate than the variance of the score directly.
| Property | Statement | Significance |
|---|---|---|
| Zero mean | E[S(θ₀)] = 0 | Score is an unbiased estimating equation; root gives consistent estimator |
| Variance = Information | Var[S(θ₀)] = I(θ₀) | Connects score variability to curvature of ℓ(θ) |
| Additivity | I_n(θ) = n · I₁(θ) for i.i.d. data | Total information grows linearly with sample size |
| Asymptotic normality | S(θ₀)/√(nI₁(θ₀)) → N(0,1) | Basis for score tests (Rao test) and confidence intervals |
Worked Example — Exponential Distribution
Let us work through a complete example using the exponential distribution, which models waiting times and is one of the most common distributions in reliability engineering and survival analysis. Suppose we observe X₁, X₂, …, Xₙ i.i.d. from Exp(λ), where the density is f(x | λ) = λe⁻ˡˣ for x ≥ 0, λ > 0. We will derive the likelihood, log-likelihood, score, Fisher information, and the MLE.
Strengths, Limitations & Comparisons
Likelihood-based inference is one of the most powerful frameworks in statistics, but it is not without limitations. Understanding when and why it excels—and where alternatives may be preferred—is essential for the practicing statistician. The table below contrasts the likelihood approach with two competing paradigms: the method of moments and Bayesian inference. Each brings distinct philosophical commitments and practical trade-offs.
| Feature | Maximum Likelihood | Method of Moments | Bayesian Inference |
|---|---|---|---|
| Efficiency | Asymptotically efficient; achieves Cramér–Rao bound | Generally less efficient; not optimal in general | Optimal under correct prior; can outperform MLE in small samples |
| Computation | Requires numerical optimization; can be complex for multimodal likelihoods | Solving system of equations; usually closed-form | Requires MCMC or variational methods for complex models |
| Regularity requirements | Needs regularity conditions for standard theory to apply | Minimal; existence of moments suffices | Needs proper posterior; prior specification required |
| Small sample behavior | Can be biased; asymptotic theory may be inaccurate | Often unbiased but high variance | Regularized by prior; can handle small samples well |
| Model misspecification | Converges to KL-closest model; pseudo-true parameter | Converges to moment-matched parameter | Posterior concentrates on KL-closest model; sensitive to prior |
Connection to Advanced Inference Theory
The likelihood, log-likelihood, and score function are not endpoints but launching pads for the most powerful results in statistical inference. They connect directly to the three pillars of asymptotic testing—the Wald test, likelihood ratio test, and score (Rao) test—all of which are asymptotically equivalent under the null hypothesis but differ in finite-sample properties and computational convenience. Moreover, the score function's structure underpins modern developments in semiparametric efficiency theory, estimating equations, and computational statistics.
| Concept | Relationship to Score/Likelihood | Key Result |
|---|---|---|
| Wald Test | Uses MLE (root of score) and its estimated variance from Fisher information | W = (θ̂ − θ₀)² · I(θ̂) →ᵈ χ²(1) under H₀ |
| Likelihood Ratio Test | Compares log-likelihood at θ̂ versus θ₀ directly | Λ = 2[ℓ(θ̂) − ℓ(θ₀)] →ᵈ χ²(1) under H₀ (Wilks' theorem) |
| Score (Rao) Test | Evaluates score at θ₀ only; no need to compute MLE | R = S(θ₀)² / I(θ₀) →ᵈ χ²(1) under H₀ |
| Cramér–Rao Bound | Fisher information sets the floor for unbiased estimator variance | Var(T) ≥ [g′(θ)]² / I(θ) for any unbiased estimator T of g(θ) |
| EM Algorithm | Maximizes expected complete-data log-likelihood; each M-step solves augmented score equations | Monotonically increases observed-data likelihood at each iteration |
| Quasi-Likelihood | Uses score-like estimating equations without full distributional assumptions | Consistent estimation of mean parameters under correct mean-variance specification |
A particularly important connection is to the multiparameter setting. When θ = (θ₁, …, θₖ)ᵀ is a vector, the score becomes a gradient vector S(θ) = ∇ℓ(θ) and the Fisher information becomes a k × k matrix I(θ) with entries Iⱼₖ = −E[∂²ℓ / ∂θⱼ∂θₖ]. The MLE is asymptotically multivariate normal: √n(θ̂ − θ₀) →ᵈ Nₖ(0, I₁(θ₀)⁻¹). This extension is the backbone of generalized linear models, mixed models, and the entire modern statistical modeling enterprise.
Practice Problems
Summary & Review
This lesson developed the three foundational objects of parametric inference. The likelihood function L(θ) = ∏ f(xᵢ | θ) reinterprets the joint density as a function of the parameter, quantifying the plausibility of each parameter value given the data. The log-likelihood ℓ(θ) = ∑ ln f(xᵢ | θ) converts products to sums, making analytical and computational manipulation tractable while preserving the location of the maximum. The score function S(θ) = ∂ℓ/∂θ is the gradient of the log-likelihood; it has zero mean at the true parameter, and setting it to zero yields the maximum likelihood estimator.
The variance of the score equals the Fisher information I(θ) = −E[∂²ℓ/∂θ²], which measures the curvature of the log-likelihood and sets the Cramér–Rao lower bound on the variance of unbiased estimators. Under regularity conditions, the MLE is consistent, asymptotically normal with variance 1/nI(θ), and asymptotically efficient. These results connect directly to the three large-sample tests—Wald, likelihood ratio, and score tests—which are the workhorses of modern statistical hypothesis testing.