Likelihood Ratio Tests & Wilks' Theorem — Likelihood ratio tests and Wilks' theorem (intro)

Historical Context & Motivation

The development of rigorous hypothesis testing in the early twentieth century was driven by a fundamental question: given a statistical model, how can we systematically determine whether a simpler explanation (the null hypothesis) is adequate, or whether the data demand a more complex model? Before the likelihood ratio test (LRT) was formalized, practitioners relied on ad hoc procedures or tests tailored to specific distributions, lacking a unifying principle. The LRT emerged from the broader maximum likelihood revolution, providing a test construction method that applies whenever a likelihood function can be written down, regardless of the particular parametric family involved.

The central question that Wilks' theorem addresses is eminently practical: once we compute the likelihood ratio statistic, how do we obtain a p-value without resorting to simulation or exact distribution theory for every specific model? By establishing a universal asymptotic reference distribution, Wilks converted the LRT from a powerful but case-by-case technique into a general-purpose testing framework applicable across parametric statistics.

Core Principles & Definitions

The likelihood ratio test is built on a deceptively simple idea: compare how well the data are explained under the null hypothesis versus the full (unrestricted) model. The comparison takes the form of a ratio of maximized likelihoods, and the resulting statistic encodes precisely how much explanatory power is lost when we impose the null constraints. To set the stage, we define the key objects and then identify the principles that make the LRT so broadly useful.

Visual Explanation

The diagram above captures the logical flow of the LRT procedure. You begin by fitting the model twice: once without constraints (obtaining θ̂ and the maximized likelihood L(θ̂)) and once under the null hypothesis (obtaining θ̂₀ and L(θ̂₀)). Because the null space Θ₀ is a subset of Θ, the restricted maximum can never exceed the unrestricted one, guaranteeing that Λ lies in [0, 1]. Taking −2 log Λ converts values near 1 (small evidence against H₀) to values near 0, and values near 0 (strong evidence against H₀) to large positive numbers. Wilks' theorem then tells us that this transformed statistic, under H₀ and with a large enough sample, behaves like a chi-squared random variable whose degrees of freedom equal the number of restrictions imposed by the null.

Mathematical Framework

We now formalize the likelihood ratio test and state Wilks' theorem with its regularity conditions. Let X₁, X₂, …, Xₙ be i.i.d. random variables from a distribution with density or mass function f(x | θ), where θ ∈ Θ ⊆ ℝᵖ. The null hypothesis H₀: θ ∈ Θ₀ restricts the parameter to a subset Θ₀ of dimension p − r, where r is the number of independent constraints.

The proof sketch of Wilks' theorem rests on a second-order Taylor expansion of the log-likelihood around the unrestricted MLE θ̂. Near θ̂, the log-likelihood is approximately quadratic: ℓ(θ) ≈ ℓ(θ̂) − ½(θ − θ̂)ᵀ I(θ̂)(θ − θ̂), where I(θ̂) is the observed information matrix. Substituting both θ̂ and θ̂₀ into this expansion, one can show that −2 log Λ reduces asymptotically to a quadratic form in a normal vector, which by definition is chi-squared. The dimension r emerges because the null constrains exactly r components (or r independent linear combinations) of the parameter, and the remaining p − r components are free under both hypotheses and cancel in the difference.

The Chi-Squared Reference Distribution

Understanding how the asymptotic chi-squared distribution arises—and what the degrees of freedom r represent—is essential for correctly applying the LRT. In practice, the degrees of freedom count the number of free parameters "lost" when we move from the full model to the null model. For example, testing whether three means are all equal (H₀: μ₁ = μ₂ = μ₃) versus unrestricted means imposes r = 2 independent constraints, so −2 log Λ is compared to χ²(2). The following diagram illustrates chi-squared densities for several values of r and shows how the rejection region shifts.

As seen in the diagram, the shape of the chi-squared distribution depends heavily on the degrees of freedom r. When r = 1 (a single restriction, such as testing whether one parameter equals a specified value), the density is a steeply decreasing curve concentrated near zero. As r increases, the density becomes more symmetric and bell-shaped, with its mode shifting to the right at r − 2. This behavior has a practical consequence: tests with many restrictions (large r) require correspondingly large values of −2 log Λ to achieve significance, reflecting the fact that it becomes increasingly unlikely for all restrictions to be binding simultaneously by chance alone.

Worked Example: Testing a Normal Mean

Suppose we observe X₁, X₂, …, Xₙ i.i.d. ~ N(μ, σ²), where both μ and σ² are unknown. We wish to test H₀: μ = μ₀ versus H₁: μ ≠ μ₀. The full parameter space is Θ = {(μ, σ²) : μ ∈ ℝ, σ² > 0} (dim = 2), and the null parameter space is Θ₀ = {(μ₀, σ²) : σ² > 0} (dim = 1), so r = 2 − 1 = 1. Let us work through this with concrete numbers: n = 25, X̄ = 5.4, s² = 4.0, and μ₀ = 5.0.

Strengths, Limitations & Comparison with Other Tests

The LRT is one member of a trio of asymptotically equivalent testing procedures in regular parametric models: the likelihood ratio test, the Wald test, and the score (Lagrange multiplier) test. All three are based on the maximum likelihood framework, and under H₀ with regularity conditions, each yields a test statistic that converges to the same χ²(r) distribution. However, they differ in computational requirements, finite-sample behavior, and sensitivity to parametrization, making each more convenient in different settings.

A key limitation of the LRT arises when Wilks' regularity conditions are violated. Common situations include testing a parameter on the boundary of the parameter space (e.g., H₀: σ² = 0 in a variance-components model), testing in the presence of nuisance parameters that exist only under the alternative (as in mixture models), and high-dimensional settings where the number of parameters grows with n. In these cases, −2 log Λ may converge to a mixture of chi-squared distributions, a non-standard distribution, or may not converge at all, and specialized theory or simulation-based calibration is needed.

Connections to Advanced Theory

The likelihood ratio test occupies a central position in the broader architecture of statistical inference. Its connections extend in several directions: toward information-theoretic model selection, toward Bayesian testing, and toward modern high-dimensional statistics. Understanding these links helps contextualize why the LRT has endured as a core technique, and where researchers have needed to go beyond it.

A particularly elegant connection is between the LRT and the Akaike Information Criterion (AIC). When comparing two nested models, AIC penalizes the larger model's log-likelihood by its additional number of parameters. The LRT asks whether the improvement in log-likelihood is "large enough" relative to a χ² threshold, while AIC asks whether it is large enough to offset the penalty for model complexity. Both are rooted in the same likelihood machinery, but AIC is designed for prediction whereas the LRT is designed for inference about parameters. As you advance in your study of statistics, you will see that these connections—between testing, estimation, and model selection—form an interconnected web that underlies modern data analysis.

Practice Problems

Summary