Likelihood-Based CIs — Likelihood-based confidence...

SECTION 1

Historical Context & Motivation

The idea of building confidence intervals from the likelihood function rather than from sampling distributions has deep roots in twentieth-century statistical theory. Classical Wald-type intervals, constructed by adding and subtracting a multiple of the standard error from a point estimate, served as the workhorse of interval estimation for decades. However, statisticians gradually recognized that Wald intervals can behave poorly when sample sizes are moderate, when parameters are near boundary values, or when the likelihood surface is markedly asymmetric. These shortcomings motivated a fundamentally different strategy: reading the interval endpoints directly off the likelihood function itself.

The likelihood-based approach treats the shape of the log-likelihood as the primary source of information about the parameter, replacing the assumption of approximate normality with a more faithful representation of the data's evidence. The profile likelihood technique extends this program to settings with nuisance parameters, enabling the analyst to project the full likelihood surface onto the axis of interest. Understanding these historical motivations clarifies why likelihood-based confidence intervals remain a cornerstone of modern inference, especially in fields like biostatistics, econometrics, and physics where the consequences of poor coverage can be severe.

1922

Fisher's Likelihood Principle

R.A. Fisher formalizes the concept of the likelihood function and the method of maximum likelihood estimation (MLE), laying the foundation for all subsequent likelihood-based inference.

1938

Neyman–Pearson Framework Matures

Jerzy Neyman and Egon Pearson develop the duality between hypothesis tests and confidence sets, establishing the inversion principle that connects likelihood ratio tests to confidence regions.

1938

Wilks' Theorem

Samuel Wilks proves that twice the log-likelihood ratio statistic converges to a chi-squared distribution under regularity conditions, providing the asymptotic justification for likelihood ratio–based intervals.

1973

Profile Likelihood Formalized

Barndorff-Nielsen, Cox, and others systematize the profile likelihood as a tool for eliminating nuisance parameters, enabling likelihood-based inference in complex multiparameter models.

1990s–Present

Computational Revolution

Advances in numerical optimization and computing power make profile likelihood CIs practical for high-dimensional models, generalized linear models, mixed-effects models, and survival analysis.

The central question that likelihood-based confidence intervals address is this: when the standard normal approximation underlying Wald intervals is unreliable—due to skewness, boundary effects, or small samples—how can we construct intervals that more faithfully reflect the information actually present in the data? The answer, as we shall see, lies in letting the curvature and shape of the likelihood speak for themselves.

SECTION 2

Core Principles & Definitions

Likelihood-based confidence intervals rest on a small number of elegant ideas that link the geometry of the likelihood surface to probabilistic coverage guarantees. Rather than relying on a single point estimate and its estimated variance, these intervals exploit the entire shape of the log-likelihood function to determine how far from the maximum likelihood estimate the parameter can plausibly lie. The following foundational concepts are essential for understanding the construction and interpretation of these intervals.

Likelihood Function

Given observed data x, the likelihood function L(θ) = f(x | θ) treats the density (or mass) as a function of the parameter θ with data fixed. It quantifies how well each candidate value of θ is supported by the observed data.

Log-Likelihood Ratio Statistic

The deviance statistic D(θ) = 2[ℓ(θ̂) − ℓ(θ)] measures the 'distance' between a candidate parameter value θ and the MLE θ̂ on the log-likelihood scale. Wilks' theorem establishes that D(θ₀) converges in distribution to χ²₁ under H₀: θ = θ₀.

Inversion of the Likelihood Ratio Test

A 100(1 − α)% confidence set is formed by collecting all θ values that would not be rejected by a likelihood ratio test at level α. This is equivalent to the set {θ : D(θ) ≤ χ²₁,₁₋α}, producing an interval that inherits the optimality properties of the LR test.

Profile Likelihood

When the model has nuisance parameters ψ alongside the interest parameter θ, the profile likelihood is ℓₚ(θ) = max_ψ ℓ(θ, ψ). It 'concentrates out' the nuisance parameters by optimizing over them for each fixed θ, reducing the problem to a single-parameter setting.

Asymmetric Intervals

Unlike Wald intervals, likelihood-based CIs are naturally asymmetric when the log-likelihood is asymmetric. This property makes them more accurate for parameters near boundaries (e.g., variance components ≥ 0) and for small-sample settings where normality is a poor approximation.

✦ KEY TAKEAWAY

Think of the log-likelihood as a mountain whose peak is the MLE. A Wald interval assumes the mountain is a perfect parabola and measures its width at a fixed height. A likelihood-based interval actually walks along the mountain's contour at that height, capturing every ridge and asymmetry. When the mountain is lopsided—as it often is in practice—the contour-based interval is a far more honest map of the terrain than a symmetric parabolic approximation.

SECTION 3

Visual Explanation: The Likelihood Ratio Interval

The geometric intuition behind likelihood-based confidence intervals is best grasped through a diagram of the log-likelihood function. The following figure depicts a typical log-likelihood curve ℓ(θ), its maximum at the MLE θ̂, and the horizontal cutoff line that defines the endpoints of the confidence interval. The cutoff is placed at ℓ(θ̂) − c/2, where c = χ²₁,₁₋α. For a 95% interval, c ≈ 3.84, so the cutoff sits 1.92 units below the peak on the log-likelihood scale.

The cyan curve shows the log-likelihood ℓ(θ). The MLE θ̂ sits at the peak. The dashed pink line is the cutoff ℓ(θ̂) − c/2. The violet dots mark the lower and upper endpoints θ_L and θ_U where the curve crosses the cutoff. The amber segment along the θ-axis represents the 95% likelihood ratio confidence interval. Note how the interval is naturally asymmetric because the log-likelihood is steeper to the left of the MLE.

Several features of this diagram deserve emphasis. First, the interval endpoints are found by solving the equation ℓ(θ) = ℓ(θ̂) − c/2 for θ, which generally requires numerical root-finding. Second, the asymmetry of the interval reflects genuine asymmetry in the data's evidence about θ—information that a symmetric Wald interval would discard. Third, as the sample size grows and the log-likelihood approaches a parabola (by the central limit theorem), the likelihood-based interval converges to the Wald interval, confirming that the two approaches agree asymptotically but may differ substantially in finite samples.

SECTION 4

Mathematical Framework

The mathematical foundations of likelihood-based confidence intervals draw on the theory of likelihood ratio tests and the elegant asymptotic result known as Wilks' theorem. We begin with a single-parameter setting and then extend to the multiparameter case via the profile likelihood.

Single-Parameter Case

LOG-LIKELIHOOD RATIO (DEVIANCE) STATISTIC

D(θ₀) = 2 [ ℓ(θ̂) − ℓ(θ₀) ]

ℓ(θ) = log L(θ) is the log-likelihood; θ̂ is the MLE; θ₀ is the hypothesized value.

Under standard regularity conditions (the parameter space is open, the model is identifiable, and the Fisher information is finite and positive), Wilks' theorem guarantees that D(θ₀) converges in distribution to a χ² random variable with one degree of freedom when the true parameter is θ₀. The 100(1 − α)% confidence interval is therefore the set of θ values for which D(θ) does not exceed the critical value.

LIKELIHOOD RATIO CONFIDENCE INTERVAL

CI₁₋α = { θ : 2 [ ℓ(θ̂) − ℓ(θ) ] ≤ χ²₁,₁₋α }

For a 95% CI, χ²₁,₀.₉₅ = 3.841. The endpoints are found by solving ℓ(θ) = ℓ(θ̂) − 1.921.

Multiparameter Case: Profile Likelihood

Suppose the full parameter vector is (θ, ψ) where θ is the scalar parameter of interest and ψ is a (possibly vector-valued) nuisance parameter. Directly inverting a multivariate likelihood ratio test would yield a confidence region in the joint (θ, ψ) space, which is not what we want. The profile likelihood resolves this by maximizing the log-likelihood over ψ for each fixed value of θ.

PROFILE LOG-LIKELIHOOD

ℓₚ(θ) = max_ψ ℓ(θ, ψ) = ℓ(θ, ψ̂(θ))

ψ̂(θ) is the constrained MLE of ψ for each fixed θ. The profile log-likelihood is a function of θ alone.

PROFILE LIKELIHOOD RATIO CI

CI₁₋α = { θ : 2 [ ℓₚ(θ̂) − ℓₚ(θ) ] ≤ χ²₁,₁₋α }

Here θ̂ is the overall MLE and ℓₚ(θ̂) = ℓ(θ̂, ψ̂), the global maximum. The profile deviance is compared to the same χ²₁ cutoff.

⚠️ Regularity Conditions

Wilks' theorem requires that the true parameter lies in the interior of the parameter space, the model is correctly specified and identifiable, and the Fisher information matrix is positive definite. When parameters lie on the boundary (e.g., variance components equal to zero), the null distribution of D(θ₀) can be a mixture of chi-squared distributions, and care is needed.

SECTION 5

Profile Likelihood in Detail

The profile likelihood is best understood through its geometric interpretation. Consider a model with a two-dimensional parameter space (θ, ψ). The full log-likelihood ℓ(θ, ψ) defines a surface over this plane, with a single peak at the joint MLE (θ̂, ψ̂). To construct the profile likelihood for θ, we slice this surface with a plane perpendicular to the θ-axis at each value θ = θ₀, find the height of the highest point on the resulting cross-section (by optimizing over ψ), and record that height. The curve traced out by these maximal heights is the profile log-likelihood ℓₚ(θ).

Left panel: Contour ellipses of the joint log-likelihood ℓ(θ, ψ). The pink vertical slice at θ₀ is optimized over ψ to find ψ̂(θ₀), the amber dot. Right panel: The resulting profile log-likelihood ℓₚ(θ) as a function of θ alone. The green endpoints θ_L and θ_U are determined by the intersection with the cutoff line, yielding the profile likelihood CI.

A crucial property of the profile likelihood is that it inherits the asymptotic chi-squared calibration. Specifically, 2[ℓₚ(θ̂) − ℓₚ(θ₀)] is asymptotically χ²₁ when θ₀ is the true value, regardless of the dimension of the nuisance parameter ψ. This remarkable result means we use exactly the same cutoff (3.841 for a 95% CI) whether the model has one or one hundred nuisance parameters. However, the shape of the profile likelihood curve may differ substantially from the full marginal likelihood, and the constrained MLE ψ̂(θ) must be recomputed at every candidate value of θ, which is the primary computational cost of the procedure.

💡 Computational Tip

In practice, computing the profile likelihood CI requires a grid search or root-finding algorithm. A common strategy is to evaluate ℓₚ(θ) on a grid of θ values, interpolate the profile deviance curve, and use bisection or Brent's method to locate the two roots of 2[ℓₚ(θ̂) − ℓₚ(θ)] = χ²₁,₁₋α. Software packages such as R's confint() function for glm objects automate this process.

SECTION 6

Worked Example: Binomial Proportion

Consider a classic setting: we observe X = 3 successes in n = 20 independent Bernoulli trials and wish to construct a 95% likelihood ratio confidence interval for the success probability p. This small-sample scenario highlights the advantage of the likelihood-based approach over the Wald interval.

95% LR Confidence Interval for Binomial p

Step 1 — Write the Log-LikelihoodThe binomial log-likelihood (dropping the constant) is ℓ(p) = x log(p) + (n − x) log(1 − p) = 3 log(p) + 17 log(1 − p).

Step 2 — Find the MLESetting dℓ/dp = x/p − (n − x)/(1 − p) = 0 gives p̂ = x/n = 3/20 = 0.15.

p̂ = 0.15

Step 3 — Evaluate the Log-Likelihood at the MLEℓ(0.15) = 3 log(0.15) + 17 log(0.85) = 3(−1.8971) + 17(−0.1625) = −5.6913 + (−2.7631) = −8.4544.

ℓ(p̂) = −8.4544

Step 4 — Set Up the Deviance InequalityThe 95% CI consists of all p such that D(p) = 2[ℓ(0.15) − ℓ(p)] ≤ 3.841. Equivalently, ℓ(p) ≥ ℓ(0.15) − 3.841/2 = −8.4544 − 1.9207 = −10.3751.

Cutoff: ℓ(p) ≥ −10.3751

Step 5 — Solve for the Endpoints NumericallyWe need the two roots of 3 log(p) + 17 log(1 − p) = −10.3751. Using numerical root-finding (e.g., bisection), we obtain p_L ≈ 0.0428 and p_U ≈ 0.3386. Note the pronounced asymmetry: the MLE is 0.15, but the lower tail extends only 0.107 units below while the upper tail extends 0.189 units above.

95% LR CI: (0.0428, 0.3386)

Step 6 — Compare with the Wald IntervalThe Wald interval is p̂ ± 1.96 × √(p̂(1 − p̂)/n) = 0.15 ± 1.96 × √(0.15 × 0.85/20) = 0.15 ± 0.1565, giving (−0.0065, 0.3065). The Wald interval includes negative values, which are impossible for a probability. The likelihood ratio interval avoids this problem entirely because the log-likelihood naturally diverges to −∞ as p → 0⁺.

Wald CI: (−0.007, 0.307) — includes impossible values!

✦ WHY THE LR INTERVAL IS BETTER HERE

In this example, the Wald interval extends into negative territory, violating the parameter space constraint 0 ≤ p ≤ 1. The likelihood ratio interval respects this constraint automatically because the log-likelihood enforces it. This is analogous to navigating with a GPS that knows the roads (likelihood) versus following a straight-line compass bearing (Wald)—the GPS won't send you through a lake.

SECTION 7

Strengths, Limitations, and Comparisons

Likelihood-based confidence intervals offer substantial advantages over Wald intervals but come with their own trade-offs. Understanding these strengths and limitations is essential for deciding which method to use in a given analysis. The following table provides a systematic comparison across several criteria.

Comparison of Wald and Likelihood Ratio / Profile Likelihood Intervals

Criterion	Wald Interval	LR / Profile Likelihood Interval
Coverage accuracy	Can be poor in small samples or near boundaries; coverage often below nominal	Generally closer to nominal coverage in finite samples due to respecting likelihood curvature
Symmetry	Always symmetric around the MLE by construction	Naturally asymmetric, reflecting the true shape of the likelihood
Parameter constraints	May extend outside the parameter space (e.g., negative probabilities)	Automatically respects the parameter space because ℓ(θ) → −∞ at boundaries
Computation	Trivial: one evaluation of θ̂ and SE(θ̂)	Requires iterative optimization at each candidate θ; more expensive for complex models
Invariance to reparameterization	Not invariant: CI for log(θ) ≠ log of CI for θ	Fully invariant under monotone transformations (equivariant)
Nuisance parameters	Handled via plug-in estimates; underestimates uncertainty	Profile likelihood optimizes over nuisance parameters, properly accounting for their uncertainty

✦ WHEN TO PREFER EACH METHOD

In large samples with parameters far from boundaries, the Wald and LR intervals will be nearly identical, and the Wald interval's computational simplicity may be preferred. However, for small to moderate samples, for parameters near the edge of their domain (variances, proportions near 0 or 1, odds ratios), or when invariance to reparameterization is desired, the profile likelihood interval is the more reliable choice. Modern statistical software has largely eliminated the computational barrier, making LR intervals the default recommendation in many applied settings.

SECTION 8

Connection to Advanced Theory

The profile likelihood approach introduced here is a gateway to several more refined techniques in modern statistical inference. As models grow more complex—incorporating random effects, high-dimensional nuisance parameters, or non-standard asymptotics—the basic profile likelihood can be improved upon in important ways. The following table contrasts the introductory profile likelihood CI with some of its advanced extensions.

Profile Likelihood CI vs. Advanced Extensions

Feature	Basic Profile Likelihood CI	Advanced Extensions
Adjustment for nuisance parameter bias	None; uses raw profile likelihood	Modified profile likelihood (Barndorff-Nielsen, Cox & Reid) applies corrections for O(n⁻¹) bias in the profile score
Higher-order accuracy	First-order: O(n⁻¹/²) error in coverage	Bartlett correction and r* statistic achieve O(n⁻³/²) accuracy, substantially improving finite-sample coverage
Integrated likelihood	Not applicable; profile maximizes over ψ	Integrates over ψ with a prior or reference measure, producing a marginal likelihood for θ that may have better properties in some settings
Bayesian connection	Purely frequentist interpretation	With a flat or Jeffreys prior, Bayesian highest posterior density (HPD) intervals often coincide numerically with profile likelihood CIs, providing a dual interpretation
Boundary parameters	Standard χ²₁ calibration may fail	Self and Liang (1987), Stram and Lee (1994) derive mixture-of-chi-squared null distributions for variance components on the boundary

Looking forward, the profile likelihood framework also connects naturally to penalized likelihood methods used in regularized regression (LASSO, ridge) and to quasi-likelihood approaches for models where the full distribution is not specified. In each case, the core idea—reading inferential information from the curvature of an objective function rather than from an assumed sampling distribution—remains the unifying thread. Students proceeding to generalized estimating equations, mixed-effects models, or survival analysis will find the profile likelihood a recurring and indispensable tool.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain in your own words why a likelihood ratio confidence interval can be asymmetric about the MLE, whereas a Wald interval is always symmetric. Under what conditions would the two intervals be nearly identical?

PROBLEM 2 — BASIC CALCULATION

Suppose you observe X = 7 successes in n = 10 Bernoulli trials. The log-likelihood is ℓ(p) = 7 log(p) + 3 log(1 − p). (a) Find the MLE p̂. (b) Compute ℓ(p̂). (c) Determine the cutoff value for a 95% likelihood ratio CI. (d) Verify that p = 0.40 lies outside the 95% CI by evaluating D(0.40).

PROBLEM 3 — INTERMEDIATE

A Poisson model is fit to count data: Y ~ Poisson(λ), and you observe Y = 2. (a) Write the log-likelihood ℓ(λ) (ignoring the constant −log(y!)). (b) Find the MLE. (c) Set up the equation whose roots define the 90% likelihood ratio CI for λ. (d) Using trial values or a sketch, determine approximately whether the 90% CI is symmetric or asymmetric about the MLE.

PROBLEM 4 — APPLIED

In a logistic regression model for disease status (Y = 0 or 1) with a single predictor X, the log-likelihood is ℓ(β₀, β₁) = Σ[yᵢ(β₀ + β₁xᵢ) − log(1 + exp(β₀ + β₁xᵢ))]. Suppose the MLEs are β̂₀ = −1.2, β̂₁ = 0.8, and you wish to construct a 95% profile likelihood CI for the log-odds ratio β₁. Describe the algorithm you would use, specifying what must be optimized at each step and what equation determines the endpoints.

PROBLEM 5 — CRITICAL THINKING

The profile likelihood CI relies on Wilks' theorem for its asymptotic calibration. (a) Give two concrete examples of models or settings where Wilks' theorem does not hold. (b) For one of your examples, explain what goes wrong with the standard χ²₁ calibration and suggest a remedy. (c) Is the profile likelihood CI still a useful starting point even when regularity conditions fail? Argue for or against.

SUMMARY

Summary & Review

Likelihood-based confidence intervals are constructed by inverting the likelihood ratio test: the interval collects all parameter values θ for which the deviance statistic D(θ) = 2[ℓ(θ̂) − ℓ(θ)] does not exceed the appropriate chi-squared critical value (e.g., 3.841 for 95% confidence). Unlike Wald intervals, which assume a symmetric normal approximation, likelihood-based intervals naturally capture asymmetry in the log-likelihood, respect parameter space boundaries, and are invariant to reparameterization.

When the model includes nuisance parameters, the profile likelihood eliminates them by maximizing the log-likelihood over the nuisance parameters for each fixed value of the interest parameter. The resulting profile deviance retains the same asymptotic χ²₁ calibration via Wilks' theorem. These methods are especially valuable in small-sample settings, near boundary parameters, and in complex models such as logistic regression, Poisson regression, and mixed-effects models. Mastery of profile likelihood confidence intervals equips the practitioner with a robust, principled tool for interval estimation that goes well beyond the limitations of the standard Wald approach.