Likelihood Manipulation — Manipulate likelihoods and log-likelihoods to derive estimators - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The problem of estimating unknown parameters from observed data is as old as probability theory itself. Before the formal advent of likelihood-based inference, statisticians relied on ad hoc methods — choosing estimators by intuition, symmetry arguments, or the method of moments. While these approaches often produced reasonable results, they lacked a unifying principle that could systematically identify the 'best' estimator for a given probabilistic model. The question that drove a century of research was deceptively simple: given a parametric family of distributions and a sample of data, how should one combine the observed values to infer the parameter that most plausibly generated the data?

1774 • Laplace's Inverse Probability

Pierre-Simon Laplace introduced the idea of reasoning backwards from observations to causes, laying groundwork for what would eventually become likelihood reasoning within a Bayesian framework.

1912 • Fisher's Undergraduate Insight

Ronald A. Fisher, still an undergraduate at Cambridge, proposed the concept of the likelihood function, distinguishing it sharply from probability and arguing that it captured all information the data provide about parameters.

1922 • Maximum Likelihood Formalized

Fisher published his landmark paper formalizing maximum likelihood estimation (MLE), proving consistency and asymptotic efficiency under regularity conditions and popularizing the log-likelihood as a computational tool.

1946 • Cramér–Rao Bound

Harald Cramér and C. R. Rao independently established the information inequality, connecting the curvature of the log-likelihood to the minimum variance achievable by any unbiased estimator.

1977 • EM Algorithm

Dempster, Laird, and Rubin formalized the Expectation–Maximization algorithm, extending likelihood manipulation to incomplete-data problems and dramatically broadening the practical reach of MLE.

The central gap that likelihood manipulation addresses is this: given a joint density or mass function f(x₁, …, xₙ | θ), how do we algebraically and analytically transform this expression — via products, logarithms, differentiation, and constraint handling — to extract the value of θ that is most consistent with the observed data? This lesson develops those techniques from first principles.

SECTION 2

Core Principles & Definitions

Likelihood manipulation rests on a small number of foundational ideas that, once internalized, make the derivation of estimators across virtually all parametric families a systematic exercise. The following principles form the conceptual scaffold upon which every MLE derivation is built.

The Likelihood Function

Given data x = (x₁, …, xₙ), the likelihood L(θ) = L(θ | x) is the joint density/mass function viewed as a function of θ with x fixed. It is NOT a probability over θ; it measures the plausibility of each θ value.

Log-Likelihood Monotonicity

Because the natural logarithm is strictly monotone increasing, maximizing L(θ) is equivalent to maximizing ℓ(θ) = ln L(θ). The log-likelihood converts products into sums, vastly simplifying differentiation.

Score Function & Score Equation

The score function S(θ) = ∂ℓ/∂θ equals zero at the MLE under regularity conditions. Setting S(θ̂) = 0 and solving for θ̂ is the standard route to the maximum likelihood estimator.

Sufficiency & Factorization

The Neyman–Fisher factorization theorem shows that the likelihood factors into a part depending on data only through T(x) and a part free of θ. This simplifies the likelihood before differentiation.

Fisher Information & Curvature

The negative expected value of the second derivative of the log-likelihood, I(θ) = −E[∂²ℓ/∂θ²], measures how sharply peaked the likelihood is. Greater curvature means more precise estimation.

✦

KEY TAKEAWAY

Think of the likelihood function as a landscape over parameter space: every point θ has an elevation L(θ). Your job is to find the summit. The log-transformation reshapes this landscape without moving the peak — it just replaces steep multiplicative ridges with gentle additive hills, making the calculus of finding the summit far more tractable. The score equation S(θ̂) = 0 is simply the condition that the slope at the summit is flat.

SECTION 3

Visual Explanation — Likelihood to Log-Likelihood

Left: the raw likelihood L(θ) formed as a product of individual densities — note how steep and numerically volatile the curve can be. Right: the log-likelihood ℓ(θ) = ln L(θ) converts products to sums, producing a smoother concave curve. The peak occurs at the same θ̂ in both plots; the score equation S(θ̂) = 0 marks where the tangent line is horizontal.

The diagram above captures the single most important insight in likelihood manipulation: the logarithmic transformation does not change where the maximum occurs, but it transforms an unwieldy product of n density values into a tractable sum. On the left panel, the likelihood curve is formed by multiplying together individual probability contributions, and for even moderate sample sizes these products can be astronomically small or numerically unstable. On the right, the log-likelihood ℓ(θ) = Σ ln f(xᵢ | θ) behaves like a well-conditioned quadratic near its peak, making differentiation straightforward and second-order Taylor approximations accurate. The score function S(θ) = dℓ/dθ is simply the slope of the right-hand curve, and setting it to zero locates the MLE θ̂.

SECTION 4

Mathematical Framework

We now formalize the algebraic pipeline that converts a parametric model into an estimator. Let X₁, X₂, …, Xₙ be an i.i.d. sample from a distribution with density or mass function f(x | θ), where θ ∈ Θ ⊆ ℝ (the multiparameter case generalizes naturally via gradient vectors and Hessian matrices).

LIKELIHOOD FUNCTION

L(θ) = ∏ᵢ₌₁ⁿ f(xᵢ | θ)

L(θ) is the joint density of the sample evaluated at the observed data, treated as a function of the parameter θ. Each factor f(xᵢ | θ) is the contribution of the i-th observation.

LOG-LIKELIHOOD FUNCTION

ℓ(θ) = ln L(θ) = Σᵢ₌₁ⁿ ln f(xᵢ | θ)

The product becomes a sum under the logarithm. This is the key algebraic step: sums are linear and term-by-term differentiable, enabling closed-form solutions in many families.

SCORE EQUATION (FIRST-ORDER CONDITION)

S(θ̂) = dℓ/dθ |_{θ=θ̂} = Σᵢ₌₁ⁿ ∂ ln f(xᵢ | θ) / ∂θ |_{θ=θ̂} = 0

S(θ) is the score function. Solving S(θ̂) = 0 for θ̂ yields the maximum likelihood estimator, provided the second derivative ∂²ℓ/∂θ² < 0 at θ̂ (confirming a maximum).

FISHER INFORMATION

I(θ) = −E[∂²ℓ/∂θ²] = E[S(θ)²] = Var(S(θ))

Fisher information quantifies the curvature of the log-likelihood at its peak. The Cramér–Rao lower bound states Var(θ̂) ≥ 1 / (nI(θ)) for any unbiased estimator θ̂, and the MLE achieves this bound asymptotically.

The derivation workflow is therefore: (1) write down L(θ) as a product, (2) take the natural log to obtain ℓ(θ), (3) differentiate with respect to θ, (4) set the derivative equal to zero, (5) solve for θ̂, and (6) verify the second-order condition. When the parameter space is restricted or the model involves multiple parameters, Lagrange multipliers or partial derivatives replace the univariate calculus, but the conceptual pipeline is identical.

SECTION 5

Detailed Techniques for Likelihood Manipulation

While the general pipeline of likelihood → log-likelihood → score equation → solve is conceptually simple, the algebraic details vary significantly across distributional families. Successful manipulation requires facility with several recurring techniques: exponential-family log-likelihood decomposition, kernel identification, use of indicator functions, reparametrization, and profile likelihood for nuisance parameters.

The six-step MLE derivation pipeline, from writing the joint density through verifying the second-order condition. The lower-left box lists frequently used log-algebra identities; the lower-right box summarizes numerical alternatives when a closed-form solution is unavailable.

A particularly powerful technique arises with exponential family distributions. Any density that can be written in the form f(x | θ) = h(x) exp[η(θ)·T(x) − A(θ)] has a log-likelihood that decomposes cleanly: ℓ(θ) = η(θ)·Σ T(xᵢ) − nA(θ) + Σ ln h(xᵢ). The last term is free of θ and drops out upon differentiation. Setting the derivative with respect to the natural parameter η equal to zero reveals that the MLE satisfies A′(η̂) = T̄, where T̄ is the sample average of the sufficient statistic. This elegant result covers the Normal, Exponential, Poisson, Binomial, Gamma, and many other distributions in one sweep.

💡

Kernel Trick

When manipulating likelihoods, always identify the kernel — the portion of the density that depends on θ — and discard all multiplicative constants (or additive constants after taking logs). This reduces clutter and focuses the algebra on the terms that matter for optimization. For instance, the normal density's

(2πσ²)^{−1/2}

factor contributes to the log-likelihood but only through −(n/2) ln(2π) − (n/2) ln σ²; the first term vanishes on differentiation with respect to μ.

SECTION 6

Worked Example — Normal Distribution MLE

Let X₁, X₂, …, Xₙ be i.i.d. Normal(μ, σ²) random variables with both μ and σ² unknown. We derive the MLEs for both parameters simultaneously, illustrating the full pipeline.

Deriving the MLE for μ and σ² in the Normal Model

1
Step 1 — Write the Likelihood

The joint density of the sample is L(μ, σ²) = ∏ᵢ₌₁ⁿ $(2πσ²)^{−1/2}$ exp[−(xᵢ − μ)² / (2σ²)]. Because the observations are independent, this product of n normal densities is the likelihood.
2
Step 2 — Take the Log-Likelihood

Applying the natural logarithm, we convert the product into a sum: ℓ(μ, σ²) = −(n/2) ln(2π) − (n/2) ln(σ²) − (1/(2σ²)) Σᵢ₌₁ⁿ (xᵢ − μ)². Notice how the product of exponentials became a single sum in the exponent, and the product of $(2πσ²)^{−1/2}$ terms became a multiplicative factor n/2 via ln.
3
Step 3 — Differentiate with Respect to μ

Taking the partial derivative: ∂ℓ/∂μ = (1/σ²) Σᵢ₌₁ⁿ (xᵢ − μ). The first two terms of ℓ do not involve μ and vanish. The chain rule applied to −(xᵢ − μ)² gives 2(xᵢ − μ), and the factor 1/(2σ²) cancels the 2.

∂ℓ/∂μ = (1/σ²) Σᵢ (xᵢ − μ)
4
Step 4 — Set Score Equal to Zero and Solve for μ̂

Setting ∂ℓ/∂μ = 0: (1/σ²) Σᵢ (xᵢ − μ) = 0. Since σ² > 0, we can multiply both sides by σ² to get Σᵢ (xᵢ − μ) = 0, which gives Σᵢ xᵢ − nμ = 0, so μ̂ = (1/n) Σᵢ xᵢ = x̄.

μ̂ = x̄ (the sample mean)
5
Step 5 — Differentiate with Respect to σ² and Solve

Taking the partial derivative with respect to σ² (treating σ² as a single variable): ∂ℓ/∂(σ²) = −n/(2σ²) + (1/(2σ⁴)) Σᵢ (xᵢ − μ)². Setting this to zero and solving: n/(2σ²) = (1/(2σ⁴)) Σᵢ (xᵢ − μ)², so nσ² = Σᵢ (xᵢ − μ)², giving σ̂² = (1/n) Σᵢ (xᵢ − μ̂)².

σ̂² = (1/n) Σᵢ (xᵢ − x̄)² (the biased sample variance)
6
Step 6 — Verify Second-Order Conditions

The Hessian matrix of ℓ with respect to (μ, σ²) is negative definite at (μ̂, σ̂²). Specifically, ∂²ℓ/∂μ² = −n/σ² < 0 and the determinant of the Hessian is positive, confirming a global maximum. Note that the MLE of σ² divides by n rather than n − 1, so it is biased downward; however, it is consistent and asymptotically efficient.

SECTION 7

Strengths, Limitations & Comparisons

Property	MLE (Likelihood-Based)	Method of Moments
Derivation technique	Differentiate log-likelihood; solve score equation	Equate population moments to sample moments; solve algebraic system
Asymptotic efficiency	Achieves the Cramér–Rao bound; asymptotically most efficient	Generally less efficient; can be substantially worse
Consistency	Consistent under regularity conditions	Consistent (generally)
Closed-form availability	Often requires numerical optimization (Gamma shape, logistic regression)	Usually yields closed-form solutions
Invariance	Equivariant: MLE of g(θ) is g(θ̂) for any function g	Not generally equivariant
Finite-sample bias	Can be biased in finite samples (e.g., σ̂² for normal)	May or may not be unbiased

Comparison of MLE with Method of Moments estimation

✦

KEY TAKEAWAY

Maximum likelihood estimation is the Swiss army knife of parametric inference. It sacrifices simplicity of derivation (compared to method of moments) in exchange for optimality properties — consistency, asymptotic normality, asymptotic efficiency, and invariance under reparametrization. When the score equation admits a closed form, MLE is the clear winner. When it does not, numerical methods like Newton–Raphson or the EM algorithm extend its reach to essentially all parametric models.

SECTION 8

Connection to Advanced Theory

The techniques of likelihood manipulation extend far beyond point estimation. The same log-likelihood that yields the MLE also underpins a rich family of inferential procedures: likelihood ratio tests, Wald tests, and score (Lagrange multiplier) tests — collectively known as the 'holy trinity' of large-sample testing. Each manipulates the log-likelihood or its derivatives in slightly different ways, yet all three converge to the same chi-squared distribution under the null hypothesis as n → ∞.

Concept in This Lesson	Advanced Extension	Key Idea
Score equation S(θ̂) = 0	Estimating equations (M-estimation)	Replace ∂ ln f/∂θ with a general ψ-function; extends MLE to robust and semiparametric settings
Fisher information I(θ)	Observed & expected information matrices	In multiparameter models, inversion of the information matrix yields asymptotic covariance of the MLE vector
Log-likelihood maximization	Bayesian posterior mode (MAP)	Adding ln π(θ) (log-prior) to ℓ(θ) and maximizing gives the MAP estimator — MLE is MAP with a flat prior
Profile likelihood	Restricted/penalized likelihood (REML)	Eliminates nuisance parameters by integrating rather than maximizing over them; used extensively in mixed models

From basic likelihood manipulation to advanced theory

Looking forward, the machinery of likelihood manipulation is not an isolated skill but a gateway to virtually all of modern parametric and semiparametric statistics. Mastery of the algebraic transformations covered in this lesson — products to sums, kernel extraction, score equations, information calculations — provides the technical vocabulary needed for advanced courses in generalized linear models, survival analysis, longitudinal data analysis, and Bayesian computation.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why maximizing the log-likelihood ℓ(θ) = ln L(θ) yields the same estimator as maximizing the likelihood L(θ) itself. Under what circumstances might one prefer to work directly with L(θ) rather than ℓ(θ)?

Show answer

The natural logarithm is a strictly monotone increasing function: if a < b then ln a < ln b. Therefore, any θ* that maximizes L(θ) also maximizes ln L(θ), and vice versa. The argmax is preserved. One might prefer working with L(θ) directly in very small discrete problems (e.g., n = 1 or 2 with a finite parameter space) where enumerating L(θ) at each θ is straightforward, or when the likelihood has a simple product form that can be maximized by inspection — for instance, maximizing the likelihood of a Uniform(0, θ) distribution, where L(θ) = θ⁻ⁿ for θ ≥ x₍ₙ₎ is trivially maximized at θ̂ = x₍ₙ₎ without requiring calculus.

Question 2: BASIC CALCULATION

Let X₁, …, Xₙ be i.i.d. Exponential(λ) with density f(x | λ) = λ

e^{−λx}

for x > 0. Derive the MLE of λ.

Show answer

The likelihood is L(λ) = ∏ᵢ λ e^{−λxᵢ} = λⁿ exp(−λ Σxᵢ). The log-likelihood is ℓ(λ) = n ln λ − λ Σxᵢ. Differentiating: dℓ/dλ = n/λ − Σxᵢ. Setting this equal to zero gives n/λ̂ = Σxᵢ, so λ̂ = n / Σxᵢ = 1/x̄. The second derivative is −n/λ² < 0, confirming a maximum. Thus the MLE of the rate parameter is the reciprocal of the sample mean, which is intuitive since E[X] = 1/λ.

Question 3: INTERMEDIATE

Let X₁, …, Xₙ be i.i.d. Poisson(λ). (a) Derive the MLE of λ. (b) Using the invariance property of MLEs, find the MLE of P(X = 0) =

e^{−λ}

Show answer

(a) The PMF is f(x | λ) =

e^{−λ}

λˣ / x!, so L(λ) =

e^{−nλ}

λ^{Σxᵢ} / ∏(xᵢ!). Taking the log: ℓ(λ) = −nλ + (Σxᵢ) ln λ − Σ ln(xᵢ!). Differentiating: dℓ/dλ = −n + Σxᵢ / λ. Setting to zero: λ̂ = Σxᵢ / n = x̄. The second derivative is −Σxᵢ/λ² < 0 (assuming at least one xᵢ > 0), confirming a maximum. (b) By the invariance property, the MLE of g(λ) =

e^{−λ}

is g(λ̂) = e^{−λ̂} = e^{−x̄}. This illustrates how the equivariance of MLEs allows immediate estimation of any smooth function of the parameter without re-deriving the likelihood.

Question 4: APPLIED

A reliability engineer records failure times (in hours) of 6 identical components: 120, 450, 310, 280, 590, 200. Assuming an Exponential failure time model, (a) find the MLE of the failure rate λ, (b) compute the MLE of the mean time to failure (MTTF = 1/λ), and (c) determine the Fisher information I(λ) and construct an approximate 95% confidence interval for λ.

Show answer

(a) Σxᵢ = 120 + 450 + 310 + 280 + 590 + 200 = 1950, n = 6, so λ̂ = 6/1950 ≈ 0.003077 failures per hour. (b) By invariance, the MLE of MTTF = 1/λ is 1/λ̂ = 1950/6 = 325 hours, which equals x̄. (c) For the Exponential, −∂²ℓ/∂λ² = n/λ², so the Fisher information is I(λ) = 1/λ². For n observations, the total information is nI(λ) = n/λ². At λ̂, the estimated asymptotic variance is λ̂²/n = (0.003077)²/6 ≈ 1.578 × 10⁻⁶. The standard error is √(1.578 × 10⁻⁶) ≈ 0.001256. A 95% CI is λ̂ ± 1.96 × SE ≈ 0.003077 ± 0.002462, yielding approximately (0.000615, 0.005539). This wide interval reflects the small sample size.

Question 5: CRITICAL THINKING

Let X₁, …, Xₙ be i.i.d. Uniform(0, θ). Show that the MLE of θ is θ̂ = X₍ₙ₎ = max(X₁, …, Xₙ). Explain why the standard regularity conditions for MLE fail in this case, and discuss the consequences for the asymptotic distribution of θ̂.

Show answer

The likelihood is L(θ) = θ⁻ⁿ · ∏ᵢ 𝟙(0 ≤ xᵢ ≤ θ) = θ⁻ⁿ for θ ≥ x₍ₙ₎ and 0 otherwise. Since θ⁻ⁿ is strictly decreasing in θ for θ > 0, L(θ) is maximized at the smallest permissible θ, namely θ̂ = x₍ₙ₎. The score equation approach fails here because the support of the distribution depends on θ (the indicator function's boundary moves with θ), violating the regularity condition that the support be independent of the parameter. Consequently, the likelihood is discontinuous (it jumps to zero at θ = x₍ₙ₎), and the standard result √n(θ̂ − θ) →ᵈ N(0, 1/I(θ)) does not apply. Instead, n(θ − θ̂) converges to an Exponential distribution, illustrating a faster-than-√n rate of convergence — a phenomenon unique to boundary-estimation problems. This example underscores the importance of verifying regularity conditions before applying standard asymptotic MLE theory.

SUMMARY

Lesson Summary

Summary

This lesson developed the full toolkit of likelihood manipulation for deriving maximum likelihood estimators. We began with the likelihood function L(θ) = ∏ f(xᵢ | θ) — the joint density viewed as a function of the parameter — and showed how the logarithmic transformation converts products into sums, yielding the log-likelihood ℓ(θ) = Σ ln f(xᵢ | θ). Differentiation produces the score function S(θ), and solving S(θ̂) = 0 identifies the MLE, which achieves the Cramér–Rao lower bound asymptotically.

Key techniques include kernel identification (dropping terms free of θ), exponential family decomposition (leveraging sufficient statistics), and invariance of the MLE under reparametrization. When closed-form solutions are unavailable, Newton–Raphson and the EM algorithm extend the reach of likelihood-based inference. These manipulation skills form the algebraic backbone of modern parametric statistics, connecting directly to Fisher information, hypothesis testing, and Bayesian posterior computation.