Method of Moments & MLE — Method of moments and maximum...

SECTION 1

Historical Context & Motivation

The problem of estimating unknown parameters from observed data has been central to statistics since its emergence as a formal discipline. In the late nineteenth and early twentieth centuries, practitioners relied largely on ad-hoc procedures—fitting lines by eye, choosing parameters to match tabulated frequencies, or appealing to least squares without a probabilistic model. The need for principled, general-purpose estimation strategies motivated two of the most important methodological contributions in statistical history: the method of moments and maximum likelihood estimation.

Karl Pearson sought a systematic technique to fit distributions to empirical data, while Ronald A. Fisher later argued that the likelihood function itself should serve as the foundation for inference. Their contrasting philosophies shaped the landscape of parametric estimation and continue to influence modern practice, from classical regression to Bayesian computation.

1894

Pearson's Method of Moments

Karl Pearson introduces the method of moments to fit his family of skew curves, equating sample moments to population moments to solve for unknown parameters.

1912

Fisher's Early Likelihood Ideas

Ronald A. Fisher, while still an undergraduate, publishes a short paper introducing the concept of maximum likelihood as an alternative criterion for parameter estimation, though the full theory would come later.

1922

Fisher Formalizes MLE

Fisher's landmark paper establishes the formal properties of MLE—consistency, efficiency, and asymptotic normality—placing it on rigorous theoretical footing and arguing for its superiority over moment-based estimators.

1948

Cramér–Rao Bound

Harald Cramér and C. R. Rao independently derive the information inequality, providing a universal lower bound on the variance of unbiased estimators and a benchmark against which MLE can be evaluated.

1970s–present

Computational Revolution

Numerical optimization and the EM algorithm make MLE tractable for complex, high-dimensional models. Generalized method of moments (GMM) extends Pearson's original idea to overidentified econometric settings.

The central question both methods address is deceptively simple: given a parametric family {f(x; θ) : θ ∈ Θ} and observed data X₁, X₂, …, Xₙ, how should we choose θ̂ so that the fitted model best represents the data-generating process? The method of moments answers by matching population and sample moments, while MLE answers by finding the parameter value that makes the observed data most probable. Understanding both approaches—their derivations, properties, and trade-offs—equips the practitioner with a versatile toolkit for statistical inference.

SECTION 2

Core Principles & Definitions

Both the method of moments and MLE belong to the broader class of point estimation procedures: they produce a single 'best guess' θ̂ for the unknown parameter vector θ. Despite sharing this goal, the two methods rest on fundamentally different principles. The method of moments exploits the algebraic relationship between parameters and population moments, whereas MLE exploits the probabilistic structure encoded in the likelihood function. The following cards distill the foundational ideas that underlie each approach.

Population vs. Sample Moments

The k-th population moment is μ'ₖ = E[Xᵏ], a function of θ. Its sample counterpart is m̂ₖ = (1/n) Σ Xᵢᵏ. Equating these yields the method of moments estimator.

Likelihood Function

The likelihood L(θ) = ∏ f(xᵢ; θ) treats the observed data as fixed and θ as variable. MLE maximizes this function—or equivalently, the log-likelihood ℓ(θ)—over the parameter space.

Score Function & Fisher Information

The score S(θ) = ∂ℓ/∂θ has mean zero under the true parameter. Its variance, I(θ) = E[S(θ)²], is the Fisher information, governing the precision achievable by any unbiased estimator.

Consistency & Efficiency

An estimator is consistent if θ̂ₙ → θ₀ in probability. It is efficient if its asymptotic variance attains the Cramér–Rao lower bound 1/[nI(θ₀)]. MLE is asymptotically efficient under regularity conditions; MoM typically is not.

Regularity Conditions

Both methods require regularity conditions to guarantee desirable properties: identifiability (distinct θ give distinct distributions), differentiability of the log-likelihood, and the existence of required moments. When these fail, alternative strategies may be needed.

✦ KEY TAKEAWAY

Think of parameter estimation like tuning a radio. The method of moments is like adjusting the dial until the station's average signal strength matches a known reference—quick and intuitive, but coarse. Maximum likelihood is like fine-tuning until the reception is the clearest possible—it uses all the information in the signal and typically achieves the sharpest result, but it requires more effort to compute.

SECTION 3

Visual Explanation — MoM vs. MLE

The diagram below contrasts the conceptual pipelines of the method of moments and maximum likelihood estimation. On the left branch, we see how sample moments are computed directly from the data and set equal to their population counterparts, yielding a system of equations whose solution is the MoM estimator. On the right branch, the data enter the likelihood function, which is maximized over the parameter space—often via differentiation—to produce the MLE. Both pipelines begin with the same data and parametric model, but they extract information through different mechanisms.

Both pipelines start from the same observed data and parametric model. The left branch (Method of Moments) equates sample moments to population moment expressions and solves algebraically. The right branch (MLE) constructs the likelihood, takes logarithms, and finds the maximizer via calculus or numerical methods.

Notice that the MoM pathway is purely algebraic once the moment equations are written down: there is no optimization problem to solve, which historically made it appealing before the advent of electronic computation. The MLE pathway, by contrast, requires maximizing a function that may not have a closed-form solution, but it compensates by using the full shape of the data-generating distribution rather than a finite number of summary statistics.

SECTION 4

Mathematical Framework

Method of Moments — Formal Setup

Suppose X₁, …, Xₙ are i.i.d. from a distribution with density (or pmf) f(x; θ₁, …, θₚ), where p parameters are unknown. The method of moments equates the first p population moments to their sample counterparts, producing a system of p equations in p unknowns.

MOM SYSTEM OF EQUATIONS

μ'ₖ(θ) = E[Xᵏ] = m̂ₖ = (1/n) Σᵢ₌₁ⁿ Xᵢᵏ , k = 1, 2, …, p

μ'ₖ(θ) is the k-th raw population moment (a known function of θ), and m̂ₖ is the k-th sample moment. Solving this system for θ₁, …, θₚ yields the method of moments estimator θ̂_MoM.

Central moments can be used instead of raw moments when the algebra is simpler. For instance, for a two-parameter family one commonly uses E[X] = X̄ and E[(X − μ)²] = (1/n) Σ (Xᵢ − X̄)². By the law of large numbers, m̂ₖ → μ'ₖ(θ₀) almost surely, so the MoM estimator is typically consistent provided the moment-to-parameter mapping is continuous and invertible.

Maximum Likelihood Estimation — Formal Setup

LIKELIHOOD FUNCTION

L(θ) = ∏ᵢ₌₁ⁿ f(Xᵢ; θ)

L(θ) is the likelihood function: the joint density of the sample evaluated at the observed data, viewed as a function of θ. It is not a probability distribution over θ.

LOG-LIKELIHOOD & SCORE EQUATION

ℓ(θ) = Σᵢ₌₁ⁿ log f(Xᵢ; θ) ; S(θ) = ∂ℓ/∂θ = 0

Taking the logarithm converts products to sums, simplifying differentiation. The score equation S(θ) = 0 is a necessary condition for an interior maximum. The solution θ̂_MLE is the maximum likelihood estimator provided the second derivative is negative (or the Hessian is negative definite in higher dimensions).

ASYMPTOTIC DISTRIBUTION OF THE MLE

√n (θ̂_MLE − θ₀) →ᵈ N(0, I(θ₀)⁻¹)

Under regularity conditions, the MLE is asymptotically normal with variance equal to the inverse Fisher information I(θ₀)⁻¹ per observation. This means the MLE is asymptotically efficient—it achieves the Cramér–Rao lower bound.

📐 Invariance Property of MLE

If θ̂ is the MLE of θ, then for any function g(θ), the MLE of g(θ) is g(θ̂). This invariance property does not generally hold for the method of moments. For example, if θ̂ is the MLE of the variance σ², then √θ̂ is automatically the MLE of the standard deviation σ.

SECTION 5

Detailed Comparison — Geometry of Likelihood

To build geometric intuition, consider a single-parameter exponential family. The log-likelihood ℓ(θ) is a concave curve whose peak identifies the MLE. The MoM estimator, by contrast, corresponds to a particular horizontal slice through a moment function. These two solutions coincide for some families (e.g., the normal distribution when parametrized by μ and σ²) but diverge for others (e.g., the gamma distribution). The diagram below illustrates a typical log-likelihood surface for a two-parameter model, showing the contours and the location of both estimators.

Elliptical contours of the log-likelihood ℓ(θ₁, θ₂) for a two-parameter model. The MLE sits at the peak (center of the innermost contour), while the MoM estimator lies nearby but generally not at the maximum. The dashed gap Δ represents the efficiency loss of MoM relative to MLE, which shrinks with sample size but may be non-negligible in finite samples.

The gap between θ̂_MoM and θ̂_MLE reflects the fact that moments capture only limited aspects of the distribution's shape. In the normal model, the first two moments fully characterize the distribution, so both estimators coincide. In skewed or heavy-tailed families, higher moments carry additional information that MoM ignores unless one explicitly matches more moments. The asymptotic relative efficiency (ARE) of MoM relative to MLE quantifies this loss: for a Gamma(α, β) model with small α, the ARE can drop well below 1, meaning MoM requires substantially more data to achieve the same precision.

Comparison of MoM and MLE across common distributions

Distribution	MoM = MLE?	ARE (MoM vs. MLE)
Normal(μ, σ²)	Yes (both coincide)	1.00
Exponential(λ)	Yes (both yield 1/X̄)	1.00
Gamma(α, β)	No	< 1, depends on α
Beta(a, b)	No	< 1, can be significantly less
Uniform(0, θ)	No (MLE = max Xᵢ)	Not directly comparable (non-regular)

SECTION 6

Worked Example — Gamma Distribution

Consider a random sample X₁, …, Xₙ drawn from a Gamma(α, β) distribution with density f(x; α, β) = [βᵅ / Γ(α)] x^(α−1) e^(−βx) for x > 0. Both α (shape) and β (rate) are unknown. We derive the MoM estimator and set up the MLE score equations.

Part A: Method of Moments for Gamma(α, β)

Step 1 — Population MomentsFor the Gamma(α, β) distribution, the first two population moments are: E[X] = α/β and E[X²] = α(α + 1)/β². From these, the population variance is Var(X) = α/β².

Step 2 — Sample MomentsCompute the sample mean m̂₁ = X̄ = (1/n) Σ Xᵢ and the sample variance S² = (1/n) Σ (Xᵢ − X̄)². We use the biased version (dividing by n) for consistency with the MoM framework, though dividing by (n − 1) would also be acceptable.

Step 3 — Equate and SolveSetting E[X] = X̄ gives α/β = X̄. Setting Var(X) = S² gives α/β² = S². Dividing the first equation by the second: (α/β) / (α/β²) = β = X̄ / S². Substituting back: α = X̄ × β = X̄² / S².

α̂_MoM = X̄² / S² , β̂_MoM = X̄ / S²

Part B: MLE for Gamma(α, β)

Step 1 — Write the Log-Likelihoodℓ(α, β) = Σ [α log β − log Γ(α) + (α − 1) log Xᵢ − β Xᵢ] = nα log β − n log Γ(α) + (α − 1) Σ log Xᵢ − β Σ Xᵢ.

Step 2 — Score Equation for β∂ℓ/∂β = nα/β − Σ Xᵢ = 0 ⟹ β̂ = α̂ / X̄. This shows β̂ is a function of α̂: once we find α̂, β̂ follows immediately.

β̂_MLE = α̂_MLE / X̄ (conditional on α̂)

Step 3 — Score Equation for α (Profile Likelihood)Substituting β̂ = α/X̄ into ℓ and differentiating with respect to α yields: log α − ψ(α) = log X̄ − (1/n) Σ log Xᵢ, where ψ(α) = Γ'(α)/Γ(α) is the digamma function. This equation has no closed-form solution and must be solved numerically, for example via Newton–Raphson.

Solve log α̂ − ψ(α̂) = log X̄ − (1/n) Σ log Xᵢ numerically for α̂_MLE

Step 4 — Numerical IllustrationSuppose n = 50, X̄ = 4.2, S² = 8.82, and (1/n) Σ log Xᵢ = 1.18. Then the MoM estimates are α̂_MoM = 4.2²/8.82 ≈ 2.00 and β̂_MoM = 4.2/8.82 ≈ 0.476. The MLE, obtained by solving log α − ψ(α) = log 4.2 − 1.18 ≈ 0.255 numerically, yields α̂_MLE ≈ 2.08 and β̂_MLE ≈ 2.08/4.2 ≈ 0.495.

MoM: (α̂, β̂) ≈ (2.00, 0.476) | MLE: (α̂, β̂) ≈ (2.08, 0.495)

This example highlights a common pattern: the MoM estimator is available in closed form and serves as an excellent starting value for the iterative numerical procedure required by MLE. In practice, many statistical software packages initialize MLE algorithms with MoM estimates.

SECTION 7

Strengths, Limitations & Trade-offs

Head-to-head comparison of MoM and MLE

Criterion	Method of Moments	Maximum Likelihood
Ease of computation	Often yields closed-form estimators; no optimization needed.	May require iterative numerical optimization (Newton–Raphson, EM).
Asymptotic efficiency	Generally NOT efficient; does not attain the Cramér–Rao bound.	Efficient under regularity conditions; attains the CRLB.
Invariance	Does not possess the invariance property in general.	Invariant: MLE of g(θ) is g(θ̂_MLE).
Finite-sample properties	May produce estimates outside the parameter space (e.g., negative variance).	Estimates always lie in the parameter space (if the likelihood is properly constrained).
Robustness	Sensitive to outliers through sample moments, especially higher-order ones.	Sensitive to model misspecification; relies on the assumed density.
Use as initial values	Frequently used to initialize MLE algorithms.	N/A — MLE is typically the final estimator.

⚖️ WHEN TO USE WHICH

In engineering terms, MoM is like a quick prototype—fast to build, good enough to validate a design concept, and invaluable as a starting point. MLE is the optimized production model—it extracts maximum performance from the available materials (data) but requires more sophisticated tooling (computation). In research, MLE is the default choice when regularity conditions hold, but MoM remains important in settings where the likelihood is intractable, such as certain latent variable models or when only summary statistics are available.

⚠️ Non-Regular Models

Be cautious with models where the support depends on θ, such as Uniform(0, θ). The MLE is θ̂ = X₍ₙ₎ (the sample maximum), which converges at rate n rather than √n, and standard asymptotic normality breaks down. The MoM estimator θ̂ = 2X̄ is √n-consistent but highly inefficient relative to X₍ₙ₎. Neither method's standard large-sample theory applies without modification.

SECTION 8

Connections to Advanced Theory

Both the method of moments and MLE serve as springboards to more sophisticated inference frameworks. Understanding their properties deeply prepares the student for advanced topics in estimation, testing, and model selection.

From classical estimation to advanced inference

Classical Method	Advanced Extension	Key Idea
Method of Moments	Generalized Method of Moments (GMM)	Handles overidentified models by minimizing a quadratic form in moment conditions; widely used in econometrics.
MLE	Quasi / Pseudo MLE	Maximizes a misspecified likelihood; consistent for certain parameters even when the full model is wrong (sandwich variance).
MLE	EM Algorithm	Iteratively maximizes the expected complete-data log-likelihood; essential for latent variable and mixture models.
MLE	Bayesian Posterior Mode (MAP)	Adding a prior π(θ) to the likelihood yields the posterior; the MAP estimator is the mode, reducing to MLE under a flat prior.
MLE	Likelihood Ratio Tests	The ratio of maximized likelihoods under H₀ and H₁ forms the basis of the LRT; Wilks' theorem gives the asymptotic χ² distribution.

The trinity of asymptotic test statistics—the likelihood ratio test, the Wald test, and the score (Rao) test—are all constructed from the MLE and the score function. Each offers a different computational trade-off: the Wald test requires only the unrestricted MLE, the score test requires only the restricted MLE (under H₀), and the likelihood ratio test requires both. Similarly, GMM generalizes MoM by allowing more moment conditions than parameters and choosing an optimal weighting matrix, yielding an efficient estimator analogous to MLE within the moment-based framework. These connections underscore the enduring importance of the foundational methods studied in this lesson.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why the MLE enjoys the invariance property (i.e., if θ̂ is the MLE of θ, then g(θ̂) is the MLE of g(θ)), while the method of moments estimator does not in general. Provide an intuitive argument grounded in the definition of each estimator.

PROBLEM 2 — BASIC CALCULATION

Let X₁, …, Xₙ ∼ Exponential(λ) with density f(x; λ) = λe^(−λx) for x > 0. Derive the method of moments estimator and the MLE for λ. Show that they coincide.

PROBLEM 3 — INTERMEDIATE

Suppose X₁, …, Xₙ ∼ Uniform(a, b) with −∞ < a < b < ∞. (a) Derive the MoM estimators for a and b. (b) Derive the MLE. (c) Comment on why the standard asymptotic theory for MLE does not apply here.

PROBLEM 4 — APPLIED

An insurance company models claim sizes using a Pareto distribution with density f(x; α, x₀) = α x₀ᵅ / x^(α+1) for x ≥ x₀ > 0. Suppose x₀ is known and a random sample of n = 200 claims has Σ log(Xᵢ/x₀) = 340. (a) Find the MLE of α. (b) Construct an approximate 95% confidence interval using the asymptotic normality of the MLE. (c) Find the MoM estimator of α using the first moment (assuming α > 1 so E[X] exists) and explain why MLE might be preferred here.

PROBLEM 5 — CRITICAL THINKING

Consider a mixture of two normals: f(x; μ₁, μ₂, σ², π) = π φ(x; μ₁, σ²) + (1 − π) φ(x; μ₂, σ²), where φ denotes the normal density and σ² is known. (a) Explain why direct MLE via differentiating the log-likelihood is problematic. (b) Sketch how the EM algorithm addresses this. (c) Describe how MoM could provide initial values for EM by matching the first three sample moments to their population counterparts. What are the limitations?

SUMMARY

Lesson Summary

The method of moments estimates parameters by equating sample moments to their population counterparts and solving the resulting system algebraically. It is computationally simple, consistent under mild conditions, and often provides excellent initial values for iterative algorithms. However, it is generally not asymptotically efficient and can yield estimates outside the parameter space. Maximum likelihood estimation maximizes the likelihood function L(θ) = ∏ f(xᵢ; θ), exploiting the full probabilistic structure of the model. Under regularity conditions, the MLE is consistent, asymptotically normal with variance equal to the inverse Fisher information, and asymptotically efficient—attaining the Cramér–Rao lower bound.

The two methods coincide for distributions whose parameters are simple functions of their first few moments (e.g., Normal, Exponential), but diverge for richer families like the Gamma or Beta. The MLE possesses the powerful invariance property and connects directly to the likelihood ratio test, the Wald test, and the score test. Extensions include the EM algorithm for latent variable models and generalized method of moments (GMM) for overidentified econometric settings. Together, MoM and MLE form the bedrock of parametric statistical inference.