Taylor Expansions for Approximations — Use Taylor expansions for delta method and approximations - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The idea of approximating a complicated function by a simpler polynomial expression has roots stretching back to the earliest days of calculus. Brook Taylor published his seminal result in 1715, showing that a smooth function could be represented as an infinite sum of terms involving its derivatives evaluated at a single point. While Taylor's original motivation was purely analytical, statisticians in the twentieth century recognized that these polynomial expansions provided an indispensable tool for deriving the distributional properties of nonlinear transformations of random variables. The delta method, which rests entirely on a first-order Taylor expansion, became the standard technique for obtaining approximate variances and asymptotic distributions of estimators that are smooth functions of other estimators whose limiting behavior is already known.

1715 • Taylor's Theorem Published

Brook Taylor publishes Methodus Incrementorum Directa et Inversa, establishing the representation of functions as power series centered at a point, laying the algebraic foundation for all subsequent approximation methods.

1812 • Laplace's Asymptotic Methods

Pierre-Simon Laplace uses Taylor-type expansions in his Théorie analytique des probabilités to approximate integrals and develop asymptotic methods that foreshadow the delta method in probability and statistics.

1938 • Cramér Formalizes Asymptotics

Harald Cramér's Random Variables and Probability Distributions rigorously connects Taylor expansions to the asymptotic normality of transformed estimators, giving the delta method a formal probabilistic underpinning.

1947 • Doob's Stochastic Framework

J.L. Doob's work on stochastic processes systematizes how linearization via Taylor expansion transfers convergence-in-distribution results through smooth mappings, anchoring the delta method as a core tool of mathematical statistics.

1980s–present • Modern Extensions

Second-order delta methods, multivariate extensions, and functional delta methods extend the classical framework to handle vector-valued parameters, non-smooth functions, and infinite-dimensional estimators such as empirical processes.

The central question that the delta method addresses is both elegant and practical: if we know that an estimator θ̂ is asymptotically normal around a parameter θ, what can we say about the distribution of g(θ̂) for a smooth function g? Taylor expansion answers this by linearizing g around θ, converting a potentially complex nonlinear problem into a simple linear one whose distributional properties follow immediately from those of the original estimator.

SECTION 2

Core Principles & Definitions

Before we can apply Taylor expansions in a statistical setting, we need to establish the analytical and probabilistic building blocks that make the approximation both valid and useful. The interplay between the smoothness of the transformation, the rate of convergence of the estimator, and the order of the expansion determines the quality and applicability of the resulting approximation.

Taylor's Theorem

If g is k-times continuously differentiable near a point θ, then g(x) = g(θ) + g′(θ)(x − θ) + g″(θ)(x − θ)²/2! + ⋯ + Rₖ, where the remainder Rₖ is controlled by the (k+1)-th derivative. The first-order expansion g(x) ≈ g(θ) + g′(θ)(x − θ) is the workhorse of the delta method.

Slutsky's Theorem

If Xₙ converges in distribution to X and Yₙ converges in probability to a constant c, then Xₙ + Yₙ →ᵈ X + c and XₙYₙ →ᵈ cX. This theorem justifies dropping higher-order remainder terms in Taylor expansions applied to sequences of random variables.

Continuous Mapping Theorem

If Xₙ →ᵈ X and g is continuous, then g(Xₙ) →ᵈ g(X). While powerful, it does not directly yield the asymptotic variance of g(Xₙ). The delta method complements it by providing exactly this variance through linearization.

Asymptotic Normality

An estimator θ̂ₙ is asymptotically normal if √n(θ̂ₙ − θ) →ᵈ N(0, σ²). This is the standard starting assumption for the delta method. The central limit theorem guarantees this property for many common estimators including sample means and MLEs.

Delta Method (First-Order)

If √n(θ̂ₙ − θ) →ᵈ N(0, σ²) and g is differentiable at θ with g′(θ) ≠ 0, then √n(g(θ̂ₙ) − g(θ)) →ᵈ N(0, σ²[g′(θ)]²). This is the principal result: the asymptotic variance of g(θ̂ₙ) is approximately σ²[g′(θ)]²/n.

✦

KEY TAKEAWAY

Think of the delta method as using a tangent-line approximation from calculus, but applied to the distribution of a random variable rather than a deterministic function. Just as the tangent line captures the local behavior of a curve near a point, the first-order Taylor expansion captures how the variability of θ̂ propagates through g, stretching or compressing the spread by the factor g′(θ). If the derivative is large, small fluctuations in θ̂ get amplified; if the derivative is small, they get dampened.

SECTION 3

Visual Explanation — Linearization and Variance Propagation

The geometric intuition behind the delta method is straightforward. Consider a smooth function g applied to an estimator θ̂ that concentrates around θ. The distribution of θ̂ occupies a small neighborhood of θ, and within that neighborhood, the function g is well approximated by its tangent line at θ. The tangent line maps the spread of θ̂ into the spread of g(θ̂), scaling the standard deviation by |g′(θ)|. The following diagram illustrates this linearization process.

The pink curve shows the nonlinear function g(x). The dashed cyan line is the tangent at (θ, g(θ)). The purple bell at the bottom represents the distribution of θ̂ centered at θ with standard deviation σ/√n. The cyan bell on the left axis shows the resulting distribution of g(θ̂), whose standard deviation is scaled by |g′(θ)|.

Notice in the diagram how the tangent line acts as a linear map between the two distributions. The spread of the input distribution along the horizontal axis is multiplied by the absolute value of the slope g′(θ) to produce the spread of the output distribution along the vertical axis. When g′(θ) is large, a small uncertainty in θ̂ amplifies into a large uncertainty in g(θ̂); when g′(θ) is close to zero, the output distribution compresses. This is precisely the variance propagation interpretation of the delta method.

SECTION 4

Mathematical Framework — Deriving the Delta Method

We now present the formal derivation of the univariate delta method, followed by its multivariate generalization. The argument relies on a first-order Taylor expansion combined with Slutsky's theorem to handle the remainder term.

Univariate Delta Method

ASSUMPTION

√n (θ̂ₙ − θ) →ᵈ N(0, σ²)

θ̂ₙ is an estimator of θ based on n observations; σ² is the asymptotic variance.

Let g be a function that is continuously differentiable in a neighborhood of θ, with g′(θ) ≠ 0. By Taylor's theorem, we expand g(θ̂ₙ) around θ to obtain g(θ̂ₙ) = g(θ) + g′(θ)(θ̂ₙ − θ) + R₂, where R₂ = g″(θ*)(θ̂ₙ − θ)²/2 for some θ* between θ̂ₙ and θ. Subtracting g(θ) from both sides and multiplying by √n gives √n(g(θ̂ₙ) − g(θ)) = g′(θ) · √n(θ̂ₙ − θ) + √n · R₂. Now, since θ̂ₙ →ᵖ θ and √n(θ̂ₙ − θ) = Oₚ(1), the remainder satisfies √n · R₂ = Oₚ(1) · oₚ(1) = oₚ(1). By Slutsky's theorem, the remainder vanishes in the limit.

UNIVARIATE DELTA METHOD RESULT

√n (g(θ̂ₙ) − g(θ)) →ᵈ N(0, σ² [g′(θ)]²)

The asymptotic variance of g(θ̂ₙ) is σ²[g′(θ)]²/n. This is often written as Var(g(θ̂ₙ)) ≈ [g′(θ)]² · Var(θ̂ₙ).

Multivariate Delta Method

When the estimator is a vector θ̂ₙ ∈ ℝᵏ with √n(θ̂ₙ − θ) →ᵈ N(0, Σ), and g: ℝᵏ → ℝ is differentiable at θ with gradient ∇g(θ) ≠ 0, the first-order Taylor expansion yields √n(g(θ̂ₙ) − g(θ)) ≈ ∇g(θ)ᵀ · √n(θ̂ₙ − θ). Since a linear transformation of a multivariate normal is normal, we obtain the multivariate delta method result.

MULTIVARIATE DELTA METHOD

√n (g(θ̂ₙ) − g(θ)) →ᵈ N(0, ∇g(θ)ᵀ Σ ∇g(θ))

∇g(θ) is the k × 1 gradient vector of g evaluated at θ; Σ is the k × k asymptotic covariance matrix. For a vector-valued function g: ℝᵏ → ℝᵐ, replace ∇g with the m × k Jacobian matrix J, yielding covariance J Σ Jᵀ.

Second-Order Delta Method

SECOND-ORDER EXPANSION (g′(θ) = 0 CASE)

n (g(θ̂ₙ) − g(θ)) →ᵈ ½ g″(θ) σ² χ²₁

When g′(θ) = 0 but g″(θ) ≠ 0, the first-order term vanishes. The leading term is quadratic: √n(θ̂ₙ − θ) →ᵈ N(0, σ²), so [√n(θ̂ₙ − θ)]² →ᵈ σ²χ²₁. Thus n(g(θ̂ₙ) − g(θ)) →ᵈ ½g″(θ)σ²χ²₁.

SECTION 5

Key Applications in Statistics

Taylor expansions and the delta method appear throughout statistics whenever we need the approximate distribution of a transformed estimator. Below we catalog the most common applications, illustrating how the same fundamental technique adapts to various settings.

Flowchart showing how the asymptotic normality assumption feeds into a Taylor expansion, which then branches into three major application families: variance stabilization (cyan), confidence interval construction (pink), and MLE standard error computation (emerald). Each branch lists common examples encountered in practice.

Variance-Stabilizing Transformations

A particularly elegant use of the delta method is to find a transformation g such that the asymptotic variance of g(θ̂ₙ) does not depend on θ. If Var(θ̂ₙ) ≈ v(θ)/n, we want [g′(θ)]² v(θ) = c for some constant c. Solving this ODE yields g(θ) = ∫ [v(θ)]^−1/2 dθ. For Poisson data where v(λ) = λ, we get g(λ) = 2√λ. For binomial proportions where v(p) = p(1 − p), we get g(p) = arcsin(√p). For the sample correlation coefficient where v(ρ) = (1 − ρ²)², Fisher's z-transformation g(r) = ½ ln((1+r)/(1−r)) produces an asymptotic variance of 1/(n−3) that is nearly independent of ρ.

Confidence Intervals for Nonlinear Functions of Parameters

In regression and survival analysis, quantities of interest are often nonlinear functions of model parameters. For instance, in logistic regression the odds ratio is exp(β), and its standard error is obtained by applying the delta method with g(β) = exp(β), giving SE(exp(β̂)) ≈ exp(β̂) · SE(β̂). Similarly, the ratio of two means μ₁/μ₂ in a bivariate setting requires the multivariate delta method with the gradient evaluated at the joint MLE. These standard errors then feed directly into Wald-type confidence intervals.

SECTION 6

Worked Example — Asymptotic Distribution of log(X̄) for Poisson Data

Suppose X₁, X₂, …, Xₙ are i.i.d. Poisson(λ) random variables with λ > 0. We observe the sample mean X̄ₙ and wish to determine the asymptotic distribution of g(X̄ₙ) = log(X̄ₙ), which is a natural parameter in the Poisson GLM context.

Delta Method for log(X̄) with Poisson Data

1
Step 1 — Establish Asymptotic Normality of X̄

By the central limit theorem, √n(X̄ₙ − λ) →ᵈ N(0, λ), since E[Xᵢ] = λ and Var(Xᵢ) = λ for the Poisson distribution. Thus σ² = λ.

√n(X̄ₙ − λ) →ᵈ N(0, λ)
2
Step 2 — Identify the Transformation and Its Derivative

We set g(x) = log(x), the natural logarithm. Its derivative is g′(x) = 1/x, so g′(λ) = 1/λ. Since λ > 0, we have g′(λ) ≠ 0, satisfying the regularity condition for the first-order delta method.

g′(λ) = 1/λ ≠ 0
3
Step 3 — Apply the Delta Method Formula

By the delta method, √n(g(X̄ₙ) − g(λ)) →ᵈ N(0, σ²[g′(λ)]²). Substituting σ² = λ and g′(λ) = 1/λ yields the asymptotic variance σ²[g′(λ)]² = λ · (1/λ)² = 1/λ.

√n(log(X̄ₙ) − log(λ)) →ᵈ N(0, 1/λ)
4
Step 4 — Express the Asymptotic Distribution

We can equivalently write log(X̄ₙ) is approximately N(log(λ), 1/(nλ)). This gives us the approximate standard error SE(log(X̄ₙ)) ≈ 1/√(nλ). In practice, we replace λ by its consistent estimate X̄ₙ.

log(X̄ₙ) ≈̇ N(log(λ), 1/(nλ))
5
Step 5 — Construct an Approximate 95% Confidence Interval

A 95% confidence interval for log(λ) is log(X̄ₙ) ± 1.96/√(nX̄ₙ). For example, if n = 100 and X̄₁₀₀ = 4.2, we compute SE ≈ 1/√(100 × 4.2) = 1/√420 ≈ 0.0488, so the interval is log(4.2) ± 1.96 × 0.0488 = 1.435 ± 0.096 = (1.339, 1.531). Exponentiating gives a CI for λ: (exp(1.339), exp(1.531)) = (3.816, 4.623).

95% CI for log(λ): (1.339, 1.531); for λ: (3.816, 4.623)

SECTION 7

Strengths, Limitations, and Comparisons

The delta method is remarkably versatile, but like all approximation techniques it has boundaries of applicability. Understanding where it excels and where it fails is crucial for choosing the right analytical tool in a given problem.

Aspect	Strengths	Limitations
Simplicity	Requires only the derivative of g and the asymptotic variance of the original estimator—no simulation or resampling needed.	May be inaccurate for small samples where higher-order terms are non-negligible.
Generality	Applies to any smooth function g and any asymptotically normal estimator, including MLEs, method-of-moments, and U-statistics.	Fails when g is not differentiable at θ (e.g., absolute value at zero) or when g′(θ) = 0 (requires second-order treatment).
Precision	Often provides excellent approximations when n is moderately large and g is nearly linear in the neighborhood of θ.	Highly curved g (large \|g″\|) or slowly converging θ̂ₙ can make the linearization poor even for moderate n.
Alternatives	Analytic closed-form result, no computational cost. Serves as a quick sanity check even when bootstrap is also used.	Bootstrap and parametric simulation can outperform it in small samples or with non-smooth transformations.

Strengths and limitations of the delta method as an approximation tool.

⚡

WHEN TO USE THE DELTA METHOD

The delta method is your first line of attack whenever you need the approximate distribution of a smooth function of an asymptotically normal estimator. Think of it as the analytical equivalent of linearizing a control system around its operating point in engineering: it works brilliantly near the equilibrium, but for large perturbations (small n, highly nonlinear g, or g′(θ) = 0), you need to either go to second order or switch to a computational method like the bootstrap.

SECTION 8

Connections to Advanced Theory

The delta method is the simplest member of a family of asymptotic techniques rooted in Taylor expansion. More advanced variants handle increasingly complex settings, from non-smooth functions to infinite-dimensional parameters. Understanding these extensions reveals the delta method's place in the broader landscape of asymptotic statistics.

Method	Setting	Key Idea
First-Order Delta Method	g differentiable, g′(θ) ≠ 0, √n-consistent estimator	Linearize g at θ. Result is asymptotically normal with variance [g′(θ)]²σ²/n.
Second-Order Delta Method	g′(θ) = 0 but g″(θ) ≠ 0	Quadratic approximation. Result converges at rate n (not √n) to a scaled chi-squared distribution.
Multivariate Delta Method	Vector estimator θ̂ ∈ ℝᵏ, scalar or vector g	Replace derivative with gradient or Jacobian. Asymptotic covariance is JΣJᵀ.
Functional Delta Method	Infinite-dimensional parameter (e.g., CDFs, quantile functions)	Replace derivative with Hadamard derivative. Used for bootstrap consistency proofs, Kaplan-Meier, and empirical processes.
Higher-Order Asymptotics (Edgeworth)	Need O(n⁻¹) correction to CLT	Retain higher Taylor terms to obtain skewness and kurtosis corrections, yielding Edgeworth expansions for refined coverage.

Comparison of Taylor-expansion-based asymptotic methods from simple to advanced.

The functional delta method deserves special attention as the natural generalization of the classical delta method to infinite-dimensional settings. In this framework, the estimator is an element of a function space (such as the empirical distribution function F̂ₙ), and the map of interest is a functional φ (such as the quantile functional). The Hadamard derivative of φ at F plays the same role as g′(θ) in the finite-dimensional case, and the resulting limit distribution is typically a Gaussian process rather than a finite-dimensional normal. This theory, developed by Gill, van der Vaart, and others, is the backbone of modern semiparametric and nonparametric inference.

ℹ

Connection to Maximum Likelihood

The delta method is implicitly used every time standard errors of nonlinear functions of MLE parameters are reported. If θ̂ is the MLE with asymptotic covariance I(θ)⁻¹/n (where I is the Fisher information), then SE(g(θ̂)) ≈ √(∇g(θ̂)ᵀ I(θ̂)⁻¹ ∇g(θ̂) / n). Software packages like R's deltamethod() in the msm package and Stata's nlcom automate this computation.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain intuitively why the delta method requires g′(θ) ≠ 0. What goes wrong with the first-order approximation when g′(θ) = 0, and what is the remedy?

Show answer

When g′(θ) = 0, the first-order Taylor expansion g(θ̂) ≈ g(θ) + g′(θ)(θ̂ − θ) reduces to g(θ̂) ≈ g(θ), which contains no randomness and thus cannot characterize the distribution of g(θ̂) − g(θ). The tangent line is flat, so it fails to capture how variability in θ̂ propagates through g. The remedy is the second-order delta method: retain the quadratic term ½g″(θ)(θ̂ − θ)², which leads to a chi-squared limiting distribution at a faster convergence rate of n rather than √n, specifically n(g(θ̂) − g(θ)) →ᵈ ½g″(θ)σ²χ²₁.

Question 2: BASIC CALCULATION

Let X₁, …, Xₙ be i.i.d. Bernoulli(p), p ∈ (0,1). Use the delta method to find the asymptotic distribution of g(p̂) = log(p̂/(1 − p̂)), the log-odds, where p̂ = X̄.

Show answer

By CLT, √n(p̂ − p) →ᵈ N(0, p(1−p)). Let g(x) = log(x/(1−x)), so g′(x) = 1/(x(1−x)). Then g′(p) = 1/(p(1−p)). Applying the delta method: √n(g(p̂) − g(p)) →ᵈ N(0, p(1−p) · [1/(p(1−p))]²) = N(0, 1/(p(1−p))). Therefore the log-odds has asymptotic variance 1/(np(1−p)). This is the well-known result used in logistic regression inference, where the estimated standard error of the log-odds is 1/√(np̂(1−p̂)).

Question 3: INTERMEDIATE

Let X₁, …, Xₙ be i.i.d. Exponential(λ), so E[Xᵢ] = 1/λ and Var(Xᵢ) = 1/λ². The MLE of λ is λ̂ = 1/X̄. Use the delta method to derive the asymptotic distribution of λ̂ and show it matches the expected result from Fisher information.

Show answer

By CLT, √n(X̄ − 1/λ) →ᵈ N(0, 1/λ²). Set g(x) = 1/x, so g′(x) = −1/x² and g′(1/λ) = −λ². Applying the delta method: √n(1/X̄ − λ) →ᵈ N(0, (1/λ²)(λ²)²) = N(0, λ²). So λ̂ = 1/X̄ has asymptotic distribution N(λ, λ²/n). From Fisher information, I(λ) = 1/λ² for the exponential, so the MLE's asymptotic variance is 1/(nI(λ)) = λ²/n, which matches exactly. This confirms that the delta method and the Fisher information approach give consistent results for the MLE of the rate parameter.

Question 4: APPLIED

In a clinical trial, the estimated hazard ratio is exp(β̂) = 0.72 with β̂ = −0.329 and SE(β̂) = 0.14. Use the delta method to compute the standard error of the hazard ratio and construct an approximate 95% confidence interval for it.

Show answer

With g(β) = exp(β), we have g′(β) = exp(β). By the delta method, SE(exp(β̂)) ≈ |g′(β̂)| · SE(β̂) = exp(−0.329) × 0.14 = 0.72 × 0.14 = 0.1008. The 95% CI for the hazard ratio is 0.72 ± 1.96 × 0.1008 = (0.522, 0.918). Alternatively, and often preferably, one constructs the CI on the log scale: −0.329 ± 1.96 × 0.14 = (−0.604, −0.054), then exponentiates to get (exp(−0.604), exp(−0.054)) = (0.547, 0.947). The two intervals differ slightly because the log-scale CI respects the asymmetry of the exponential transformation and typically provides better coverage.

Question 5: CRITICAL THINKING

Let X₁, …, Xₙ be i.i.d. N(μ, σ²) with both μ and σ² unknown. Consider the coefficient of variation CV = σ/μ (assume μ > 0). The vector estimator is (X̄, S²) with asymptotic covariance matrix Σ = diag(σ², 2σ⁴)/n. Use the multivariate delta method to derive the asymptotic variance of the plug-in estimator ĈV = S/X̄.

Show answer

Define θ = (μ, σ²)ᵀ and g(μ, σ²) = √(σ²)/μ = σ/μ. The partial derivatives are ∂g/∂μ = −σ/μ² and ∂g/∂(σ²) = 1/(2μ√(σ²)) = 1/(2μσ). So ∇g = (−σ/μ², 1/(2μσ))ᵀ. The asymptotic covariance matrix of √n((X̄, S²)ᵀ − (μ, σ²)ᵀ) is Σ = diag(σ², 2σ⁴). By the multivariate delta method, the asymptotic variance of √n(ĈV − CV) is ∇gᵀ Σ ∇g = (σ/μ²)² · σ² + (1/(2μσ))² · 2σ⁴ = σ⁴/μ⁴ + σ²/(2μ²) = (CV)⁴ · (σ²/σ²) + ... Simplifying: σ⁴/μ⁴ + σ²/(2μ²) = CV² · (CV² + 1/2). Therefore Var(ĈV) ≈ CV²(CV² + 1/2)/n. This result shows that the coefficient of variation is harder to estimate precisely when CV itself is large, which accords with intuition since larger relative variability compounds estimation difficulty.

SUMMARY

Summary

The Taylor expansion provides the analytical bridge between the known asymptotic behavior of an estimator θ̂ and the distributional properties of a smooth transformation g(θ̂). The first-order delta method linearizes g around θ, yielding the result that if √n(θ̂ − θ) →ᵈ N(0, σ²) and g′(θ) ≠ 0, then √n(g(θ̂) − g(θ)) →ᵈ N(0, σ²[g′(θ)]²). The multivariate extension replaces the scalar derivative with a gradient or Jacobian, giving asymptotic covariance ∇gᵀΣ∇g. When g′(θ) = 0, the second-order delta method yields a chi-squared limit at convergence rate n.

Key applications include constructing confidence intervals for nonlinear functions of parameters (odds ratios, hazard ratios, ratios of means), deriving variance-stabilizing transformations (square root for Poisson, arcsine for binomial, Fisher's z for correlations), and computing standard errors of MLE-derived quantities. The method works best when n is large, g is smooth, and the expansion point is not a critical point of g. For small samples or non-smooth transformations, the bootstrap offers a robust alternative.