MGF & Characteristic Functions — Moment generating...

SECTION 1

Historical Context & Motivation

Probability theory advanced rapidly in the eighteenth and nineteenth centuries, but mathematicians repeatedly encountered a fundamental challenge: how does one efficiently extract all moments of a distribution, prove that sums of independent random variables follow particular laws, or establish that a sequence of distributions converges to a limiting form? Direct integration of probability densities was often intractable, especially for sums of random variables where convolutions quickly become unwieldy. The idea of transforming a distribution into a different mathematical object — one that is easier to manipulate algebraically — provided a powerful resolution. The moment generating function (MGF) and the characteristic function (CF) are two such transforms, each encoding a distribution's complete information in a single analytic or continuous function of a real or complex parameter.

1812

Laplace's Generating Functions

Pierre-Simon Laplace developed probability generating functions in his Théorie analytique des probabilités, laying the algebraic groundwork for encoding distributions via power series and transforms.

1894

Characteristic Functions Formalized

Paul Lévy and others began formalizing the characteristic function φ(t) = E[e^{itX}] as a Fourier–Stieltjes transform of the distribution, recognizing its universal existence for all random variables.

1920s

Lévy Continuity Theorem

Paul Lévy proved that pointwise convergence of characteristic functions implies convergence in distribution, providing the most elegant route to the Central Limit Theorem.

1930s–40s

Cramér–Wold and Uniqueness Theorems

Harald Cramér and Hermann Wold extended characteristic function methods to multivariate settings, while uniqueness and inversion theorems solidified the CF as a complete representation of a distribution.

Modern Era

Computational & Theoretical Applications

MGFs and CFs are now indispensable in mathematical statistics for deriving sampling distributions, proving limit theorems, and characterizing distributional families in areas from actuarial science to quantum mechanics.

The central question these transforms answer is deceptively simple: given a random variable X, can we construct a single function that faithfully encodes all information about its distribution — its moments, its shape, its behavior under summation — in a form that converts difficult integral operations into simple algebraic ones? The MGF and CF each provide an affirmative answer, with important differences in their domains of applicability and mathematical properties.

SECTION 2

Core Principles & Definitions

Both the moment generating function and the characteristic function belong to the broader family of distributional transforms — functions of a parameter that capture the entire probability structure of a random variable. Their utility rests on several foundational principles that govern when they exist, how they encode moments, and why they uniquely determine distributions.

Transform as Expectation

Both MGF and CF are defined as expectations of exponential functions of X. The MGF uses M(t) = E[e^{tX}] with a real parameter t, while the CF uses φ(t) = E[e^{itX}] with an imaginary exponent, making it a Fourier transform of the density.

Moment Extraction via Differentiation

Successive derivatives of M(t) evaluated at t = 0 yield raw moments: M⁽ⁿ⁾(0) = E[Xⁿ]. For the CF, the n-th moment is extracted via φ⁽ⁿ⁾(0) = iⁿE[Xⁿ], converting integration into differentiation.

Uniqueness Theorem

If two distributions share the same MGF on an open interval containing 0, or the same CF everywhere, they are identical. This makes transforms definitive fingerprints of distributions.

Convolution Becomes Multiplication

For independent random variables X and Y, the transform of X + Y is the product of their individual transforms. This converts convolution of densities into simple multiplication, enormously simplifying sums of independent variables.

Existence & Domain

The CF always exists for any random variable because |e^{itX}| = 1. The MGF may not exist if E[e^{tX}] diverges, which occurs for heavy-tailed distributions like Cauchy. This makes the CF the universally applicable tool.

✦ KEY TAKEAWAY

Think of the MGF and CF as two different types of barcode scanners for probability distributions. The MGF is like a high-resolution scanner that reveals fine details (all moments) quickly, but it can only scan certain barcodes (distributions with finite exponential moments). The CF is a universal scanner: it works on every barcode (every distribution), though extracting certain details may require more sophisticated decoding (inversion formulas). Both barcodes uniquely identify their product — knowing the transform means knowing the distribution.

SECTION 3

Visual Explanation: How Transforms Encode Distributions

The following diagram illustrates the core pipeline: a probability distribution is passed through either the MGF or CF transform, yielding a function in the transform domain. From that function, moments can be extracted by differentiation, sums of independent variables are computed by multiplication, and the original distribution can be recovered through inversion. The diagram also highlights the critical difference in existence between the two transforms.

The transform pipeline: a distribution is mapped to either the MGF or CF domain, where moments are extracted by differentiation, sums of independent variables become products, and the original density is recoverable via inversion. The dashed box at the bottom highlights the crucial existence distinction between the two transforms.

Notice the structural symmetry: both transforms serve as algebraic encodings of the distribution, but the CF's use of complex exponentials guarantees boundedness (since |e^itX| = 1 for all real X and t), whereas the MGF's real exponential e^tX can grow without bound, causing the expectation to diverge for heavy-tailed distributions.

SECTION 4

Mathematical Framework

Moment Generating Function

MGF DEFINITION

M_X(t) = E[e^{tX}] = ∫_{−∞}^{∞} e^{tx} f(x) dx

where t ∈ ℝ, f(x) is the PDF (or replace the integral with a sum for discrete X). The MGF exists if this expectation is finite on some open interval (−h, h) with h > 0.

The power of the MGF lies in its Taylor expansion. Expanding e^tX and exchanging expectation with summation (justified when the MGF exists in a neighborhood of zero), we obtain M_X(t) = Σ E[Xⁿ] tⁿ / n!. This immediately shows that the n-th derivative at t = 0 yields the n-th raw moment.

MOMENT EXTRACTION

M_X^{(n)}(0) = E[Xⁿ]

The n-th derivative of M_X(t) evaluated at t = 0 gives the n-th raw moment. In particular, M'(0) = E[X] and M''(0) = E[X²], so Var(X) = M''(0) − [M'(0)]².

Characteristic Function

CF DEFINITION

φ_X(t) = E[e^{itX}] = ∫_{−∞}^{∞} e^{itx} f(x) dx

where i = √(−1) and t ∈ ℝ. This is the Fourier transform of f(x) (with a sign convention common in probability). Since |e^{itx}| = 1 for all real x and t, the expectation is always finite — the CF exists for every random variable.

The CF is complex-valued: φ_X(t) = E[cos(tX)] + i E[sin(tX)]. Its real part captures even moments and its imaginary part captures odd moments. The relationship between the CF and moments is given by φ⁽ⁿ⁾_X(0) = iⁿ E[Xⁿ], provided the n-th moment exists.

INDEPENDENCE & SUMS

M_{X+Y}(t) = M_X(t) · M_Y(t) and φ_{X+Y}(t) = φ_X(t) · φ_Y(t)

For independent random variables X and Y. This property extends to any finite collection of mutually independent variables and is the algebraic manifestation of convolution becoming multiplication under the transform.

🔄 Inversion Theorem

The CF uniquely determines the distribution through Lévy's inversion formula: f(x) = (1/2π) ∫_{−∞}^{∞} e^{−itx} φ_X(t) dt, valid when φ_X is integrable. Even when this integral form is not directly applicable, the uniqueness theorem guarantees that two distributions with the same CF must be identical.

SECTION 5

Transforms of Common Distributions

To build fluency with transforms, it is essential to know the MGFs and CFs of standard distributions. The following table provides a reference for the most important families. Notice how the functional form of the transform often reveals structural properties: the normal distribution's MGF is itself an exponential of a quadratic in t, the Poisson's is an exponential of a shifted exponential, and the Cauchy distribution — notably — has no MGF at all.

MGFs and CFs of standard distributions

Distribution	MGF M(t)	CF φ(t)	Domain of MGF
N(μ, σ²)	exp(μt + σ²t²/2)	exp(iμt − σ²t²/2)	All t ∈ ℝ
Exp(λ)	λ / (λ − t)	λ / (λ − it)	t < λ
Poisson(λ)	exp(λ(e^t − 1))	exp(λ(e^{it} − 1))	All t ∈ ℝ
Gamma(α, β)	(1 − t/β)^{−α}	(1 − it/β)^{−α}	t < β
Bernoulli(p)	1 − p + pe^t	1 − p + pe^{it}	All t ∈ ℝ
Cauchy(0,1)	Does not exist	e^{−\|t\|}	—

The real part of the characteristic function for three distributions. The standard normal CF (cyan) is a Gaussian e^−t²/2 that decays smoothly, the Cauchy CF (violet) is e^−|t| with a cusp at the origin, and the Uniform CF (amber) exhibits the oscillatory sinc-like pattern sin(t)/t. All CFs satisfy φ(0) = 1.

The visual behavior of the CF is deeply tied to the tail properties of the distribution. Distributions with lighter tails (like the normal) produce CFs that decay rapidly, while heavy-tailed distributions (like the Cauchy) have CFs that decay slowly. The smoothness of the CF at the origin directly reflects how many moments the distribution possesses: a CF that is n-times differentiable at zero corresponds to a distribution with finite n-th moment.

SECTION 6

Worked Example: Deriving the Distribution of a Sum

One of the most powerful applications of the MGF is proving that the sum of independent normal random variables is itself normal. Let X ~ N(μ₁, σ₁²) and Y ~ N(μ₂, σ₂²) be independent. We wish to find the distribution of S = X + Y.

Sum of Independent Normals via MGF

Step 1 — Write the MGFs of X and YFrom the table of standard MGFs, we have M_X(t) = exp(μ₁t + σ₁²t²/2) and M_Y(t) = exp(μ₂t + σ₂²t²/2). Both exist for all t ∈ ℝ.

M_X(t) = exp(μ₁t + σ₁²t²/2), M_Y(t) = exp(μ₂t + σ₂²t²/2)

Step 2 — Apply the multiplication property for independent sumsSince X and Y are independent, the MGF of S = X + Y is the product M_S(t) = M_X(t) · M_Y(t). We multiply the two exponentials by adding their exponents.

M_S(t) = exp(μ₁t + σ₁²t²/2) · exp(μ₂t + σ₂²t²/2)

Step 3 — Combine exponentsAdding the exponents: (μ₁t + σ₁²t²/2) + (μ₂t + σ₂²t²/2) = (μ₁ + μ₂)t + (σ₁² + σ₂²)t²/2. This produces a single exponential whose exponent has the same functional form as a normal MGF.

M_S(t) = exp((μ₁ + μ₂)t + (σ₁² + σ₂²)t²/2)

Step 4 — Identify the distribution by uniquenessBy the uniqueness theorem, the MGF exp((μ₁ + μ₂)t + (σ₁² + σ₂²)t²/2) is exactly the MGF of a N(μ₁ + μ₂, σ₁² + σ₂²) random variable. Since the MGF determines the distribution, we conclude S follows this normal distribution.

S = X + Y ~ N(μ₁ + μ₂, σ₁² + σ₂²)

Step 5 — Numerical verificationIf X ~ N(3, 4) and Y ~ N(−1, 9), then S ~ N(3 + (−1), 4 + 9) = N(2, 13). The mean of S is 2 and the variance is 13, so the standard deviation is √13 ≈ 3.606. We can verify: E[S] = E[X] + E[Y] = 3 + (−1) = 2 ✓, and Var(S) = Var(X) + Var(Y) = 4 + 9 = 13 ✓ (using independence).

S ~ N(2, 13) with σ_S ≈ 3.606

💡 Why Not Convolution?

To find the distribution of S = X + Y by direct methods, we would need to compute the convolution integral f_S(s) = ∫ f_X(x) f_Y(s − x) dx, which for normal densities involves completing the square inside a product of exponentials — a calculation requiring considerable care. The MGF method accomplishes the same result in four lines of algebra.

SECTION 7

MGF vs. Characteristic Function: Strengths & Limitations

While the MGF and CF share the same algebraic structure — both are expectations of exponential functions — their domains, existence conditions, and analytical properties differ substantially. The choice between them depends on the problem at hand: if the MGF exists, it is often more convenient for moment calculations and identifying distributions; if it does not, the CF is the indispensable alternative.

Comparison of MGF and CF properties

Property	MGF M(t)	CF φ(t)
Existence	May not exist (requires E[e^{tX}] < ∞ near t = 0)	Always exists for every random variable
Range	Real-valued, M(t) ≥ 1 near t = 0	Complex-valued, \|φ(t)\| ≤ 1 with φ(0) = 1
Moment extraction	M⁽ⁿ⁾(0) = E[Xⁿ] directly	φ⁽ⁿ⁾(0) = iⁿE[Xⁿ] (factor of iⁿ)
Uniqueness	Yes, if it exists in a neighborhood of 0	Yes, always (by inversion theorem)
Convergence theorems	If M_n(t) → M(t) for t in neighborhood of 0, and M is an MGF, then convergence in distribution	Lévy continuity theorem: pointwise convergence of CFs ↔ convergence in distribution (to a proper distribution if limit is continuous at 0)
Heavy-tailed distributions	Fails (e.g., Cauchy, log-normal in general)	Works perfectly for all distributions
Connection to transforms	Two-sided Laplace transform of f(x)	Fourier transform of f(x)

✦ KEY TAKEAWAY

In applied work, the MGF is your preferred tool when it exists — it gives moments directly without complex arithmetic and makes algebraic manipulations transparent. But in theoretical work — particularly proving limit theorems like the CLT — the characteristic function is indispensable because it always exists and because the Lévy continuity theorem provides a clean equivalence between pointwise convergence of CFs and convergence in distribution. Think of the MGF as a specialized high-performance wrench and the CF as a universal adjustable wrench: the former is faster when it fits, but the latter works on every bolt.

SECTION 8

Connections to Advanced Theory

The MGF and CF are not isolated constructs; they are embedded in a rich network of related transforms and advanced theorems that underpin modern probability theory and mathematical statistics. Understanding these connections elevates one's ability to apply transform methods in research-level problems.

Advanced connections from MGF and CF theory

Related Concept	Connection to MGF / CF	Key Application
Cumulant Generating Function (CGF)	K(t) = log M(t). Cumulants κₙ = K⁽ⁿ⁾(0); κ₁ = mean, κ₂ = variance, κ₃ = related to skewness.	Saddlepoint approximations, Edgeworth expansions, exponential family theory.
Probability Generating Function (PGF)	G(z) = E[z^X] for non-negative integer-valued X. Related to MGF by G(z) = M(log z).	Branching processes, queueing theory, discrete distribution families.
Central Limit Theorem via CF	The CF of the standardized sample mean converges pointwise to e^{−t²/2}, the CF of N(0,1), establishing the CLT via Lévy continuity.	The most elegant proof of the CLT; generalizations to non-identically distributed summands.
Large Deviations (Cramér's Theorem)	The rate function I(x) = sup_t {tx − log M(t)} is the Legendre–Fenchel transform of the CGF.	Quantifying exponentially rare events, insurance risk theory, information theory.
Multivariate Extensions	M(t₁,...,tₖ) = E[exp(t₁X₁ + ··· + tₖXₖ)] and φ(t₁,...,tₖ) = E[exp(i(t₁X₁ + ··· + tₖXₖ))].	Joint distributions, Cramér–Wold theorem, multivariate CLT, copula characterization.

Looking forward, the cumulant generating function K(t) = log M(t) deserves special mention because cumulants have an additive property for independent variables (κₙ(X + Y) = κₙ(X) + κₙ(Y)), which makes them natural parameters for exponential families and central objects in saddlepoint approximation theory. The Cramér–Wold theorem extends CF methods to multivariate settings by asserting that a random vector's distribution is determined by the CFs of all its one-dimensional projections. These tools collectively form the analytical backbone of modern mathematical statistics.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why the Cauchy distribution has a well-defined characteristic function but no moment generating function. What property of the Cauchy distribution's tails causes this discrepancy, and what does this imply about the existence of moments?

PROBLEM 2 — BASIC CALCULATION

Let X ~ Poisson(λ). Derive E[X] and E[X²] using the MGF M(t) = exp(λ(e^t − 1)). Then compute Var(X).

PROBLEM 3 — INTERMEDIATE

Let X₁, X₂, ..., Xₙ be i.i.d. Exponential(λ) random variables. Use the MGF to prove that S = X₁ + X₂ + ··· + Xₙ follows a Gamma(n, λ) distribution. State clearly where you use independence and the uniqueness theorem.

PROBLEM 4 — APPLIED

An insurance company models individual claim sizes as i.i.d. N(500, 100²) random variables. If 25 independent claims arrive in a month, use the MGF to determine the exact distribution of the total claim amount T. What is P(T > 13,000)? Express your answer in terms of the standard normal CDF Φ.

PROBLEM 5 — CRITICAL THINKING

Outline the proof of the Central Limit Theorem using characteristic functions. Specifically, for i.i.d. random variables X₁, X₂, ... with mean μ and variance σ², define Zₙ = (Σ Xᵢ − nμ)/(σ√n). Show that φ_{Zₙ}(t) → e^{−t²/2} as n → ∞, and explain how the Lévy continuity theorem completes the argument.

SUMMARY

Summary & Review

The moment generating function M(t) = E[e^tX] and the characteristic function φ(t) = E[e^itX] are distributional transforms that convert probability distributions into functions of a single parameter. Both encode all moments (extractable by differentiation at zero), convert convolution of independent sums into multiplication, and uniquely determine their distributions. The critical distinction is that the MGF may fail to exist for heavy-tailed distributions, while the CF always exists because |e^itX| = 1.

Key applications include proving that sums of independent normals are normal (via MGF multiplication), deriving the Central Limit Theorem (via CF convergence and the Lévy continuity theorem), and computing rate functions in large deviations theory through the cumulant generating function K(t) = log M(t). Mastery of these transforms provides the analytical foundation for virtually all of modern probability theory and mathematical statistics.