Central Limit Theorem - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The Central Limit Theorem (CLT) stands as one of the most profound results in probability theory, providing the theoretical justification for why the normal distribution appears so ubiquitously across scientific disciplines. Its development spans nearly three centuries, originating from attempts to approximate the binomial distribution and culminating in the rigorous measure-theoretic formulations of the early twentieth century. Understanding the historical trajectory of the CLT illuminates not only the theorem itself but also the evolution of mathematical rigor in probability theory—from the combinatorial calculations of de Moivre to the characteristic function methods of Lévy and Lindeberg.

1733 • De Moivre's Approximation

Abraham de Moivre published the first version of the CLT in The Doctrine of Chances, demonstrating that the binomial distribution with parameters n and p = 1/2 converges to a bell-shaped curve as n → ∞. His result was restricted to the symmetric case and used Stirling's approximation for factorials.

1812 • Laplace's Generalization

Pierre-Simon Laplace extended de Moivre's result to arbitrary p in his Théorie analytique des probabilités, and further generalized the theorem to sums of independent random variables with identical distributions, using generating functions as his primary analytical tool.

1901 • Lyapunov's Condition

Aleksandr Lyapunov provided the first rigorous proof of the CLT for independent but not necessarily identically distributed random variables, introducing the Lyapunov condition involving absolute moments of order 2 + δ, and pioneering the use of characteristic functions in the proof.

1922 • Lindeberg's Condition

Jarl Waldemar Lindeberg introduced the Lindeberg condition—a necessary and sufficient criterion (with Feller's additional constraint) for the CLT to hold for triangular arrays of independent random variables, substantially generalizing Lyapunov's result.

1935 • Feller's Completeness

William Feller proved that the Lindeberg condition, combined with a negligibility (or 'uniformly asymptotically negligible') condition, is both necessary and sufficient for the CLT, providing the definitive characterization of when normalized partial sums converge in distribution to the standard normal.

The central question that drove three centuries of development can be stated concisely: under what conditions does the standardized sum of independent random variables converge in distribution to a Gaussian? The answer—encapsulated in the Lindeberg–Feller theorem—reveals that normality arises whenever no single summand dominates the aggregate variance, a condition of remarkable generality that explains the near-universal appearance of bell curves in empirical data.

SECTION 2

Core Principles & Definitions

Before stating the CLT formally, it is essential to establish the underlying concepts upon which the theorem rests. The CLT is fundamentally a statement about convergence in distribution (also called weak convergence), which is the weakest of the standard modes of stochastic convergence. It asserts that the cumulative distribution functions of standardized partial sums approach the standard normal CDF pointwise at every continuity point. The theorem requires careful attention to the interplay between independence, moment conditions, and the behavior of individual summands relative to the total variance.

Convergence in Distribution

A sequence of random variables X_n converges in distribution to X (written X_n →^d X) if F_n(x) → F(x) at every point x where F is continuous. Equivalently, E[g(X_n)] → E[g(X)] for all bounded continuous functions g.

Characteristic Functions

The characteristic function φ_X(t) = E[e^itX] uniquely determines a distribution. By Lévy's continuity theorem, φ_n(t) → φ(t) pointwise implies X_n →^d X, making this the principal proof technique for CLT results.

Standardization

Given S_n = X₁ + ⋯ + X_n, the standardized sum Z_n = (S_n − E[S_n]) / √Var(S_n) is centered at zero with unit variance, enabling comparison with the standard normal N(0, 1).

Independence & Moment Conditions

The classical CLT requires the summands to be independent and identically distributed (i.i.d.) with finite second moment. Extensions relax the identical distribution assumption but impose conditions—such as Lindeberg's or Lyapunov's—that prevent any single summand from dominating the sum.

Negligibility (UAN Condition)

The uniformly asymptotically negligible condition requires max_k Var(X_k) / Var(S_n) → 0, ensuring that no individual term has outsized influence on the sum's distribution as n grows.

✦

KEY TAKEAWAY

Think of the Central Limit Theorem as the statistical analog of constructive interference in wave physics. Each independent random variable contributes a 'signal' with its own distributional shape, but when many such signals are superimposed, the idiosyncratic features cancel out while the aggregate behavior converges to a universal pattern—the Gaussian. Just as diverse wave sources can produce a coherent diffraction pattern, diverse random variables yield a coherent normal distribution under summation, provided no single source dominates the collective.

SECTION 3

Visual Explanation — Convergence to Normality

The following diagram illustrates the essence of the CLT by showing how the distribution of the standardized sample mean evolves with increasing sample size. The parent population is taken to be a strongly right-skewed exponential distribution—deliberately non-normal—to emphasize that the CLT holds regardless of the original distribution's shape. As n increases from 1 to 30, the sampling distribution of the standardized mean progressively approaches the standard normal density, plotted as the dashed reference curve.

Four panels showing the sampling distribution of the standardized mean X̄ for an Exponential(1) parent population. At n = 1 (cyan), the distribution matches the skewed parent. By n = 5 (violet) the skewness diminishes noticeably. At n = 15 (pink) the distribution closely tracks the dashed N(0, 1) reference curve, and at n = 30 (emerald) the fit is excellent.

Several features of the diagram are worth noting. In the upper-left panel (n = 1), the shaded region reproduces the exponential distribution itself—steeply decaying and strongly right-skewed with skewness γ₁ = 2. As n grows, the skewness of the sampling distribution decays at rate 1/√n, so by n = 30 (lower-right, emerald shading) the asymmetry is nearly imperceptible and the shaded density essentially overlays the dashed standard normal curve. This visual progression captures the core content of the CLT: regardless of the parent distribution, standardized partial sums approach normality, and the rate of approach depends on the higher-order moment structure of the underlying variables.

SECTION 4

Mathematical Framework

We now present the formal statements of the CLT in increasing generality, beginning with the classical i.i.d. version and progressing through the Lindeberg–Feller theorem. Each formulation is accompanied by the key equation and a discussion of the proof strategy, which in all cases relies on the convergence of characteristic functions via Lévy's continuity theorem.

Classical CLT (i.i.d. case)

CLASSICAL CLT

Zₙ = (S_n − nμ) / (σ√n) →ᵈ N(0, 1) as n → ∞

Let X₁, X₂, … be i.i.d. random variables with E[X₁] = μ and Var(X₁) = σ² ∈ (0, ∞). Define S_n = X₁ + ⋯ + X_n. Then the standardized sum Z_n converges in distribution to the standard normal.

The proof proceeds by computing the characteristic function of Z_n. Since the X_k are i.i.d., φ_Zₙ(t) = [φ_X(t/(σ√n)) × e^{−itμ/(σ√n)}]ⁿ. A Taylor expansion of φ_X about 0, using φ_X(0) = 1, φ'_X(0) = iμ, and φ''_X(0) = −(σ² + μ²), yields φ_Zₙ(t) → e^−t²/2, which is the characteristic function of N(0, 1). By Lévy's continuity theorem, pointwise convergence of characteristic functions to a function continuous at 0 implies convergence in distribution.

Lindeberg–Feller CLT (Non-Identically Distributed Case)

LINDEBERG CONDITION

Lₙ(ε) = (1/s²ₙ) Σₖ₌₁ⁿ E[( Xₖ − μₖ )² · 𝟙{ |Xₖ − μₖ| > εsₙ }] → 0 for every ε > 0

Here μ_k = E[X_k], σ²_k = Var(X_k), and s²_n = Σ σ²_k. The indicator function 𝟙{·} truncates the contribution of large deviations, and convergence of L_n(ε) to 0 ensures that no single summand contributes disproportionately to the variance.

LYAPUNOV CONDITION (SUFFICIENT FOR LINDEBERG)

(1/s²⁺ᵟₙ) Σₖ₌₁ⁿ E[ |Xₖ − μₖ|²⁺ᵟ ] → 0 for some δ > 0

Lyapunov's condition is strictly stronger than Lindeberg's: it requires the existence of absolute moments of order 2 + δ and is often simpler to verify in practice. When it holds, the Lindeberg condition is automatically satisfied, and therefore the CLT applies.

BERRY–ESSEEN BOUND (RATE OF CONVERGENCE)

sup_x |P(Zₙ ≤ x) − Φ(x)| ≤ C · ρ / (σ³√n)

Where Φ is the standard normal CDF, ρ = E[|X₁ − μ|³] is the third absolute central moment, and C is a universal constant (currently known to satisfy C ≤ 0.4748). This quantifies the uniform rate of convergence in the i.i.d. case, showing an O(1/√n) rate that depends on the ratio ρ/σ³.

SECTION 5

Conditions, Variants, and the Hierarchy of CLTs

The Central Limit Theorem is not a single result but rather a family of theorems with varying hypotheses and conclusions. The relationships among the major variants form a logical hierarchy: Lyapunov's condition implies Lindeberg's, which (combined with negligibility) is necessary and sufficient. The classical i.i.d. CLT is a corollary of Lindeberg–Feller with identical distributions automatically satisfying the Lindeberg condition when σ² < ∞. Beyond these, multidimensional and functional CLTs extend the framework to vector-valued random variables and stochastic processes, respectively.

The hierarchy of CLT variants. The Lindeberg–Feller theorem (amber border) sits at the top as the most general characterization. The Lyapunov CLT (violet) provides a sufficient condition that is easier to verify. The classical i.i.d. CLT (cyan) is the most commonly applied special case. Dashed lines connect the Berry–Esseen bound and the De Moivre–Laplace theorem as related refinements.

CLT Variant	Hypothesis	Key Distinction
Classical (i.i.d.)	X_k i.i.d., 0 < σ² < ∞	Simplest statement; finite variance is the only moment requirement
Lyapunov	Independent (not necessarily identical), Lyapunov condition for some δ > 0	Sufficient but not necessary; easier to check than Lindeberg
Lindeberg–Feller	Independent, UAN + Lindeberg condition	Necessary and sufficient characterization of asymptotic normality
Multivariate	ℝ^d-valued, finite covariance matrix Σ	Convergence to N(0, Σ) in distribution on ℝ^d
Functional (Donsker)	i.i.d., finite variance; convergence in C[0,1]	Random walk → Brownian motion; the CLT for stochastic processes

Comparison of major CLT variants by hypothesis and distinguishing feature

SECTION 6

Worked Example

We now illustrate the application of the CLT through a detailed worked example that demonstrates both the approximation and the use of the Berry–Esseen bound to assess its quality.

Approximating the Distribution of a Sample Mean from a Skewed Population

1
Step 1 — Identify the Problem Setup

A call center records the duration (in minutes) of individual customer calls. Historical data shows that call durations follow an Exponential distribution with rate parameter λ = 0.2, giving μ = E[X] = 1/λ = 5 minutes and σ² = Var(X) = 1/λ² = 25. A supervisor samples n = 64 calls at random. We wish to find P(X̄ > 5.8), the probability that the sample mean exceeds 5.8 minutes.

μ = 5, σ² = 25, σ = 5, n = 64
2
Step 2 — Verify CLT Applicability

The call durations are independent (calls are randomly sampled) and identically distributed with finite variance σ² = 25 < ∞. Therefore the classical i.i.d. CLT applies. Although the Exponential(0.2) distribution is strongly right-skewed (γ₁ = 2), the CLT guarantees that the standardized sample mean converges in distribution to N(0, 1). With n = 64, the approximation is expected to be reasonably accurate.

CLT conditions satisfied: i.i.d. with 0 < σ² < ∞
3
Step 3 — Standardize the Sample Mean

Under the CLT, X̄ is approximately N(μ, σ²/n) = N(5, 25/64). The standard error is σ/√n = 5/√64 = 5/8 = 0.625. We compute the z-score: Z = (X̄ − μ) / (σ/√n) = (5.8 − 5) / 0.625 = 0.8 / 0.625 = 1.28.

Z = 1.28
4
Step 4 — Compute the Probability

Using the standard normal CDF, P(X̄ > 5.8) = P(Z > 1.28) = 1 − Φ(1.28). From standard normal tables or software, Φ(1.28) ≈ 0.8997. Therefore P(X̄ > 5.8) ≈ 1 − 0.8997 = 0.1003.

P(X̄ > 5.8) ≈ 0.1003
5
Step 5 — Assess Approximation Quality via Berry–Esseen

For the Exponential(λ) distribution, the third absolute central moment is ρ = E[|X − μ|³] = (2 + 6)/λ³ = 2/λ³ (this follows from the moment generating function). With λ = 0.2, ρ = 2/(0.008) = 250. The Berry–Esseen bound gives |P(Z_n ≤ x) − Φ(x)| ≤ C × ρ / (σ³√n) = 0.4748 × 250 / (125 × 8) = 0.4748 × 250 / 1000 ≈ 0.1187. While this uniform bound is not especially tight, it confirms that the maximum pointwise error is bounded by roughly 12%, and for moderate z-values like 1.28 the actual error is considerably smaller—simulation shows the true probability is approximately 0.098, differing from our CLT estimate by only about 0.002.

Berry–Esseen bound: ≤ 0.1187 (uniform); actual error at z = 1.28 ≈ 0.002

SECTION 7

Strengths, Limitations, and Common Pitfalls

The CLT is extraordinarily powerful, but its correct application requires awareness of the conditions under which it holds and the situations where it may fail or converge slowly. The following table summarizes the key strengths alongside important limitations that practitioners must keep in mind.

Strengths	Limitations
Distribution-free: applies regardless of the parent distribution's shape, requiring only finite variance for the i.i.d. case.	Infinite variance distributions (e.g., Cauchy, stable distributions with α < 2) are excluded; sums of Cauchy variables remain Cauchy rather than converging to normal.
Provides the theoretical foundation for confidence intervals, hypothesis tests, and the asymptotic normality of maximum likelihood estimators.	The CLT is an asymptotic result (n → ∞) and gives no finite-sample guarantee without supplementary bounds like Berry–Esseen.
Extends naturally to non-identical distributions via Lindeberg–Feller, providing broad generality.	Rate of convergence depends on higher moments: heavy-tailed distributions (large kurtosis or skewness) converge more slowly and may require large n.
Computationally simple to apply: only μ and σ² are needed for the approximation.	Independence is crucial; the standard CLT fails for dependent data. Mixing conditions or martingale CLTs are required for correlated sequences.
Berry–Esseen provides a computable error bound of order O(1/√n) in the i.i.d. case.	In the tails of the distribution, the CLT approximation degrades; Cramér-type large deviation results are needed for accurate tail probabilities.

Strengths and limitations of the Central Limit Theorem

⚠

KEY TAKEAWAY

The CLT is often casually invoked with the heuristic 'n ≥ 30 is sufficient,' but this can be dangerously misleading. For symmetric, light-tailed distributions, even n = 10 may yield an excellent normal approximation. Conversely, for heavily skewed or leptokurtic populations (e.g., lognormal with large σ), n in the hundreds or thousands may be needed. Always consider the parent distribution's shape when assessing whether the CLT approximation is adequate for a given sample size.

SECTION 8

Connections to Advanced Theory

The Central Limit Theorem occupies a central node in a web of interconnected results in probability theory and mathematical statistics. Understanding how the CLT relates to other convergence theorems, stable distributions, and functional analysis deepens one's appreciation of its structural role and points toward research frontiers.

Classical CLT Framework	Advanced Extension	Key Insight
i.i.d. summands with finite σ²	Stable distributions (Generalized CLT)	When variance is infinite (α-stable with α < 2), normalized sums converge to a non-Gaussian stable law rather than N(0,1).
Convergence in distribution of scalars	Donsker's Invariance Principle	The entire partial-sum process (S_⌊nt⌋)_t∈[0,1] converges weakly in C[0,1] to Brownian motion—a functional CLT.
Independence required	Martingale CLT	For martingale difference sequences with appropriate conditional variance conditions, a CLT still holds, enabling applications to time series and adaptive designs.
Finite-dimensional CLT	CLT in Banach/Hilbert spaces	The CLT extends to random elements in infinite-dimensional spaces, with convergence to Gaussian measures characterized by covariance operators.
Berry–Esseen rate O(1/√n)	Edgeworth expansions	Asymptotic expansions of the CDF provide corrections involving skewness and kurtosis, yielding O(1/n) accuracy.

From the classical CLT to advanced extensions

Perhaps the most conceptually striking extension is Donsker's theorem (1951), also known as the functional central limit theorem or invariance principle. Where the classical CLT addresses the distribution of S_n at a single time point n, Donsker's theorem shows that the rescaled random walk process, viewed as a random element of C[0,1], converges in distribution to a standard Brownian motion W(t). This result is the foundation of modern mathematical finance, queueing theory, and the study of empirical processes. The Kolmogorov–Smirnov test statistic, for instance, derives its asymptotic distribution directly from Donsker's theorem applied to the empirical CDF process.

💡

Looking Ahead

In a subsequent lesson on Stein's method, we will explore a modern proof technique that not only provides an alternative derivation of the CLT but also yields explicit bounds on distributional distance (in total variation or Wasserstein metrics) without recourse to characteristic functions. Stein's method has become a powerful tool for establishing CLT-type results in settings with complex dependence structures.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why the Central Limit Theorem does not apply to the sum of i.i.d. Cauchy random variables. What happens to the distribution of the sample mean in this case, and which generalization of the CLT governs the limiting behavior?

Show answer

The Cauchy distribution has no finite mean (E[|X|] = ∞) and therefore no finite variance. Since the CLT requires at minimum a finite second moment (σ² < ∞), it does not apply. In fact, the sample mean of n i.i.d. Cauchy variables has the same Cauchy distribution as a single observation—the Cauchy distribution is a stable distribution with index α = 1. The appropriate generalization is the Generalized Central Limit Theorem, which states that properly normalized sums of i.i.d. random variables can only converge to an α-stable distribution. For the Cauchy case, the limiting law is Cauchy itself (1-stable).

Question 2: BASIC CALCULATION

Let X₁, X₂, …, X₁₀₀ be i.i.d. with E[X_i] = 12 and Var(X_i) = 36. Use the CLT to approximate P(S₁₀₀ > 1250), where S₁₀₀ = X₁ + ⋯ + X₁₀₀.

Show answer

Under the CLT, S₁₀₀ is approximately N(nμ, nσ²) = N(1200, 3600). The standard deviation of S₁₀₀ is √3600 = 60. Standardizing: Z = (1250 − 1200)/60 = 50/60 ≈ 0.833. Therefore P(S₁₀₀ > 1250) = P(Z > 0.833) = 1 − Φ(0.833) ≈ 1 − 0.7975 = 0.2025. There is approximately a 20.25% chance that the sum exceeds 1250.

Question 3: INTERMEDIATE

Suppose X₁, X₂, … are independent with X_k ~ Uniform(0, k) for k = 1, 2, …, n. Verify that Lyapunov's condition holds with δ = 1 and conclude that the CLT applies. Compute the necessary quantities explicitly.

Show answer

For X_k ~ Uniform(0, k): μ_k = k/2, σ²_k = k²/12, and E[|X_k − μ_k|³] = k³/32 (third absolute central moment of Uniform). Thus s²_n = Σ k²/12 = n(n+1)(2n+1)/72, which grows as n³/36. The Lyapunov sum with δ = 1 is Σ k³/32 = [n(n+1)/2]²/32, which grows as n⁴/128. The Lyapunov ratio is [n⁴/128] / [n³/36]^(3/2) = [n⁴/128] / [

n^{9/2}

(36)^{3/2}

]. Since

(36)^{3/2}

= 216, the ratio is (216 × n⁴) / (128 ×

n^{9/2}

) = 216/(128√n) → 0 as n → ∞. The Lyapunov condition is satisfied, so the CLT holds for these non-identically distributed summands.

Question 4: APPLIED

A manufacturing process produces resistors whose resistance (in Ω) follows a distribution with mean 100 and standard deviation 4. Quality control samples 49 resistors and computes X̄. (a) Use the CLT to find the probability that X̄ falls between 99 and 101 Ω. (b) How large must n be so that P(|X̄ − 100| < 0.5) ≥ 0.99?

Show answer

(a) The standard error is σ/√n = 4/7 ≈ 0.5714. We standardize: Z₁ = (99 − 100)/0.5714 ≈ −1.75 and Z₂ = (101 − 100)/0.5714 ≈ 1.75. So P(99 < X̄ < 101) = Φ(1.75) − Φ(−1.75) = 2 × Φ(1.75) − 1 ≈ 2(0.9599) − 1 = 0.9199. (b) We need P(|X̄ − 100| < 0.5) = P(|Z| < 0.5√n/4) ≥ 0.99. This requires 0.5√n/4 ≥ z₀.₀₀₅ = 2.576, giving √n ≥ 2.576 × 8 = 20.608, so n ≥ 424.7, meaning n ≥ 425 resistors must be sampled.

Question 5: CRITICAL THINKING

Consider independent random variables X₁, X₂, … where X_k takes values ±√k each with probability 1/(2k) and value 0 with probability 1 − 1/k. Show that the Lindeberg condition fails and the CLT does not hold. Identify which property of the sequence causes the failure.

Show answer

First, note E[X_k] = 0 and Var(X_k) = k × (1/k) = 1, so s²_n = n. For the Lindeberg condition with a given ε > 0, the contribution from X_k is E[X²_k 𝟙{|X_k| > ε√n}] = k × (1/k) × 𝟙{√k > ε√n} = 𝟙{k > ε²n}. The Lindeberg sum is (1/n) Σ_k 𝟙{k > ε²n}. For k from ⌈ε²n⌉ to n, this sum equals (n − ⌈ε²n⌉ + 1)/n → 1 − ε², which does not converge to 0. The Lindeberg condition fails because the UAN (negligibility) condition is violated: max_k P(|X_k| > ε√n) = 1/n for k = n, but X_n takes values ±√n with probability 1/(2n), meaning the largest term has magnitude comparable to s_n = √n itself, violating the requirement that individual summands be asymptotically negligible relative to the total standard deviation.

SUMMARY

Summary

The Central Limit Theorem establishes that the standardized sum of independent random variables converges in distribution to the standard normal distribution under mild conditions. In the classical i.i.d. case, only a finite variance is required: Z_n = (S_n − nμ)/(σ√n) →^d N(0, 1). For non-identically distributed summands, the Lindeberg condition combined with the UAN (negligibility) condition provides a necessary and sufficient characterization, while Lyapunov's condition offers a convenient sufficient criterion involving moments of order 2 + δ.

The Berry–Esseen bound quantifies the rate of convergence at O(1/√n), while Edgeworth expansions provide higher-order corrections. Extensions include the multivariate CLT for vector-valued data, Donsker's invariance principle for stochastic processes (convergence to Brownian motion in C[0,1]), and the martingale CLT for dependent sequences. The theorem's power lies in its universality—it explains why the normal distribution appears across all quantitative disciplines—but its limitations must be respected: infinite-variance distributions, dependence structures, and heavy tails can all invalidate or slow the convergence.