Law of Large Numbers - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The Law of Large Numbers (LLN) is one of the cornerstones of probability theory, providing a rigorous justification for the intuitive idea that long-run relative frequencies stabilize around expected values. Before its formalization, gamblers and natural philosophers had observed empirically that the proportion of heads in repeated coin tosses seemed to settle near one-half, and that mortality tables became more reliable as the number of recorded deaths increased. Yet no mathematical framework existed to explain why averaging over many observations should produce a predictable outcome. The LLN bridges the gap between deterministic mathematics and the inherently random nature of individual observations, and its development over three centuries reflects the evolution of probability from philosophical speculation to a precise analytic discipline.

1713 • Bernoulli's Ars Conjectandi

Jacob Bernoulli published the first rigorous version of the LLN, proving that the relative frequency of successes in independent Bernoulli trials converges to the true probability. His proof, which took over twenty years to develop, established that for any desired level of accuracy, a sufficiently large number of trials would make the sample proportion arbitrarily close to p.

1835 • Poisson's Generalization

Siméon Denis Poisson extended the result to sequences of independent trials where the success probability may vary from trial to trial and coined the phrase loi des grands nombres. His generalization significantly broadened the theorem's applicability beyond identically distributed settings.

1867 • Chebyshev's Inequality Proof

Pafnuty Chebyshev provided an elegant proof of the Weak Law of Large Numbers using the inequality that now bears his name. His approach—requiring only the existence of the first two moments—demonstrated the power of moment-based methods and influenced the entire trajectory of probability theory.

1909 • Borel's Normal Number Theorem

Émile Borel established what is effectively the Strong Law of Large Numbers for Bernoulli trials, proving that the set of sequences whose relative frequencies do not converge to p has Lebesgue measure zero. This marked the first almost-sure convergence result in the LLN lineage.

1933 • Kolmogorov's Strong Law

Andrey Kolmogorov proved the definitive Strong Law of Large Numbers for independent, identically distributed random variables with a finite first moment. His axiomatization of probability in the same year provided the measure-theoretic foundation on which the modern SLLN rests.

The historical progression reveals a recurring theme: each generation sought a weaker set of assumptions under which convergence of sample averages could still be guaranteed. Bernoulli required identical Bernoulli trials, Chebyshev demanded finite variance, and Kolmogorov ultimately showed that a finite expectation alone suffices for almost-sure convergence. This motivates the central question of modern convergence theory: in what precise sense—and under what minimal conditions—does the empirical mean converge to the theoretical mean?

SECTION 2

Core Principles & Definitions

The Law of Large Numbers exists in two principal forms, each asserting convergence of the sample mean to the population mean but differing in the mode of convergence employed. Understanding the distinction between convergence in probability and almost-sure convergence is essential before studying the theorems themselves. At its core, the LLN formalizes a surprisingly deep idea: randomness at the level of individual observations does not preclude deterministic behavior at the level of aggregate statistics, provided the aggregation is performed over sufficiently many independent observations.

Weak Law of Large Numbers (WLLN)

The sample mean X̄ₙ converges in probability to the population mean μ. Formally, for every ε > 0, P(|X̄ₙ − μ| ≥ ε) → 0 as n → ∞. This means that the probability of the sample mean deviating from μ by more than any fixed amount can be made arbitrarily small.

Strong Law of Large Numbers (SLLN)

The sample mean X̄ₙ converges almost surely to μ. That is, P(limₙ→∞ X̄ₙ = μ) = 1. Almost-sure convergence implies convergence in probability, making the SLLN the strictly stronger result. It guarantees that any single infinite sequence of observations will, with probability one, yield sample averages converging to μ.

Convergence in Probability

A sequence of random variables {Yₙ} converges in probability to Y if for every ε > 0, P(|Yₙ − Y| > ε) → 0. This mode of convergence permits occasional deviations that become increasingly rare but need not vanish entirely along any single realization of the sequence.

Almost-Sure Convergence

A sequence {Yₙ} converges almost surely (a.s.) to Y if P(ω : Yₙ(ω) → Y(ω)) = 1. Equivalently, except on a set of measure zero, every sample path of the sequence converges pointwise. This is stronger than convergence in probability because it controls the pathwise behavior of the sequence.

I.I.D. Assumption

The classical forms of both the WLLN and SLLN apply to sequences of independent and identically distributed (i.i.d.) random variables. Extensions exist for non-identically distributed and weakly dependent sequences, but the i.i.d. setting provides the cleanest statements and proofs.

✦

KEY TAKEAWAY

Think of estimating a population parameter as tuning a radio dial. Each additional observation slightly adjusts the dial. Under the WLLN, for any desired level of reception quality, you can eventually get close enough to the target frequency with high probability—but there is no guarantee that the dial will not briefly jump away. Under the SLLN, the dial locks in on the target frequency and stays there permanently (with probability one). Both laws assure us that averaging works, but the SLLN provides the stronger pathwise guarantee that practitioners implicitly rely on when computing running averages in Monte Carlo simulations or actuarial modeling.

SECTION 3

Visual Explanation — Convergence of Sample Means

The following diagram illustrates the convergence behavior of the sample mean for multiple independent sequences of i.i.d. random variables drawn from the same distribution. Each colored path represents the running average X̄ₙ = (X₁ + X₂ + ⋯ + Xₙ)/n for a different realization. Observe that all paths eventually funnel toward the true population mean μ, with the fluctuations decreasing roughly as 1/√n. The shaded confidence band narrows over time, visually encoding the variance reduction that drives convergence.

Five independent sample-mean paths (colored lines) all converge to the population mean μ (dashed pink line). The shaded cyan band represents a ±σ/√n confidence envelope that shrinks as n increases, illustrating the variance reduction central to both forms of the LLN.

Notice that individual paths may temporarily deviate from μ, and some paths approach from above while others from below. The crucial observation is that all paths eventually collapse toward the horizontal dashed line. Under the WLLN, we can only guarantee that the probability of a path straying outside the shaded band becomes negligible; under the SLLN, we know that each individual path will, with probability one, eventually remain within any prescribed neighborhood of μ forever after some finite (random) index N(ω). The funnel shape of the confidence band reflects the O(1/√n) rate at which the standard deviation of the sample mean decays.

SECTION 4

Mathematical Framework

We now formalize both forms of the Law of Large Numbers, beginning with the setup common to both. Let X₁, X₂, … be a sequence of i.i.d. random variables defined on a probability space (Ω, ℱ, P), each with finite mean E[Xᵢ] = μ. Define the sample mean as the arithmetic average of the first n observations.

SAMPLE MEAN

X̄ₙ = (1/n) ∑ᵢ₌₁ⁿ Xᵢ

Where X̄ₙ is the sample mean of n observations, Xᵢ are i.i.d. random variables, and n is the sample size. The sample mean is itself a random variable whose distribution concentrates around μ as n grows.

Weak Law of Large Numbers (WLLN)

The WLLN asserts convergence in probability. Historically, the simplest proof proceeds via Chebyshev's inequality, which requires the additional assumption that σ² = Var(Xᵢ) < ∞. However, the WLLN holds under the weaker condition E[|Xᵢ|] < ∞ alone, as can be shown using characteristic functions or truncation arguments.

WLLN STATEMENT

∀ ε > 0 : lim_{n→∞} P(|X̄ₙ − μ| ≥ ε) = 0

For every tolerance ε > 0, the probability that the sample mean deviates from μ by at least ε vanishes as n → ∞. Equivalently, X̄ₙ →^P μ.

Chebyshev Proof of the WLLN (finite variance case)

Suppose Var(Xᵢ) = σ² < ∞. Then Var(X̄ₙ) = σ²/n by independence. Applying Chebyshev's inequality directly: P(|X̄ₙ − μ| ≥ ε) ≤ Var(X̄ₙ)/ε² = σ²/(nε²) → 0 as n → ∞. This two-line argument is elegant but it overstates the required assumptions—Khinchin showed in 1929 that only a finite first moment is needed, by leveraging the continuity theorem for characteristic functions.

CHEBYSHEV BOUND

P(|X̄ₙ − μ| ≥ ε) ≤ σ² / (nε²)

This non-asymptotic bound provides a concrete rate of convergence in probability: the tail probability decays at least as fast as O(1/n). In practice, the true rate is often much faster depending on the tail behavior of the Xᵢ.

Strong Law of Large Numbers (SLLN)

The SLLN upgrades the mode of convergence to almost-sure convergence. Kolmogorov's 1933 proof uses the Kolmogorov maximal inequality combined with a subsequence argument and the Borel–Cantelli lemma. The key condition is simply E[|X₁|] < ∞. Conversely, Kolmogorov also showed that if E[|X₁|] = ∞, then X̄ₙ does not converge almost surely, making finite expectation both necessary and sufficient.

SLLN STATEMENT

P(lim_{n→∞} X̄ₙ = μ) = 1

The sample mean converges to μ along almost every sample path ω ∈ Ω. The set of ω for which convergence fails is a P-null set. This is the strongest standard convergence guarantee for sample averages.

SECTION 5

Modes of Convergence & Classification

A rigorous understanding of the LLN requires placing convergence in probability and almost-sure convergence within the broader hierarchy of stochastic convergence modes. The diagram below maps the logical implications among the four principal modes, and the table that follows details their definitions and relationships to the two laws.

The hierarchy of convergence modes: almost-sure convergence (top) implies convergence in probability, which in turn implies convergence in distribution. Lᵖ convergence implies convergence in probability under appropriate integrability conditions. The boxes at the bottom identify which mode each LLN invokes.

Mode	Definition	LLN Form	Key Condition
Almost Sure	P(ω : Xₙ(ω) → X(ω)) = 1	SLLN (Kolmogorov)	E[\|X₁\|] < ∞ (necessary and sufficient for i.i.d.)
In Probability	∀ε > 0: P(\|Xₙ − X\| > ε) → 0	WLLN (Khinchin)	E[\|X₁\|] < ∞ (or σ² < ∞ for Chebyshev proof)
In L²	E[\|Xₙ − X\|²] → 0	Implies WLLN	σ² < ∞ ensures Var(X̄ₙ) = σ²/n → 0
In Distribution	Fₙ(x) → F(x) at all continuity points	Implied by WLLN (limit is constant)	Weakest mode; convergence to a constant is equivalent to conv. in probability

Summary of convergence modes relevant to the Law of Large Numbers

⚠

Important Distinction

When the limit is a constant (as in the LLN, where X̄ₙ → μ), convergence in probability and convergence in distribution are equivalent. This is a special property that simplifies many proofs in the LLN context but does not hold when the limit is a non-degenerate random variable.

SECTION 6

Worked Example — Bounding Deviations via the WLLN

Consider a quality control engineer sampling resistors from a production line. Each resistor's resistance Xᵢ (in ohms) is modeled as an i.i.d. random variable with mean μ = 100 Ω and variance σ² = 25. The engineer wishes to determine how many resistors must be sampled so that the probability of the sample mean deviating from 100 Ω by more than 1 Ω is at most 5%. We will apply Chebyshev's inequality within the WLLN framework.

Determining Sample Size via Chebyshev's Bound

1
Step 1 — Identify Given Quantities

We have i.i.d. random variables X₁, X₂, …, Xₙ with μ = E[Xᵢ] = 100, σ² = Var(Xᵢ) = 25. The tolerance is ε = 1, and the desired probability bound is P(|X̄ₙ − 100| ≥ 1) ≤ 0.05.

μ = 100, σ² = 25, ε = 1, target probability ≤ 0.05
2
Step 2 — Apply Chebyshev's Inequality

By Chebyshev's inequality applied to X̄ₙ, we have P(|X̄ₙ − μ| ≥ ε) ≤ Var(X̄ₙ)/ε². Since the Xᵢ are independent and identically distributed, Var(X̄ₙ) = σ²/n = 25/n. Substituting: P(|X̄ₙ − 100| ≥ 1) ≤ 25/(n × 1²) = 25/n.

P(|X̄ₙ − 100| ≥ 1) ≤ 25/n
3
Step 3 — Solve for n

We require 25/n ≤ 0.05, which yields n ≥ 25/0.05 = 500. Therefore, the engineer must sample at least 500 resistors to guarantee—via the Chebyshev bound—that the sample mean resistance is within 1 Ω of the true mean with probability at least 95%.

n ≥ 500 resistors
4
Step 4 — Interpret in Context

Note that Chebyshev's inequality is a worst-case bound valid for any distribution with finite variance. If the resistances are known to be normally distributed, the Central Limit Theorem would give a much tighter bound: n ≥ (z₀.₀₂₅)²σ²/ε² = (1.96)² × 25/1 ≈ 96.04, so n = 97 would suffice. This illustrates that the WLLN (via Chebyshev) provides distribution-free guarantees at the cost of a more conservative sample size.

Chebyshev: n = 500 (distribution-free) vs. CLT: n = 97 (assuming normality)

SECTION 7

Comparing the Weak and Strong Laws

While both forms of the LLN guarantee that the sample mean approaches the population mean, the nature and strength of the guarantee differ substantially. In applied settings—particularly in Monte Carlo methods, stochastic optimization, and Bayesian computation—the distinction between convergence in probability and almost-sure convergence has real consequences for algorithm design and the interpretation of simulation output.

Feature	Weak Law (WLLN)	Strong Law (SLLN)
Convergence mode	In probability	Almost sure (with probability 1)
Minimal assumption (i.i.d.)	E[\|X₁\|] < ∞	E[\|X₁\|] < ∞ (same!)
What it says about paths	For any fixed ε, the fraction of 'bad' paths → 0	Every path (except a null set) converges
Proof complexity	Elementary (Chebyshev or characteristic functions)	Harder (Borel–Cantelli, maximal inequalities)
Non-i.i.d. extensions	Readily available (Lindeberg-type conditions)	Require stronger tools (Kolmogorov three-series, ergodic theory)
Practical implication	Justifies point estimation for large n	Justifies Monte Carlo simulation: a single long run converges

Comparison of the Weak and Strong Laws of Large Numbers

✦

KEY TAKEAWAY

In the i.i.d. setting, the WLLN and SLLN require the same moment condition (finite first moment), so the SLLN strictly dominates. The distinction becomes operationally significant in non-i.i.d. or dependent settings, where establishing almost-sure convergence may require stronger tail or mixing conditions. Think of it in terms of experimental replication: the WLLN tells you that if you could run an infinite number of parallel experiments, most of them would yield the right answer. The SLLN tells you that a single experiment, run long enough, will converge—this is exactly the guarantee you need for a Monte Carlo simulation on a single machine.

SECTION 8

Connection to the Central Limit Theorem & Ergodic Theory

The Law of Large Numbers and the Central Limit Theorem (CLT) are complementary pillars of asymptotic statistics. While the LLN identifies the first-order behavior of the sample mean—namely, that it converges to μ—the CLT characterizes the second-order fluctuations around that limit by showing that √n(X̄ₙ − μ) converges in distribution to N(0, σ²). The LLN tells us where the sample mean goes; the CLT tells us how fast and in what shape it gets there.

Feature	Law of Large Numbers	Central Limit Theorem
What converges	X̄ₙ → μ	√n(X̄ₙ − μ) →ᵈ N(0, σ²)
Mode of convergence	In probability (WLLN) or a.s. (SLLN)	In distribution
Minimal moment condition	E[\|X₁\|] < ∞	E[X₁²] < ∞ (finite variance)
Information provided	Location of the limit	Scale and shape of fluctuations around the limit
Primary application	Justifying consistency of estimators	Constructing confidence intervals and hypothesis tests

Law of Large Numbers vs. Central Limit Theorem

Beyond the i.i.d. setting, the LLN connects to ergodic theory. Birkhoff's Ergodic Theorem (1931) generalizes the SLLN to stationary ergodic processes: if (Ω, ℱ, P, T) is a measure-preserving dynamical system and T is ergodic, then for any f ∈ L¹(P), the time average (1/n)∑ₖ₌₀ⁿ⁻¹ f(Tᵏω) converges almost surely to E[f]. This is precisely the SLLN when T is the shift operator on an i.i.d. sequence. The ergodic perspective reveals the LLN as a special case of a far more general principle: time averages equal space averages for sufficiently mixing systems. This connection underpins Markov Chain Monte Carlo (MCMC) methods, where the SLLN for dependent sequences justifies convergence of MCMC estimators under appropriate ergodicity conditions.

💡

Looking Ahead

The LLN establishes that X̄ₙ → μ but says nothing about the rate of convergence. For rates, one turns to concentration inequalities (Hoeffding, Bernstein, sub-Gaussian bounds) and large deviation theory (Cramér's theorem), which quantify the exponential decay of P(|X̄ₙ − μ| > ε) for bounded or light-tailed random variables.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why the Strong Law of Large Numbers is a strictly stronger result than the Weak Law of Large Numbers, even though both require only E[|X₁|] < ∞ in the i.i.d. setting. In your answer, describe a scenario where convergence in probability holds but almost-sure convergence fails.

Show answer

Almost-sure convergence implies convergence in probability, but not conversely. In the SLLN, we have P(ω : X̄ₙ(ω) → μ) = 1, meaning the set of sample paths that fail to converge has probability zero. In contrast, the WLLN only states that for each fixed ε, the probability of a deviation exceeding ε vanishes, which permits the existence of a non-null set of paths that deviate infinitely often (though with decreasing frequency). A classic counterexample: define Yₙ = 1 with probability 1/n and 0 otherwise (independently). Then Yₙ →^P 0, but by Borel–Cantelli (second lemma, with independence), Yₙ = 1 infinitely often a.s., so Yₙ does not converge to 0 almost surely.

Question 2: BASIC CALCULATION

Let X₁, X₂, … be i.i.d. with E[Xᵢ] = 50 and Var(Xᵢ) = 400. Using Chebyshev's inequality, find the minimum sample size n such that P(|X̄ₙ − 50| ≥ 5) ≤ 0.02.

Show answer

By Chebyshev's inequality, P(|X̄ₙ − 50| ≥ 5) ≤ Var(X̄ₙ)/5² = (400/n)/25 = 16/n. Setting 16/n ≤ 0.02 gives n ≥ 16/0.02 = 800. Therefore, at least n = 800 observations are needed. Note this is a distribution-free bound; if the underlying distribution were known to be normal, the required sample size would be much smaller (approximately 25 using the CLT with z₀.₀₁ = 2.326).

Question 3: INTERMEDIATE

Prove the WLLN for i.i.d. random variables with finite variance σ² using Chebyshev's inequality. Then explain why the Chebyshev-based proof fails to establish the SLLN, and briefly describe what additional tools Kolmogorov's proof employs.

Show answer

Proof: Let X₁, …, Xₙ be i.i.d. with mean μ and variance σ². Then E[X̄ₙ] = μ and Var(X̄ₙ) = σ²/n. For any ε > 0, Chebyshev gives P(|X̄ₙ − μ| ≥ ε) ≤ σ²/(nε²) → 0 as n → ∞, establishing convergence in probability. This proof cannot establish the SLLN because it bounds P(|X̄ₙ − μ| ≥ ε) for each fixed n separately; to get a.s. convergence, one needs these probabilities to be summable (via Borel–Cantelli). The Chebyshev bound σ²/(nε²) is O(1/n), and ∑1/n diverges. Kolmogorov's proof works along a subsequence nₖ = k⁴ (where the fourth-moment bound gives summability), proves a.s. convergence along that subsequence, and then interpolates using the maximal inequality to fill in the gaps.

Question 4: APPLIED

A Monte Carlo simulation estimates the integral I = ∫₀¹

e^{−x²}

dx by sampling n independent uniform random variables U₁, …, Uₙ on [0,1] and computing Îₙ = (1/n)∑ᵢ₌₁ⁿ e^(−Uᵢ²). (a) State which law guarantees Îₙ → I and verify that its assumptions hold. (b) Compute Var(Îₙ) in terms of n and the second moment of

e^{−U²}

. (c) If each evaluation costs 0.001 seconds, estimate the time needed so that a Chebyshev-based 99% confidence bound on |Îₙ − I| is within 0.001 of I. Use the fact that E[

e^{−2U²}

] ≈ 0.6267 and I ≈ 0.7468.

Show answer

(a) The SLLN guarantees Îₙ → I a.s. because f(U) =

e^{−U²}

is bounded on [0,1], hence E[|f(U)|] < ∞. (b) Var(

e^{−U²}

) = E[

e^{−2U²}

] − (E[

e^{−U²}

])² ≈ 0.6267 − (0.7468)² ≈ 0.6267 − 0.5577 = 0.0690. Thus Var(Îₙ) = 0.0690/n. (c) By Chebyshev: P(|Îₙ − I| ≥ 0.001) ≤ 0.0690/(n × 10⁻⁶) = 69000/n. Setting this ≤ 0.01: n ≥ 6,900,000. At 0.001 seconds per evaluation, this requires approximately 6,900 seconds ≈ 1.92 hours. In practice, one would use the CLT instead of Chebyshev, drastically reducing the required n.

Question 5: CRITICAL THINKING

Consider the Cauchy distribution, which has density f(x) = 1/(π(1 + x²)). Show that the sample mean X̄ₙ of i.i.d. Cauchy random variables does not converge to any constant in probability. What does this tell us about the necessity of the finite first moment assumption in the LLN? How does this relate to the Cauchy distribution's characteristic function?

Show answer

The Cauchy distribution has no finite mean: E[|X|] = ∫|x|/(π(1+x²))dx = ∞ because the integrand decays as 1/|x| near infinity. A key property of the Cauchy distribution is that if X₁, …, Xₙ are i.i.d. Cauchy(0,1), then X̄ₙ is itself Cauchy(0,1)—the distribution of the sample mean does not concentrate at all as n grows. This can be seen from the characteristic function: φ_X(t) =

e^{−|t|}

, so φ_{X̄ₙ}(t) = [φ_X(t/n)]ⁿ = [

e^{−|t|/n}

]ⁿ =

e^{−|t|}

, which is again the characteristic function of a standard Cauchy. Since the distribution of X̄ₙ is identical for all n, there can be no convergence to a constant. This example demonstrates that E[|X₁|] < ∞ is not merely a technical convenience but is necessary for the LLN: without it, averaging does not produce concentration, and the entire statistical enterprise of estimation via sample means fails.

SUMMARY

Summary — Law of Large Numbers

Summary

The Law of Large Numbers guarantees that the sample mean X̄ₙ of i.i.d. random variables converges to the population mean μ as the sample size n grows. The Weak Law (WLLN) establishes convergence in probability, asserting that for any tolerance ε, the probability P(|X̄ₙ − μ| ≥ ε) vanishes. The Strong Law (SLLN) provides the stronger guarantee of almost-sure convergence: P(limₙ→∞ X̄ₙ = μ) = 1. Both forms require only E[|X₁|] < ∞ in the i.i.d. setting, and Kolmogorov showed this condition is both necessary and sufficient for the SLLN.

The Chebyshev inequality proof of the WLLN provides a non-asymptotic bound P(|X̄ₙ − μ| ≥ ε) ≤ σ²/(nε²), useful for determining minimum sample sizes in a distribution-free manner. The LLN sits alongside the Central Limit Theorem: while the LLN identifies the location of the limit, the CLT characterizes the Gaussian fluctuations around it. Through Birkhoff's Ergodic Theorem, the SLLN generalizes to stationary ergodic processes, forming the theoretical backbone of Monte Carlo methods and MCMC algorithms.