Convergence Mode Relationships — Relationships among convergence modes (intro) - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The study of convergence in probability theory evolved over more than a century as mathematicians sought to formalize the behavior of sequences of random variables. Early results in the law of large numbers demonstrated that sample averages converge to population means, but the precise meaning of "converge" remained ambiguous until the twentieth century brought measure-theoretic rigor to the field. Different notions of convergence arose organically from distinct mathematical questions—some from analysis, some from statistics, and some from the need to characterize weak limits in functional spaces. The relationships among these convergence modes became one of the most fundamental topics in advanced probability, clarifying which results imply others and under what additional conditions one can upgrade a weaker form of convergence to a stronger one.

1713 • Bernoulli's Weak Law

Jacob Bernoulli posthumously published his proof that sample proportions converge to the true probability, an early form of convergence in probability, though the formal measure-theoretic definition would come much later.

1909 • Borel's Strong Law

Émile Borel proved the first strong law of large numbers for Bernoulli trials, introducing the concept of almost sure convergence and distinguishing it from convergence in probability.

1933 • Kolmogorov's Axiomatization

Andrey Kolmogorov's Grundbegriffe der Wahrscheinlichkeitsrechnung placed probability on a rigorous measure-theoretic foundation, enabling precise definitions of all convergence modes and their interrelationships.

1940s–1960s • Systematic Classification

Researchers including Riesz, Slutsky, and Billingsley systematically mapped out the implications and non-implications among convergence modes, establishing the hierarchy diagram central to modern probability courses.

The central question motivating this lesson is deceptively simple: if we know that a sequence of random variables converges in one sense, what can we infer about convergence in other senses? The answer reveals a rich hierarchy, with almost sure convergence and Lᵖ convergence both implying convergence in probability, which in turn implies convergence in distribution—but with none of these arrows being reversible in general without additional conditions.

SECTION 2

Core Definitions & Foundational Principles

Before we can understand the relationships among convergence modes, we must establish precise definitions for each one. Consider a probability space (Ω, ℱ, P) and a sequence of random variables {Xₙ} together with a target random variable X defined on this space. Each convergence mode captures a different sense in which the Xₙ become "close" to X, and these differ in strength depending on whether they control individual sample-path behavior, moment behavior, or distributional behavior.

Almost Sure (a.s.) Convergence

Xₙ → X a.s. if P({ω ∈ Ω : Xₙ(ω) → X(ω)}) = 1. Equivalently, the set of sample points where pointwise convergence fails has probability zero. This is the strongest standard mode, controlling path-by-path behavior.

Convergence in Lᵖ (Mean)

Xₙ → X in Lᵖ (for p ≥ 1) if E[|Xₙ − X|ᵖ] → 0 as n → ∞. When p = 1 this is convergence in mean; when p = 2 it is mean-square convergence. Lᵖ convergence controls the integrated deviation via moments.

Convergence in Probability

Xₙ →ᴾ X if for every ε > 0, P(|Xₙ − X| > ε) → 0 as n → ∞. This requires only that large deviations become unlikely, without demanding path-by-path or moment control.

Convergence in Distribution

Xₙ →ᵈ X if Fₙ(x) → F(x) at every continuity point x of F, where Fₙ and F are the CDFs. This is the weakest mode—it only requires that the distribution functions align, and the random variables need not live on the same probability space.

✦

KEY TAKEAWAY

Think of the convergence modes as different levels of surveillance on a factory assembly line. Almost sure convergence is like inspecting every single product on the line and confirming nearly all of them approach the target specification. Lᵖ convergence checks that the average defect magnitude across all products vanishes. Convergence in probability only verifies that the proportion of defective products shrinks to zero. Convergence in distribution merely checks that the histogram of product measurements looks like the target distribution—without tracking individual items at all.

SECTION 3

The Convergence Hierarchy Diagram

The relationships among the four principal convergence modes are most effectively communicated through a directed implication diagram. Each arrow in the diagram below signifies a one-way logical implication: if the convergence mode at the tail of the arrow holds, then the convergence mode at the head must also hold. Critically, the absence of an arrow between two modes means that neither implies the other in general, and crossed arrows or annotations indicate that the converse fails without additional hypotheses.

The hierarchy shows that almost sure convergence and Lᵖ convergence each independently imply convergence in probability, which in turn implies convergence in distribution. The dashed line between a.s. and Lᵖ indicates that neither implies the other without additional conditions. Converse implications (marked ✗) fail in general but can be recovered with supplementary hypotheses such as uniform integrability or a subsequence argument.

Notice the diamond-like structure of the diagram. The two "strong" modes—almost sure and Lᵖ—sit at the top, neither dominating the other. Both funnel into convergence in probability as a common intermediate, which then implies the weakest mode, convergence in distribution. This structure means that when you establish almost sure convergence (as in the strong law of large numbers), you automatically obtain convergence in probability and in distribution for free. Conversely, proving only convergence in distribution (as in the central limit theorem) tells you nothing about almost sure or Lᵖ behavior without additional work.

SECTION 4

Mathematical Framework of Implications

We now formalize the three main implication arrows in the hierarchy. Each proof leverages a classic inequality or a measure-theoretic argument, and understanding the proof structure reveals precisely where each implication derives its power—and why the converses fail.

Implication 1: Almost Sure ⟹ In Probability

ALMOST SURE CONVERGENCE DEFINITION

P( lim_{n→∞} Xₙ = X ) = 1

This means P({ω : |Xₙ(ω) − X(ω)| > ε infinitely often}) = 0 for every ε > 0.

To show a.s. convergence implies convergence in probability, fix ε > 0 and define Aₙ = {ω : |Xₙ(ω) − X(ω)| > ε}. Almost sure convergence implies that for almost every ω, Xₙ(ω) → X(ω), so the indicator 𝟙_{Aₙ}(ω) → 0 a.s. By the dominated convergence theorem (since $𝟙_{Aₙ}$ ≤ 1), we obtain E[𝟙_{Aₙ}] = P(Aₙ) → 0, which is precisely the definition of convergence in probability.

Implication 2: Lᵖ ⟹ In Probability

MARKOV / CHEBYSHEV BOUND

P(|Xₙ − X| > ε) = P(|Xₙ − X|ᵖ > εᵖ) ≤ E[|Xₙ − X|ᵖ] / εᵖ → 0

This is a direct application of Markov's inequality. If E[|Xₙ − X|ᵖ] → 0, then the right-hand side vanishes, yielding P(|Xₙ − X| > ε) → 0.

Implication 3: In Probability ⟹ In Distribution

CDF CONVERGENCE BOUND

Fₙ(x) = P(Xₙ ≤ x) ≤ P(X ≤ x + ε) + P(|Xₙ − X| > ε)

For any continuity point x of F and any ε > 0, this decomposition shows Fₙ(x) is squeezed toward F(x) as n → ∞ since P(|Xₙ − X| > ε) → 0. A symmetric lower bound completes the sandwich argument.

⚠

Why Converses Fail

Convergence in distribution does not even require the random variables to be on the same probability space, so it cannot imply convergence in probability. Convergence in probability allows for rare but extreme excursions, so it cannot imply a.s. convergence (which demands path-by-path control) or Lᵖ convergence (which demands moment control). However, partial converses exist: convergence in probability implies the existence of an almost surely convergent subsequence, and convergence in distribution to a constant implies convergence in probability.

SECTION 5

Counterexamples & Boundary Cases

Understanding why certain implications fail is just as important as knowing which ones hold. The standard counterexamples are carefully constructed sequences of random variables that converge in one mode but fail to converge in another. These examples serve as indispensable diagnostic tools: when confronted with a new sequence of random variables, recognizing the telltale patterns of these counterexamples helps determine which convergence modes can be expected.

Left panel: the typewriter (or sliding-indicator) sequence on [0,1], where indicators sweep across the unit interval in cycles of decreasing length. P(Xₙ = 1) → 0 (convergence in probability), but every ω is visited infinitely often, so a.s. convergence fails. Right panel: the spike sequence where the height grows as n while the support shrinks as 1/n, so E[Xₙ] stays constant even as P(Xₙ ≠ 0) → 0. The bottom table catalogs all standard non-implications with their canonical counterexamples.

The typewriter sequence (also called the sliding-bump or roving-indicator sequence) is perhaps the most instructive counterexample in this subject. On the probability space ([0,1], Borel sets, Lebesgue measure), define X₁ = 𝟙[0,1), X₂ = 𝟙[0,1/2), X₃ = 𝟙[1/2,1), X₄ = 𝟙[0,1/3), and so on—each successive "wave" partitions [0,1] into finer intervals. For any fixed ε > 0, the Lebesgue measure of the support of Xₙ tends to zero, so Xₙ →ᴾ 0. Yet for every ω ∈ [0,1], the sequence Xₙ(ω) equals 1 infinitely often, so pointwise convergence—and hence almost sure convergence—fails everywhere. This elegantly demonstrates the gap between controlling the probability of deviation and controlling the sample-path behavior.

SECTION 6

Worked Example: Verifying Convergence Modes

Let us consider a concrete sequence of random variables and systematically determine which convergence modes hold. Define U ~ Uniform(0,1) and set Xₙ = n² · 𝟙(U < 1/n) for n = 1, 2, 3, … Our goal is to check convergence to X = 0 in each of the four modes.

Determining Convergence Modes for Xₙ = n² · 𝟙(U < 1/n)

1
Step 1 — Check Almost Sure Convergence

For a fixed ω with U(ω) = u > 0, there exists N such that 1/n < u for all n ≥ N. Then Xₙ(ω) = n² · 0 = 0 for all n ≥ N. Since U = 0 has probability zero, we conclude Xₙ(ω) → 0 for almost every ω.

✓ Xₙ → 0 almost surely
2
Step 2 — Conclude Convergence in Probability

Since a.s. convergence implies convergence in probability (by the first implication in the hierarchy), we immediately deduce Xₙ →ᴾ 0 without any additional calculation.

✓ Xₙ →ᴾ 0 (by implication from a.s.)
3
Step 3 — Conclude Convergence in Distribution

Convergence in probability implies convergence in distribution, so Xₙ →ᵈ 0 follows automatically. Alternatively, note Xₙ converges in probability to the constant 0, confirming it.

✓ Xₙ →ᵈ 0 (by implication from probability)
4
Step 4 — Check L¹ Convergence

Compute E[|Xₙ − 0|] = E[n² · 𝟙(U < 1/n)] = n² · P(U < 1/n) = n² · (1/n) = n. As n → ∞, E[|Xₙ|] = n → ∞, so Xₙ does not converge to 0 in L¹.

✗ Xₙ does NOT converge to 0 in L¹ (E[|Xₙ|] → ∞)
5
Step 5 — Interpret the Results

This example confirms that almost sure convergence does not imply Lᵖ convergence. Despite path-by-path convergence to 0, the rare events {U < 1/n} produce spikes of height n² that are tall enough to keep the mean diverging. This illustrates precisely why a.s. and Lᵖ sit in parallel branches of the hierarchy, with neither dominating the other.

a.s. ✓, in prob. ✓, in dist. ✓, in L¹ ✗ — demonstrates a.s. ⇏ Lᵖ

SECTION 7

When Converses Hold: Sufficient Conditions

While the converses of the three main implications fail in general, probability theory provides well-known additional conditions under which they can be recovered. These "conditional converses" are among the most practically important results in the subject, because applied problems frequently satisfy the extra hypotheses needed. The table below collects the key results.

Desired Upgrade	Additional Condition	Key Theorem / Result
In prob. → a.s.	Fast enough rate: Σ P(\|Xₙ − X\| > ε) < ∞	Borel–Cantelli lemma (first)
In prob. → a.s. (subsequence)	None—always holds	Every sequence converging in probability has a subsequence converging a.s.
In prob. → Lᵖ	Uniform integrability of {\|Xₙ − X\|ᵖ}	Vitali convergence theorem
In dist. → in prob.	Limit X = c is a constant	Convergence in distribution to a constant implies convergence in probability to that constant
a.s. → Lᵖ	Dominated: \|Xₙ\| ≤ Y with E[Yᵖ] < ∞	Dominated convergence theorem

Sufficient conditions for upgrading weaker convergence to stronger convergence

✦

KEY TAKEAWAY

In applied statistics and machine learning, these conditional converses matter enormously. When establishing the consistency of an estimator, one typically proves convergence in probability first (the weak law route) and then upgrades to almost sure convergence by showing the Borel–Cantelli summability condition holds—often by bounding the tail probabilities using moment inequalities. Similarly, uniform integrability is the standard tool for converting convergence in probability into Lᵖ convergence, which is essential when one needs to interchange limits and expectations—a ubiquitous requirement in Bayesian asymptotics and M-estimation theory.

SECTION 8

Connections to Advanced Theory

The introductory hierarchy of convergence modes extends into deeper territory in several directions. Understanding these connections motivates further study and reveals why the basic relationships matter so much in modern probability and statistics.

Introductory Concept	Advanced Extension	Why It Matters
Convergence in distribution	Weak convergence of measures on metric spaces; Prokhorov's theorem; Skorokhod's representation	Extends distributional convergence to random elements of general spaces (e.g., stochastic processes via Donsker's theorem)
a.s. convergence	Martingale convergence theorems; ergodic theorems	Provides a.s. convergence under structural (martingale, ergodic) assumptions rather than independence
Lᵖ convergence	Uniform integrability and Lᵖ-bounded martingales	Vitali's theorem gives the precise equivalence: convergence in probability + UI ⟺ Lᵖ convergence
Subsequence a.s. extraction	Skorokhod representation theorem	If Xₙ →ᵈ X, there exist copies X̃ₙ =ᵈ Xₙ on a common space with X̃ₙ → X̃ a.s.; bridges distributional and a.s. convergence

Bridges from introductory convergence theory to advanced topics

In empirical process theory and high-dimensional statistics, the convergence mode hierarchy takes on new dimensions. The continuous mapping theorem and Slutsky's theorem provide algebraic rules for combining convergent sequences, and these rules depend critically on which mode of convergence applies. For instance, Slutsky's theorem requires one sequence to converge in distribution and the other to converge in probability to a constant—a combination that leverages the hierarchy in a precise and non-interchangeable way. As you move into the study of functional central limit theorems and delta-method expansions, fluency with these mode relationships becomes indispensable.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why convergence in distribution is strictly weaker than convergence in probability. Specifically, why is it possible for Xₙ →ᵈ X without Xₙ →ᴾ X? Give an intuitive explanation and a concrete example.

Show answer

Convergence in distribution only constrains the CDFs of the Xₙ to approach the CDF of X; it says nothing about the joint behavior of Xₙ and X. In particular, Xₙ and X need not even be defined on the same probability space. A concrete example: let X ~ N(0,1) and define Xₙ = −X for all n. Then Xₙ ~ N(0,1) = X in distribution, so trivially Xₙ →ᵈ X. However, |Xₙ − X| = |−X − X| = 2|X|, and P(2|X| > ε) > 0 for any ε > 0, so convergence in probability fails. The key insight is that distributional convergence tracks marginal distributions only, while convergence in probability tracks the joint discrepancy |Xₙ − X|.

Question 2: BASIC CALCULATION

Let U ~ Uniform(0,1) and define Xₙ = (1 + 1/n) · 𝟙(U ≤ 1 − 1/n). Show that Xₙ → 1 a.s., and use the hierarchy to deduce which other convergence modes hold.

Show answer

For any fixed ω with U(ω) = u < 1, there exists N such that u < 1 − 1/n for all n ≥ N. Then Xₙ(ω) = 1 + 1/n → 1. The only exceptional point is U(ω) = 1, which has probability 0 under Lebesgue measure. Hence Xₙ → 1 a.s. By the hierarchy: a.s. convergence implies convergence in probability (Xₙ →ᴾ 1), which in turn implies convergence in distribution (Xₙ →ᵈ 1). Furthermore, since |Xₙ − 1| ≤ 1 + 1/n ≤ 2 and E[|2|ᵖ] < ∞ for all p, the dominated convergence theorem gives Xₙ → 1 in Lᵖ for every p ≥ 1 as well. So all four modes hold.

Question 3: INTERMEDIATE

Let Xₙ = √n · 𝟙(U < 1/n) where U ~ Uniform(0,1). Determine which convergence modes hold for Xₙ → 0. Compute E[|Xₙ|] and E[|Xₙ|²] explicitly.

Show answer

For a.s. convergence: fix ω with U(ω) = u > 0. For n > 1/u, Xₙ(ω) = 0, so Xₙ(ω) → 0 for all ω except U = 0 (a null set). Hence Xₙ → 0 a.s., which implies Xₙ →ᴾ 0 and Xₙ →ᵈ 0 by the hierarchy. For L¹: E[|Xₙ|] = √n · (1/n) = 1/√n → 0, so Xₙ → 0 in L¹. For L²: E[|Xₙ|²] = n · (1/n) = 1 ↛ 0, so Xₙ does not converge to 0 in L². This example shows that Lᵖ convergence can hold for one value of p but fail for larger p—the convergence mode depends on the order of the moment.

Question 4: APPLIED

In a statistical estimation context, let θ̂ₙ be the maximum likelihood estimator of θ based on n i.i.d. observations. You have established that √n(θ̂ₙ − θ) →ᵈ N(0, I(θ)⁻¹) by the CLT. A colleague claims this implies θ̂ₙ → θ almost surely. Is this claim correct? Explain using the convergence hierarchy, and state what additional result is actually needed to conclude a.s. convergence of θ̂ₙ to θ.

Show answer

The claim is not directly justified by the hierarchy. The CLT gives √n(θ̂ₙ − θ) →ᵈ N(0, I(θ)⁻¹), which implies that θ̂ₙ − θ = Oₚ(1/√n) → 0 in probability (since the limit distribution, when divided by √n, degenerates to 0). But convergence in probability does not imply a.s. convergence in general. To establish θ̂ₙ → θ a.s., one needs a separate argument—typically the strong law of large numbers applied to the score equation, or a direct Borel–Cantelli argument showing Σₙ P(|θ̂ₙ − θ| > ε) < ∞ for all ε > 0. In the MLE context, almost sure consistency is indeed true under standard regularity conditions, but it requires additional proof beyond the asymptotic normality result.

Question 5: CRITICAL THINKING

Prove or disprove: if Xₙ → 0 in Lᵖ for all p ≥ 1, then Xₙ → 0 almost surely. If the statement is false, construct an explicit counterexample and explain why even controlling all moments simultaneously is insufficient to guarantee path-by-path convergence.

Show answer

The statement is false. Consider the typewriter sequence on [0,1] with Lebesgue measure: enumerate intervals Iₙ = [jₙ/kₙ, (jₙ+1)/kₙ) cycling through partitions of [0,1] with kₙ → ∞, and set Xₙ = 𝟙_{Iₙ}. Then E[|Xₙ|ᵖ] = λ(Iₙ) = 1/kₙ → 0 for every p ≥ 1, so Xₙ → 0 in Lᵖ for all p. However, for every ω ∈ [0,1], Xₙ(ω) = 1 for infinitely many n (because the intervals sweep across [0,1] repeatedly), so lim sup Xₙ(ω) = 1 everywhere. Hence Xₙ does not converge to 0 at any ω, and a.s. convergence fails completely. The fundamental reason is that Lᵖ convergence controls the aggregate size of the set where Xₙ deviates, but it does not prevent the deviations from persistently revisiting every sample point. Almost sure convergence requires eventual settling down along each path, which is a qualitatively different (non-average) condition that moment control alone cannot enforce.

SUMMARY

Summary & Key Takeaways

Summary

The four principal convergence modes for sequences of random variables form a well-defined hierarchy. Almost sure convergence (path-by-path control) and Lᵖ convergence (moment control) both independently imply convergence in probability (the probability of large deviations vanishes), which in turn implies convergence in distribution (CDF pointwise convergence at continuity points). Neither a.s. nor Lᵖ convergence implies the other without additional conditions. The proofs of the forward implications rely on the dominated convergence theorem, Markov's inequality, and a CDF sandwich argument, respectively.

The converses fail in general, as demonstrated by canonical counterexamples: the typewriter sequence (in prob. ⇏ a.s.) and the spike sequence (in prob. ⇏ Lᵖ). However, converses can be recovered under additional hypotheses: the Borel–Cantelli lemma upgrades convergence in probability to a.s. when tail probabilities are summable; uniform integrability upgrades convergence in probability to Lᵖ via the Vitali convergence theorem; and convergence in distribution to a constant automatically implies convergence in probability. Mastery of this hierarchy and its boundary cases is foundational for all subsequent work in limit theorems, asymptotic statistics, and stochastic processes.