Conditional Probability & Expectation — Conditional probability and conditional expectation (intro)

The idea that new information should systematically update our beliefs about uncertain events is arguably the most powerful conceptual tool in probability theory. Long before any formal axiomatization, mathematicians and philosophers grappled with a deceptively simple question: if we learn that some event has occurred, how should our assessment of other events change? The resolution of this question gave rise to conditional probability and, ultimately, to conditional expectation — constructs that form the backbone of modern statistical inference, stochastic processes, and Bayesian analysis.

The earliest seeds of conditional reasoning appeared in games of chance during the 17th century, when correspondents like Pascal and Fermat debated how to divide stakes in an interrupted game given what had already transpired. Later, Thomas Bayes posed his famous problem of inverse probability — inferring the likelihood of causes given observed effects — which was published posthumously in 1763. This line of thinking crystallized into a formal definition only after Kolmogorov laid the measure-theoretic groundwork in the 1930s, providing the rigorous foundation upon which conditional expectation as a random variable (rather than a mere number) could be constructed.

The central question this lesson addresses is: given a probability space (Ω, ℱ, P) and information that some event B has occurred (or more generally, that the value of some random variable has been observed), how do we rigorously update both probabilities of events and expected values of random variables? Answering this question requires understanding conditional probability as a ratio, extending it to conditional distributions, and ultimately constructing conditional expectation as a random variable measurable with respect to the conditioning information.

Before diving into formulas and measure-theoretic subtleties, it is essential to internalize the foundational ideas that govern conditional probability and conditional expectation. These concepts are not merely computational recipes; they embody a philosophy of belief revision under partial information, and every advanced topic — from Bayesian inference to filtering theory — rests on the principles articulated here.

The most intuitive way to grasp conditional probability is through a Venn diagram that shows how conditioning on event B transforms the probability landscape. In the diagram below, the full sample space Ω is represented by the outer rectangle. Events A and B are overlapping regions. The conditional probability P(A | B) is simply the proportion of B's area that also belongs to A — the intersection A ∩ B relative to B, not relative to the whole of Ω.

Notice the conceptual shift: unconditional probability P(A) measures A against all of Ω, while conditional probability P(A | B) measures A only within B. Geometrically, conditioning shrinks the reference set. This simple idea has profound implications — it is why, for instance, the probability of a positive medical test being a true positive depends critically on the prevalence of the disease (the conditioning event). When B has small probability, even a highly accurate test can yield a conditional probability of disease that is surprisingly low, a fact that Bayes' rule makes precise.

We now formalize the definitions, beginning with the elementary ratio definition of conditional probability, proceeding through Bayes' theorem, and then extending to conditional expectation — first in the discrete case and then via the abstract (measure-theoretic) definition that handles continuous and general random variables.

In the continuous setting, conditional expectation given Y = y is defined through the conditional density: E[X | Y = y] = ∫ x · $f_{X|Y}$(x | y) dx, where $f_{X|Y}$(x | y) = $f_{X,Y}$(x, y) / $f_Y$(y). The fully abstract version — conditioning on a sub-σ-algebra 𝒢 — defines E[X | 𝒢] as the unique (a.s.) 𝒢-measurable random variable Z satisfying ∫_G Z dP = ∫_G X dP for every G ∈ 𝒢. This definition, guaranteed by the Radon–Nikodým theorem, is the cornerstone of advanced probability and will be explored in detail in subsequent lessons.

Conditional probability and expectation take different forms depending on what kind of object we condition on and the nature of the random variables involved. Understanding this taxonomy is crucial because each setting demands its own technical machinery and carries its own set of pitfalls — most notably the Borel–Kolmogorov paradox, which arises when conditioning on continuous random variables is done carelessly.

Consider two jointly distributed discrete random variables X and Y with the following joint probability mass function. We will compute P(X = 1 | Y = 0), the conditional PMF of X given Y = 1, E[X | Y = 1], and verify the tower property E[E[X | Y]] = E[X].

Understanding when to use conditional versus unconditional quantities, and appreciating the trade-offs, is essential for applied probability and statistics. Conditional reasoning is more powerful but introduces additional complexity — both computational (you must know joint distributions or likelihoods) and conceptual (the conditioning variable itself may be random, leading to random expectations that require further averaging).

The introductory definitions of conditional probability and conditional expectation presented here serve as stepping stones to far deeper constructions. In advanced probability theory, conditioning on a σ-algebra replaces the naive ratio definition and enables conditioning on events of probability zero. This leap is not merely pedantic — it is the foundation upon which martingale theory, Markov process theory, and stochastic filtering rest.

A particularly important generalization is the concept of a martingale: a stochastic process (Mₙ) adapted to a filtration (ℱₙ) such that E[Mₙ₊₁ | ℱₙ] = Mₙ. This is an iterated conditional expectation condition that captures the idea of a 'fair game' — the best prediction of the future, given the present, is the present value. The optional stopping theorem, convergence theorems for martingales, and the pricing of financial derivatives all flow from this single conditional expectation identity.