Historical Context & Motivation
The intellectual roots of Bayesian inference stretch back to the eighteenth century, when the Reverend Thomas Bayes posed a deceptively simple question: given that a particular event has occurred, what can we infer about the probability of its underlying cause? Unlike the frequentist tradition that would later dominate twentieth-century statistics, Bayes treated probability as a measure of rational belief rather than long-run frequency. This epistemic interpretation of probability opened the door to a powerful reasoning framework in which prior knowledge and observed evidence are combined mathematically to produce updated conclusions.
For much of the twentieth century, Bayesian methods were sidelined by computational intractability—evaluating the required integrals was simply too costly for most practical problems. The resurgence of Bayesian statistics in the late twentieth century was driven by two forces: the philosophical clarity of the Bayesian paradigm and the advent of Markov Chain Monte Carlo (MCMC) algorithms, which made high-dimensional posterior computation feasible. Today, Bayesian inference is indispensable across fields ranging from genomics and epidemiology to machine learning and astrophysics.
The central question that Bayesian inference addresses is both conceptually elegant and practically urgent: How should a rational agent revise beliefs about unknown quantities in light of new evidence? Understanding the mechanics of this revision—through prior distributions, likelihood functions, and posterior distributions connected by Bayes' rule—is the subject of this lesson.
Core Principles & Definitions
Bayesian inference rests on a small set of foundational ideas that together compose a complete system for learning from data. Before diving into the mathematics, it is essential to understand the role each component plays and how they interrelate. Every Bayesian analysis begins with a statement of prior belief, confronts that belief with observed data through a likelihood function, and yields a posterior distribution that synthesizes both sources of information.
Prior Distribution π(θ)
Likelihood Function L(θ | x)
Posterior Distribution π(θ | x)
Marginal Likelihood (Evidence) m(x)
Bayes' Rule as the Engine
Visual Explanation — The Bayesian Update
The following diagram illustrates the core logic of a single Bayesian update. A prior distribution (shown in blue) represents initial beliefs about a parameter θ. After observing data, the likelihood function (shown in violet) indicates which parameter values are most compatible with the evidence. Multiplying these two functions point-wise and renormalizing yields the posterior distribution (shown in cyan), which is narrower and shifted toward the data.
Several features of the diagram deserve emphasis. First, the posterior is always narrower than both the prior and the likelihood alone, because it incorporates information from both sources. Second, the posterior mean θ̂B lies between the prior mean and the maximum likelihood estimate, reflecting a precision-weighted average of the two. As the sample size grows, the likelihood sharpens and the posterior converges toward the MLE, a property known as posterior consistency. In the limit of infinite data, the prior is washed out entirely.
Mathematical Framework
The mathematical backbone of Bayesian inference is Bayes' theorem, a direct consequence of the definition of conditional probability. Let θ ∈ Θ denote an unknown parameter and x = (x₁, …, xₙ) denote observed data. Bayes' theorem expresses the posterior distribution of θ given x as the normalized product of the prior and the likelihood.
Because the marginal likelihood m(x) does not depend on θ, practitioners often write the proportionality form, which is sufficient for identifying the posterior's functional form.
A particularly clean illustration arises in the conjugate setting. When the prior and posterior belong to the same parametric family, the update reduces to simple parameter arithmetic. For instance, if the data are Bernoulli with unknown success probability θ and we place a Beta(α, β) prior on θ, then after observing s successes in n trials, the posterior is Beta(α + s, β + n − s). The prior-to-posterior update involves nothing more than adding counts.
Detailed Breakdown — Choosing the Prior
The choice of prior distribution is often the most debated aspect of Bayesian analysis. Priors range from highly informative encodings of expert knowledge to diffuse specifications that aim to let the data speak. Understanding the taxonomy of priors—and the trade-offs each entails—is essential for responsible Bayesian practice.
An informative prior concentrates probability mass in a specific region of the parameter space, reflecting prior studies or expert judgment. For instance, a clinical researcher who knows from previous trials that a drug's effect size is roughly 0.3 might set a prior centered tightly around that value. A weakly informative prior is broad enough to accommodate many plausible parameter values but still prevents pathological extremes—Gelman's recommended half-Cauchy or half-Normal priors for variance parameters are canonical examples. Finally, a non-informative prior (such as Jeffreys' prior, which is invariant to reparameterization) attempts to let the data dominate entirely, though such priors can be improper and must be handled with care.
Worked Example — Beta–Binomial Model
Suppose you are estimating the probability θ that a newly manufactured component passes a quality test. Based on historical data from similar products, you believe that θ is likely around 0.7, so you encode this belief as a Beta(7, 3) prior (prior mean = 7/10 = 0.70). You then test n = 20 components and observe s = 16 successes. What is the posterior distribution of θ?
Bayesian vs. Frequentist Inference
Understanding Bayesian inference is enriched by contrasting it with the frequentist paradigm that dominated statistics for most of the twentieth century. Both frameworks share the same mathematical foundation—probability theory—but they diverge sharply in how they interpret probability and structure inference about unknown parameters.
| Dimension | Bayesian | Frequentist |
|---|---|---|
| Probability | Degree of belief (epistemic); can be assigned to parameters. | Long-run frequency; applies only to repeatable events, not parameters. |
| Parameters | Random variables with distributions reflecting uncertainty. | Fixed but unknown constants; distributions describe data, not parameters. |
| Prior Information | Formally incorporated via the prior distribution π(θ). | Not formally part of the inference; may influence study design. |
| Interval Estimation | Credible interval: P(θ ∈ C | x) = 0.95. Direct probability statement. | Confidence interval: covers θ in 95% of repeated samples. Not about this particular interval. |
| Hypothesis Testing | Posterior probability of hypothesis; Bayes factors for model comparison. | p-values; rejection at significance level α; power analysis. |
| Computation | Often requires MCMC, variational inference, or quadrature for complex models. | Typically relies on closed-form estimators, asymptotic theory, bootstrapping. |
Connection to Advanced Bayesian Theory
The basics of prior, likelihood, and posterior form the foundation upon which a rich edifice of modern Bayesian theory is built. Several advanced topics extend the core framework in important directions, and awareness of these extensions motivates deeper study.
| Basic Concept | Advanced Extension | Key Idea |
|---|---|---|
| Single-parameter Bayes' rule | Hierarchical / Multilevel Models | Parameters themselves have parameters (hyperparameters) with their own priors, enabling partial pooling across groups. |
| Conjugate posteriors | MCMC & Variational Inference | When conjugacy fails, simulation (Gibbs, HMC) or optimization-based (ADVI, normalizing flows) methods approximate the posterior. |
| Marginal likelihood m(x) | Bayesian Model Selection | Bayes factors B₁₂ = m₁(x)/m₂(x) provide a principled metric for comparing models, automatically penalizing complexity via Occam's razor. |
| Point estimates (posterior mean) | Decision Theory | The optimal estimator depends on the loss function; the posterior mean minimizes squared error loss, the posterior median minimizes absolute loss. |
| Credible intervals | Posterior Predictive Checks | Simulating replicated data from the posterior predictive distribution p(x̃ | x) = ∫ f(x̃ | θ) π(θ | x) dθ provides model adequacy diagnostics. |
A particularly important result is the Bernstein–von Mises theorem, which establishes that under regularity conditions, the posterior distribution concentrates around the true parameter value and is asymptotically normal as n → ∞. This theorem provides a bridge between Bayesian and frequentist inference: in large samples, Bayesian credible intervals and frequentist confidence intervals coincide, and the influence of the prior vanishes. However, in finite samples and high-dimensional settings, the choice of prior remains consequential, and careful Bayesian modeling offers advantages that purely frequentist methods cannot replicate.
Practice Problems
Summary
Bayesian inference provides a coherent, probabilistic framework for learning about unknown parameters from data. The process begins with a prior distribution π(θ) that encodes pre-data beliefs, which is combined with the likelihood function f(x | θ)—a measure of how well each parameter value explains the observed data—to produce the posterior distribution π(θ | x) via Bayes' rule: posterior ∝ likelihood × prior. The marginal likelihood m(x) serves as the normalizing constant and plays a pivotal role in model comparison through Bayes factors.
In conjugate families (e.g., Beta–Binomial, Normal–Normal), the posterior has a closed-form update rule, making computation straightforward. For more complex models, MCMC methods and variational inference approximate the posterior numerically. The choice of prior—informative, weakly informative, or non-informative—can materially affect conclusions in small samples but is washed out as data accumulate, a property guaranteed by the Bernstein–von Mises theorem. By producing full posterior distributions rather than single point estimates, Bayesian inference offers direct probabilistic answers to inferential questions—credible intervals, posterior probabilities of hypotheses, and posterior predictive distributions—that are often more intuitive and actionable than their frequentist counterparts.