Bayesian Inference Basics — Prior, likelihood, posterior, and Bayes' rule - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

The intellectual roots of Bayesian inference stretch back to the eighteenth century, when the Reverend Thomas Bayes posed a deceptively simple question: given that a particular event has occurred, what can we infer about the probability of its underlying cause? Unlike the frequentist tradition that would later dominate twentieth-century statistics, Bayes treated probability as a measure of rational belief rather than long-run frequency. This epistemic interpretation of probability opened the door to a powerful reasoning framework in which prior knowledge and observed evidence are combined mathematically to produce updated conclusions.

For much of the twentieth century, Bayesian methods were sidelined by computational intractability—evaluating the required integrals was simply too costly for most practical problems. The resurgence of Bayesian statistics in the late twentieth century was driven by two forces: the philosophical clarity of the Bayesian paradigm and the advent of Markov Chain Monte Carlo (MCMC) algorithms, which made high-dimensional posterior computation feasible. Today, Bayesian inference is indispensable across fields ranging from genomics and epidemiology to machine learning and astrophysics.

1763 • Bayes' Essay Published Posthumously

Richard Price edits and presents Thomas Bayes' An Essay towards solving a Problem in the Doctrine of Chances to the Royal Society, introducing the concept of inverse probability for the first time.

1812 • Laplace Generalizes Bayes' Rule

Pierre-Simon Laplace independently derives and extends the theorem in Théorie analytique des probabilités, applying it to astronomical observations and establishing the mathematical formulation we use today.

1950s • Decision-Theoretic Foundations

Leonard Jimmie Savage and Bruno de Finetti provide rigorous axiomatic foundations for subjective probability, legitimizing the Bayesian framework as a coherent theory of decision-making under uncertainty.

1990s • MCMC Revolution

The Gibbs sampler and Metropolis–Hastings algorithms become widely accessible, enabling practitioners to compute posterior distributions for complex hierarchical models that were previously intractable.

2010s–Present • Probabilistic Programming

Frameworks such as Stan, PyMC, and TensorFlow Probability allow users to specify Bayesian models in high-level code, automating inference via Hamiltonian Monte Carlo and variational methods.

The central question that Bayesian inference addresses is both conceptually elegant and practically urgent: How should a rational agent revise beliefs about unknown quantities in light of new evidence? Understanding the mechanics of this revision—through prior distributions, likelihood functions, and posterior distributions connected by Bayes' rule—is the subject of this lesson.

SECTION 2

Core Principles & Definitions

Bayesian inference rests on a small set of foundational ideas that together compose a complete system for learning from data. Before diving into the mathematics, it is essential to understand the role each component plays and how they interrelate. Every Bayesian analysis begins with a statement of prior belief, confronts that belief with observed data through a likelihood function, and yields a posterior distribution that synthesizes both sources of information.

Prior Distribution π(θ)

Encodes the analyst's beliefs about the parameter before observing any data. It may reflect genuine domain expertise (informative prior) or deliberate agnosticism (weakly informative or non-informative prior). The prior is a full probability distribution over the parameter space Θ.

Likelihood Function L(θ | x)

Measures how plausible each parameter value is given the observed data x. Formally, it is the sampling distribution f(x | θ) viewed as a function of θ for fixed x. The likelihood is not a probability distribution over θ—it need not integrate to one.

Posterior Distribution π(θ | x)

The updated distribution of θ after observing data x. It combines the prior and the likelihood, representing the complete state of knowledge about θ. All Bayesian inferences—point estimates, credible intervals, hypothesis tests—derive from the posterior.

Marginal Likelihood (Evidence) m(x)

The normalizing constant obtained by integrating the product of the prior and likelihood over the entire parameter space. While often intractable analytically, m(x) = ∫ f(x | θ) π(θ) dθ plays a critical role in model comparison through Bayes factors.

Bayes' Rule as the Engine

Bayes' theorem provides the mechanical link: posterior ∝ likelihood × prior. It is not merely a formula but a normative prescription for how rational belief revision should operate under the axioms of probability.

✦

KEY TAKEAWAY

Think of Bayesian inference as adjusting the dial on a radio. The prior is where you initially set the dial based on a friend's recommendation. The likelihood is the static you hear—data that tells you whether to nudge the dial left or right. The posterior is the final position of the dial after you have incorporated the signal. With more data (a clearer signal), the dial converges toward the true station regardless of where your friend initially pointed you.

SECTION 3

Visual Explanation — The Bayesian Update

The following diagram illustrates the core logic of a single Bayesian update. A prior distribution (shown in blue) represents initial beliefs about a parameter θ. After observing data, the likelihood function (shown in violet) indicates which parameter values are most compatible with the evidence. Multiplying these two functions point-wise and renormalizing yields the posterior distribution (shown in cyan), which is narrower and shifted toward the data.

The prior (blue, solid) is centered at θ₀, the prior mean. The likelihood (violet, dashed) peaks near the MLE θ_MLE. The posterior (cyan, bold) is a compromise, centered at the Bayes estimate θ̂_B between the two, with reduced variance reflecting the information gained from the data.

Several features of the diagram deserve emphasis. First, the posterior is always narrower than both the prior and the likelihood alone, because it incorporates information from both sources. Second, the posterior mean θ̂_B lies between the prior mean and the maximum likelihood estimate, reflecting a precision-weighted average of the two. As the sample size grows, the likelihood sharpens and the posterior converges toward the MLE, a property known as posterior consistency. In the limit of infinite data, the prior is washed out entirely.

SECTION 4

Mathematical Framework

The mathematical backbone of Bayesian inference is Bayes' theorem, a direct consequence of the definition of conditional probability. Let θ ∈ Θ denote an unknown parameter and x = (x₁, …, xₙ) denote observed data. Bayes' theorem expresses the posterior distribution of θ given x as the normalized product of the prior and the likelihood.

BAYES' THEOREM

π(θ | x) = f(x | θ) · π(θ) / m(x)

Where π(θ | x) is the posterior, f(x | θ) is the likelihood, π(θ) is the prior, and m(x) = ∫ f(x | θ) π(θ) dθ is the marginal likelihood (evidence).

Because the marginal likelihood m(x) does not depend on θ, practitioners often write the proportionality form, which is sufficient for identifying the posterior's functional form.

PROPORTIONALITY FORM

π(θ | x) ∝ f(x | θ) · π(θ)

The symbol ∝ means 'proportional to.' The normalizing constant can be recovered by integrating the right-hand side over θ and ensuring the result integrates to one.

A particularly clean illustration arises in the conjugate setting. When the prior and posterior belong to the same parametric family, the update reduces to simple parameter arithmetic. For instance, if the data are Bernoulli with unknown success probability θ and we place a Beta(α, β) prior on θ, then after observing s successes in n trials, the posterior is Beta(α + s, β + n − s). The prior-to-posterior update involves nothing more than adding counts.

BETA–BINOMIAL CONJUGATE UPDATE

θ | x ~ Beta(α + s, β + n − s)

Here s = Σxᵢ is the number of successes, n is the number of trials, and α, β are the prior hyperparameters. The posterior mean is (α + s) / (α + β + n), a weighted average of the prior mean α/(α + β) and the sample proportion s/n.

NORMAL–NORMAL CONJUGATE UPDATE

μ | x ~ N(μ_post, σ²_post) where μ_post = (σ² · μ₀ + n · τ² · x̄) / (σ² + n · τ²)

For data xᵢ ~ N(μ, σ²) with known variance σ² and a N(μ₀, τ²) prior on μ, the posterior mean μ_post is a precision-weighted average of the prior mean μ₀ and the sample mean x̄. The posterior variance σ²_post = (1/τ² + n/σ²)⁻¹ is always smaller than either the prior variance or the sampling variance alone.

SECTION 5

Detailed Breakdown — Choosing the Prior

The choice of prior distribution is often the most debated aspect of Bayesian analysis. Priors range from highly informative encodings of expert knowledge to diffuse specifications that aim to let the data speak. Understanding the taxonomy of priors—and the trade-offs each entails—is essential for responsible Bayesian practice.

A taxonomy of prior distributions from most informative (left) to least informative (right). Informative priors encode specific domain knowledge; weakly informative priors provide gentle regularization; non-informative priors attempt to minimize the prior's influence, sometimes yielding improper distributions that are valid only when the resulting posterior is proper.

An informative prior concentrates probability mass in a specific region of the parameter space, reflecting prior studies or expert judgment. For instance, a clinical researcher who knows from previous trials that a drug's effect size is roughly 0.3 might set a prior centered tightly around that value. A weakly informative prior is broad enough to accommodate many plausible parameter values but still prevents pathological extremes—Gelman's recommended half-Cauchy or half-Normal priors for variance parameters are canonical examples. Finally, a non-informative prior (such as Jeffreys' prior, which is invariant to reparameterization) attempts to let the data dominate entirely, though such priors can be improper and must be handled with care.

⚠

Prior Sensitivity Analysis

Good Bayesian practice demands checking how sensitive the posterior is to the choice of prior. If substantively different priors lead to meaningfully different posteriors, additional data or stronger justification for a particular prior is needed. Tools like prior predictive checks (simulating data from the prior and verifying it looks scientifically plausible) are indispensable for calibrating prior specifications.

SECTION 6

Worked Example — Beta–Binomial Model

Suppose you are estimating the probability θ that a newly manufactured component passes a quality test. Based on historical data from similar products, you believe that θ is likely around 0.7, so you encode this belief as a Beta(7, 3) prior (prior mean = 7/10 = 0.70). You then test n = 20 components and observe s = 16 successes. What is the posterior distribution of θ?

Beta–Binomial Bayesian Update

1
Step 1 — Specify the Prior

We choose θ ~ Beta(α₀, β₀) = Beta(7, 3). The prior mean is α₀/(α₀ + β₀) = 7/10 = 0.70, and the prior standard deviation is approximately √(α₀β₀ / ((α₀ + β₀)²(α₀ + β₀ + 1))) ≈ 0.138. This prior places most mass between 0.5 and 0.9, reflecting moderate certainty that the pass rate is near 70%.

Prior: Beta(7, 3), mean = 0.70
2
Step 2 — Write Down the Likelihood

The data model is x₁, …, x₂₀ iid Bernoulli(θ). The likelihood for the sufficient statistic s = Σxᵢ = 16 successes in n = 20 trials is L(θ | s, n) = C(20,16) · θ¹⁶ · (1 − θ)⁴. For the purpose of the posterior, the binomial coefficient C(20,16) is a constant that cancels in the proportionality, so L(θ | data) ∝ θ¹⁶(1 − θ)⁴.

Likelihood: ∝ θ¹⁶(1 − θ)⁴
3
Step 3 — Compute the Posterior (Conjugate Update)

By conjugacy, the posterior is Beta(α₀ + s, β₀ + n − s) = Beta(7 + 16, 3 + 20 − 16) = Beta(23, 7). No integral computation is necessary—the posterior family is known in closed form from the conjugate relationship.

Posterior: Beta(23, 7)
4
Step 4 — Extract Point Estimates

The posterior mean is α₁/(α₁ + β₁) = 23/30 ≈ 0.767. Compare this with the MLE = s/n = 16/20 = 0.80 and the prior mean = 0.70. As expected, the posterior mean lies between the prior mean and the MLE, closer to the data because n = 20 provides substantial information relative to the prior pseudo-count of α₀ + β₀ = 10.

Posterior mean ≈ 0.767, between prior mean (0.70) and MLE (0.80)
5
Step 5 — Construct a 95% Credible Interval

A 95% equal-tailed credible interval is obtained from the 0.025 and 0.975 quantiles of Beta(23, 7). Using standard software (e.g., qbeta in R or scipy.stats.beta.ppf in Python), we obtain approximately (0.596, 0.902). This interval has a direct probabilistic interpretation: there is a 95% posterior probability that θ falls in this range.

95% Credible Interval: (0.596, 0.902)

💡

Interpretation Note

Unlike a frequentist 95% confidence interval, the Bayesian credible interval admits the interpretation 'there is a 0.95 probability that θ lies in this interval, given the observed data and the chosen prior.' This direct probabilistic statement is one of the chief practical appeals of the Bayesian framework.

SECTION 7

Bayesian vs. Frequentist Inference

Understanding Bayesian inference is enriched by contrasting it with the frequentist paradigm that dominated statistics for most of the twentieth century. Both frameworks share the same mathematical foundation—probability theory—but they diverge sharply in how they interpret probability and structure inference about unknown parameters.

Dimension	Bayesian	Frequentist
Probability	Degree of belief (epistemic); can be assigned to parameters.	Long-run frequency; applies only to repeatable events, not parameters.
Parameters	Random variables with distributions reflecting uncertainty.	Fixed but unknown constants; distributions describe data, not parameters.
Prior Information	Formally incorporated via the prior distribution π(θ).	Not formally part of the inference; may influence study design.
Interval Estimation	Credible interval: P(θ ∈ C \| x) = 0.95. Direct probability statement.	Confidence interval: covers θ in 95% of repeated samples. Not about this particular interval.
Hypothesis Testing	Posterior probability of hypothesis; Bayes factors for model comparison.	p-values; rejection at significance level α; power analysis.
Computation	Often requires MCMC, variational inference, or quadrature for complex models.	Typically relies on closed-form estimators, asymptotic theory, bootstrapping.

Bayesian vs. Frequentist paradigms across key inferential dimensions

✦

KEY TAKEAWAY

The Bayesian and frequentist paradigms are not enemies but complementary lenses. Frequentist methods guarantee long-run operating characteristics (coverage, type-I error control) that are vital in regulatory settings. Bayesian methods excel when prior information is genuinely available, when the research question naturally asks 'what do I believe about θ given the data?', or when the model is complex enough that the posterior distribution is the most natural inferential summary. In modern practice, many statisticians adopt a pragmatic stance, using whichever framework best serves the scientific question at hand.

SECTION 8

Connection to Advanced Bayesian Theory

The basics of prior, likelihood, and posterior form the foundation upon which a rich edifice of modern Bayesian theory is built. Several advanced topics extend the core framework in important directions, and awareness of these extensions motivates deeper study.

Basic Concept	Advanced Extension	Key Idea
Single-parameter Bayes' rule	Hierarchical / Multilevel Models	Parameters themselves have parameters (hyperparameters) with their own priors, enabling partial pooling across groups.
Conjugate posteriors	MCMC & Variational Inference	When conjugacy fails, simulation (Gibbs, HMC) or optimization-based (ADVI, normalizing flows) methods approximate the posterior.
Marginal likelihood m(x)	Bayesian Model Selection	Bayes factors B₁₂ = m₁(x)/m₂(x) provide a principled metric for comparing models, automatically penalizing complexity via Occam's razor.
Point estimates (posterior mean)	Decision Theory	The optimal estimator depends on the loss function; the posterior mean minimizes squared error loss, the posterior median minimizes absolute loss.
Credible intervals	Posterior Predictive Checks	Simulating replicated data from the posterior predictive distribution p(x̃ \| x) = ∫ f(x̃ \| θ) π(θ \| x) dθ provides model adequacy diagnostics.

From basics to advanced Bayesian theory

A particularly important result is the Bernstein–von Mises theorem, which establishes that under regularity conditions, the posterior distribution concentrates around the true parameter value and is asymptotically normal as n → ∞. This theorem provides a bridge between Bayesian and frequentist inference: in large samples, Bayesian credible intervals and frequentist confidence intervals coincide, and the influence of the prior vanishes. However, in finite samples and high-dimensional settings, the choice of prior remains consequential, and careful Bayesian modeling offers advantages that purely frequentist methods cannot replicate.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain in your own words why the posterior distribution is guaranteed to be a proper probability distribution (i.e., integrates to one) even though the likelihood function generally does not. What role does the marginal likelihood m(x) play in ensuring this property?

Show answer

The likelihood f(x | θ) is a function of θ for fixed data x and is not required to integrate to one over θ—it measures relative plausibility, not probability. The posterior π(θ | x) is obtained by multiplying the likelihood by the prior π(θ), which is a proper distribution, and then dividing by the marginal likelihood m(x) = ∫ f(x | θ)π(θ) dθ. This denominator is precisely the integral of the numerator over θ, so the ratio is guaranteed to integrate to one. In essence, m(x) serves as the normalizing constant that converts the unnormalized product 'likelihood × prior' into a legitimate probability density over θ. If the prior is improper (not integrating to one), the posterior may still be proper provided the integral m(x) is finite, but this must be verified.

Question 2: BASIC CALCULATION

You observe x₁, …, x₅ iid Bernoulli(θ) with outcomes {1, 0, 1, 1, 0}, so s = 3 successes in n = 5 trials. Using a Beta(2, 2) prior on θ, find the posterior distribution, its mean, and its mode.

Show answer

By the Beta–Binomial conjugate update, the posterior is Beta(α₀ + s, β₀ + n − s) = Beta(2 + 3, 2 + 5 − 3) = Beta(5, 4). The posterior mean is 5/(5 + 4) = 5/9 ≈ 0.556. The posterior mode (MAP estimate) is (α − 1)/(α + β − 2) = (5 − 1)/(5 + 4 − 2) = 4/7 ≈ 0.571. For comparison, the MLE is s/n = 3/5 = 0.60 and the prior mean is 2/(2 + 2) = 0.50. The posterior mean lies between the prior mean and the MLE, weighted toward the data because n = 5 is comparable to the prior pseudo-count of 4.

Question 3: INTERMEDIATE

Suppose x₁, …, x₁₀ iid N(μ, σ² = 25) with x̄ = 52. You place a N(μ₀ = 50, τ² = 9) prior on μ. Derive the posterior distribution of μ, compute the posterior mean and variance, and construct a 95% equal-tailed credible interval.

Show answer

Using the Normal–Normal conjugate update with known variance: the posterior precision is 1/τ² + n/σ² = 1/9 + 10/25 = 1/9 + 2/5 = 5/45 + 18/45 = 23/45. Thus the posterior variance is σ²_post = 45/23 ≈ 1.957, and the posterior standard deviation is √(1.957) ≈ 1.399. The posterior mean is μ_post = σ²_post × (μ₀/τ² + n·x̄/σ²) = (45/23) × (50/9 + 10 × 52/25) = (45/23) × (5.556 + 20.8) = (45/23) × 26.356 ≈ 51.55. The posterior is N(51.55, 1.957). A 95% credible interval is 51.55 ± 1.96 × 1.399 ≈ (48.81, 54.29). Notice that the posterior mean has been pulled toward the data mean of 52 from the prior mean of 50, and the posterior variance (1.957) is smaller than both the prior variance (9) and the sampling variance of x̄ (25/10 = 2.5).

Question 4: APPLIED

A medical researcher is studying the efficacy of a new treatment. From pilot studies, she believes the response rate θ is approximately 0.5 and encodes this as a Beta(10, 10) prior. In a clinical trial with n = 100 patients, s = 65 respond. (a) Find the posterior distribution. (b) Compute a 95% credible interval. (c) Compute the posterior probability that θ > 0.5. (d) Compare the posterior mean to the MLE and discuss the prior's influence.

Show answer

(a) Posterior: Beta(10 + 65, 10 + 100 − 65) = Beta(75, 45). (b) The posterior mean is 75/120 = 0.625 and the posterior standard deviation is approximately √(75 × 45 / (120² × 121)) ≈ 0.044. A 95% credible interval is approximately (0.537, 0.710) (computed via Beta quantile functions). (c) P(θ > 0.5 | data) can be computed as 1 − F_Beta(0.5; 75, 45), which is approximately 0.998, providing very strong evidence that the response rate exceeds 50%. (d) The MLE is 65/100 = 0.65, and the posterior mean is 0.625, shifted slightly toward the prior mean of 0.50. The prior pseudo-count of 20 is modest relative to n = 100, so the data dominate. With a flat Beta(1,1) prior, the posterior mean would be 66/102 ≈ 0.647, illustrating the diminishing influence of the informative prior as sample size grows.

Question 5: CRITICAL THINKING

Consider two researchers analyzing the same data set of n = 15 observations from a Normal(μ, σ² = 1) model with x̄ = 3.0. Researcher A uses a N(0, 100) prior (vague) and Researcher B uses a N(0, 1) prior (informative, centered at zero). (a) Derive the posterior means and variances for both researchers. (b) Compute the 95% credible intervals for both. (c) Discuss whether the choice of prior is consequential here and under what conditions the two posteriors would converge. What does this tell you about the relationship between prior specification and sample size?

Show answer

For Researcher A: posterior precision = 1/100 + 15/1 = 15.01, posterior variance = 1/15.01 ≈ 0.0666, posterior mean = 0.0666 × (0/100 + 15 × 3.0) = 0.0666 × 45.0 ≈ 2.997. So A's posterior is approximately N(2.997, 0.0666), with 95% CI ≈ (2.49, 3.50). For Researcher B: posterior precision = 1/1 + 15/1 = 16, posterior variance = 1/16 = 0.0625, posterior mean = 0.0625 × (0/1 + 45) = 2.8125. So B's posterior is approximately N(2.8125, 0.0625), with 95% CI ≈ (2.32, 3.30). The informative prior centered at 0 pulls B's posterior mean away from x̄ = 3.0 noticeably (to 2.81), whereas A's vague prior barely affects the result. The difference demonstrates that with only n = 15 observations, the prior can still matter. As n → ∞, both posteriors converge to the same distribution centered on the true μ (Bernstein–von Mises theorem). The practical lesson is that for moderate sample sizes, prior sensitivity analysis is essential; for large n, even moderately informative priors are overwhelmed by the data.

SUMMARY

Summary

Bayesian inference provides a coherent, probabilistic framework for learning about unknown parameters from data. The process begins with a prior distribution π(θ) that encodes pre-data beliefs, which is combined with the likelihood function f(x | θ)—a measure of how well each parameter value explains the observed data—to produce the posterior distribution π(θ | x) via Bayes' rule: posterior ∝ likelihood × prior. The marginal likelihood m(x) serves as the normalizing constant and plays a pivotal role in model comparison through Bayes factors.

In conjugate families (e.g., Beta–Binomial, Normal–Normal), the posterior has a closed-form update rule, making computation straightforward. For more complex models, MCMC methods and variational inference approximate the posterior numerically. The choice of prior—informative, weakly informative, or non-informative—can materially affect conclusions in small samples but is washed out as data accumulate, a property guaranteed by the Bernstein–von Mises theorem. By producing full posterior distributions rather than single point estimates, Bayesian inference offers direct probabilistic answers to inferential questions—credible intervals, posterior probabilities of hypotheses, and posterior predictive distributions—that are often more intuitive and actionable than their frequentist counterparts.