Historical Context & Motivation
The idea that probability can express degrees of belief rather than merely long-run frequencies has deep roots in the work of Thomas Bayes and Pierre-Simon Laplace. Once a researcher computes a posterior distribution over an unknown parameter, the natural next question becomes: how do we summarize that distribution in a way that is both concise and faithful to its shape? This challenge motivated the development of posterior summaries—point estimates, interval estimates, and loss-function-based decisions—that have become central tools in modern Bayesian practice.
Frequentist statistics addressed the summarization problem through confidence intervals and maximum likelihood estimators, but these tools carry interpretive constraints: a 95% confidence interval, for instance, does not claim that the parameter lies within it with probability 0.95. Bayesians sought a framework in which intervals would carry exactly that direct probability interpretation. The resulting concept, the credible interval, assigns posterior probability directly to a set of parameter values, providing an intuitive and coherent summary of uncertainty.
The central question this lesson addresses is: given a full posterior distribution π(θ | y), what are the principled ways to collapse it into point summaries (posterior mean, median, mode) and interval summaries (equal-tailed intervals, highest posterior density regions), and how do different choices reflect different loss functions and inferential goals?
Core Principles & Definitions
After observing data y and specifying a prior π(θ), the posterior distribution π(θ | y) encodes all updated information about the parameter θ. Because this distribution is often a complex density over a continuous space, analysts require principled tools to extract actionable summaries. The following core ideas underpin the choice of point estimates and interval summaries in Bayesian inference.
Posterior Mean
Posterior Median
Posterior Mode (MAP)
Equal-Tailed Credible Interval
Highest Posterior Density (HPD) Region
Visual Explanation — Posterior Distribution & Summaries
The diagram below illustrates a mildly skewed posterior density for a parameter θ, along with the three common point estimates and two types of 95% credible intervals. Observe how the posterior mode sits at the peak, the posterior mean is pulled toward the heavier right tail, and the posterior median lies between them.
The key observation from this diagram is that the three point estimates differ whenever the posterior is asymmetric. For a perfectly symmetric posterior—such as a normal distribution—the mode, median, and mean coincide. As skewness increases the mean is pulled toward the heavier tail because it integrates over the entire density, whereas the mode remains anchored at the peak. Similarly, the equal-tailed interval places identical probability (α/2) in each tail, which can result in a wider interval compared to the HPD region that instead seeks the shortest set with the required coverage.
Mathematical Framework
Each posterior summary can be derived as the optimizer of a specific expected loss under the posterior distribution. Let π(θ | y) denote the posterior density and L(θ, δ) the loss incurred when the true parameter is θ and our summary is δ. The Bayes estimator is the δ that minimizes the posterior expected loss, also called the posterior risk.
Credible Intervals
A (1 − α) credible interval is any set C ⊆ Θ such that P(θ ∈ C | y) = 1 − α. Two canonical constructions dominate practice.
Detailed Comparison — Equal-Tailed vs. HPD Intervals
While both the equal-tailed and HPD intervals achieve the same nominal coverage probability, they can differ substantially in width and interpretation when the posterior is skewed or multimodal. The diagram below contrasts both interval types on a Gamma(3, 1) posterior, a distribution with notable right skewness. The HPD interval is shorter because it capitalizes on the asymmetry, capturing the tall left portion of the density rather than trimming equal probability from both tails.
| Feature | Equal-Tailed Interval | HPD Interval |
|---|---|---|
| Definition | Quantile-based: places α/2 probability in each tail. | Density-based: collects all θ with π(θ | y) ≥ k_α. |
| Width | May be wider for skewed posteriors. | Always the shortest interval (for unimodal posteriors). |
| Transformation invariance | Yes — quantiles are preserved under monotone transformations. | No — density values change under reparameterization. |
| Multimodal posteriors | Always a single contiguous interval. | May consist of disjoint intervals. |
| Ease of computation | Very easy from MCMC samples (sort and take percentiles). | Requires density estimation or specialized algorithms. |
Worked Example — Beta-Binomial Model
Suppose we observe y = 7 successes in n = 20 Bernoulli trials and adopt a Beta(2, 2) prior on the success probability θ. Because the Beta prior is conjugate to the Binomial likelihood, the posterior is available in closed form: θ | y ∼ Beta(α + y, β + n − y) = Beta(9, 15). We wish to compute the posterior mean, median, mode, and both the 95% equal-tailed and 95% HPD credible intervals.
Credible Intervals vs. Confidence Intervals
One of the most persistent sources of confusion in applied statistics is the distinction between a Bayesian credible interval and a frequentist confidence interval. Although both produce numeric intervals and can even coincide numerically in certain conjugate settings, they answer fundamentally different questions and carry distinct philosophical interpretations. Understanding this difference is essential for reporting results accurately and for choosing the right framework for a given inferential goal.
| Aspect | Bayesian Credible Interval | Frequentist Confidence Interval |
|---|---|---|
| Probability statement | P(θ ∈ C | y) = 1 − α. The parameter has a (1 − α) probability of lying in C given the observed data. | Under repeated sampling, (1 − α)×100% of constructed intervals contain the true θ. No probability statement about a specific interval. |
| θ treated as | Random (has a distribution reflecting uncertainty). | Fixed but unknown (randomness comes from the data). |
| Depends on prior? | Yes — the prior influences the posterior and hence the interval. | No — the interval is determined solely by the sampling distribution of the estimator. |
| Interpretation | Direct: 'There is a 95% probability θ is in this interval.' | Indirect: 'The procedure covers θ 95% of the time.' |
| Numerical agreement | Coincides with confidence interval under flat priors and regular likelihoods (Bernstein–von Mises). | Can differ from credible intervals, especially in small samples or with informative priors. |
Connections to Advanced Bayesian Theory
Posterior summaries and credible intervals serve as the foundation for more advanced Bayesian methods. As models grow in complexity—hierarchical structures, high-dimensional parameter spaces, non-parametric priors—the challenges of summarizing posteriors intensify. This section surveys the connections between the core tools introduced above and their extensions in modern Bayesian practice.
| Core Concept | Advanced Extension | Key Idea |
|---|---|---|
| Posterior mean | Rao-Blackwellized estimates | Conditioning MCMC estimates on sufficient statistics reduces variance, yielding sharper posterior mean estimates. |
| HPD region | Loss-function-based decision regions | Generalized to arbitrary loss functions and multivariate parameter spaces; HPD becomes a highest-density credible set in ℝᵈ. |
| Equal-tailed interval | Quantile regression & posterior predictive intervals | The quantile machinery extends from parameter intervals to prediction intervals over future observations. |
| MAP estimate | Variational inference (VI) | VI approximates the posterior with a tractable family; the mode of the variational distribution is a MAP-like summary of the approximate posterior. |
| Credible interval calibration | Posterior predictive checks & calibration diagnostics | Simulation-based calibration (SBC) verifies that nominal credible interval coverage matches empirical coverage across simulated datasets. |
In high-dimensional settings the concept of a credible interval extends to credible regions and marginal credible intervals. A joint 95% HPD region in ℝd is the smallest-volume set with 95% posterior probability—analogous to the one-dimensional HPD interval but often computationally intractable for large d. Practitioners frequently report marginal intervals (the credible interval for each component θj obtained from its marginal posterior), acknowledging that these do not capture dependence between parameters. The Bernstein–von Mises theorem provides an asymptotic bridge: under regularity conditions, the posterior converges to a normal distribution centered at the MLE, so credible and confidence intervals become numerically equivalent as n → ∞.
Practice Problems
Summary
The posterior distribution π(θ | y) encapsulates all updated knowledge about a parameter θ after observing data y, but communicating its full shape is often impractical. Point summaries reduce the posterior to a single number: the posterior mean (optimal under squared-error loss), the posterior median (optimal under absolute-error loss), and the posterior mode (MAP) (optimal under 0-1 loss). These estimators coincide for symmetric unimodal posteriors but diverge under skewness or multimodality.
Credible intervals provide interval summaries with a direct probability interpretation: a 95% credible interval contains the parameter with posterior probability 0.95. The equal-tailed interval places equal probability in each tail and is invariant to monotone reparameterization, while the highest posterior density (HPD) region is the shortest credible set and can reveal multimodality by splitting into disjoint pieces. Choosing among these tools depends on the loss function relevant to the problem, the posterior's shape, and the communication goals of the analysis. Together, posterior summaries and credible intervals form the essential toolkit for translating Bayesian inference into actionable conclusions.