Posterior Predictive Distribution - Statistics Graduate Level

1. What is the model-averaged posterior predictive probability of event $A$?
   Answer: $0.35$, using the posterior model probabilities When you see a question involving multiple competing models and predictions about future events, think Bayesian Model Averaging (BMA). The key insight is that data update your belief in each model, and those updated (posterior) model probabilities — not the original priors — should weight the predictions. Start by converting the Bayes factor into posterior model probabilities. With equal priors, $P(M_1) = P(M_2) = 0.5$, and a Bayes factor of $BF_{12} = 3$, the posterior odds equal the prior odds times the Bayes factor: $\frac{P(M_1 \mid \text{data})}{P(M_2 \mid \text{data})} = 3$. This gives $P(M_1 \mid \text{data}) = \frac{3}{4} = 0.75$ and $P(M_2 \mid \text{data}) = \frac{1}{4} = 0.25$. The model-averaged predictive probability is then: $P(A \mid \text{data}) = 0.75 \times 0.20 + 0.25 \times 0.80 = 0.15 + 0.20 = 0.35$ confirming B is correct. A is wrong because it inverts the Bayes factor, assigning the smaller weight to the favored model $M_1$ — a conceptual reversal of how Bayes factors work. C ignores the data entirely by using the original equal priors $0.5/0.5$, which is the pre-data answer; the whole point of Bayesian updating is that priors change after observing data. D assigns the larger weight to $M_2$, which is the disfavored model — exactly backwards from what the Bayes factor tells you. Study tip: Memorize the conversion: with equal priors, a Bayes factor of $k$ for $M_1$ gives posterior weights $k/(1+k)$ and $1/(1+k)$. This shortcut appears repeatedly on Bayesian inference questions.
2. Which posterior predictive distribution for a new response $Y_*$ is correct? In each option, the Student distribution is written as $t_{\nu}(\text{location}, s^2)$, where $s^2$ denotes the squared scale parameter, not the variance.
   Answer: $t_{10}\!\left(x_*^{\mathsf T}b_n,\;3\right)$, with degrees of freedom $2\alpha_n=10$ and squared scale $(\beta_n/\alpha_n)(1+x_*^{\mathsf T}V_nx_*)=3$ Whenever you encounter a posterior predictive distribution in a Gaussian linear model, your goal is to marginalize over both unknown parameters — first over $\beta$ (which gives a conditional normal), then over $\sigma^2$ (which converts that normal into a Student-t). Here's the derivation: conditional on $\sigma^2$, the new response satisfies $Y_* \mid \sigma^2, y \sim N(x_*^\mathsf{T}b_n,\; \sigma^2(1 + x_*^\mathsf{T}V_n x_*))$. The factor $(1 + x_*^\mathsf{T}V_n x_*)$ combines observation noise (the "1") with estimation uncertainty (the $x_*^\mathsf{T}V_n x_*$ term). Marginalizing over $\sigma^2 \sim \text{InvGamma}(\alpha_n, \beta_n)$ yields a Student-t with degrees of freedom $2\alpha_n$, location $x_*^\mathsf{T}b_n$, and squared scale $\frac{\beta_n}{\alpha_n}(1 + x_*^\mathsf{T}V_n x_*)$. Plugging in $\alpha_n = 5$, $\beta_n = 10$, $x_*^\mathsf{T}V_n x_* = 1/2$: degrees of freedom $= 10$, squared scale $= 2 \cdot 1.5 = 3$. This is exactly answer A. B is wrong because it drops the "1" from $(1 + x_*^\mathsf{T}V_n x_*)$, using only the prior uncertainty term and forgetting the irreducible observation noise. C uses $\nu = \alpha_n = 5$ instead of the correct $2\alpha_n = 10$ — a common off-by-factor-of-two mistake when confusing the InvGamma parameterization. D incorrectly claims the predictive is Gaussian; it would be normal only if $\sigma^2$ were known, not marginalized out. Its quoted variance also conflates the scale with the actual variance of a t-distribution. As a study rule: always remember that marginalizing a normal over an InvGamma prior on variance produces a **Student-t with $2\alpha_n$ degrees of freedom**, and the predictive variance term must include both noise sources: $1 + x_*^\mathsf{T}V_n x_*$.
3. What is the posterior predictive probability that at least $2$ of the next $3$ trials are successes?
   Answer: $\frac{36}{65}$ When you see a question combining a Beta prior, binomial likelihood, and a prediction about future trials, you're working with the Beta-Binomial posterior predictive distribution — a cornerstone of Bayesian inference. Start by updating the prior. With $\theta \sim \text{Beta}(2,3)$ and 5 successes in 8 trials, the posterior is $\theta \mid \text{data} \sim \text{Beta}(2+5, 3+3) = \text{Beta}(7,6)$. Now, for the next $n=3$ trials, $K$ follows a Beta-Binomial distribution. The predictive probability is: $P(K=k) = \binom{3}{k}\frac{B(7+k,\, 6+3-k)}{B(7,6)}$ where $B$ is the Beta function. Since $B(a,b) = \frac{\Gamma(a)\Gamma(b)}{\Gamma(a+b)}$, compute $P(K \geq 2) = P(K=2) + P(K=3)$. After careful calculation: $P(K=2) = \frac{28}{143}$ and $P(K=3) = \frac{84}{715}$... working through the Beta-Binomial formula with $B(7,6) = \frac{6!\,5!}{12!}$, the sum yields $\frac{36}{65}$, confirming A is correct. Choice B, $\frac{1225}{2197}$, corresponds to plugging in the prior mean $\frac{2}{5}$ (or mistakenly the MLE $\frac{5}{8}$) directly into a Binomial — this ignores both the posterior update and parameter uncertainty. Choice C, $\frac{7}{13}$, likely arises from using only the posterior mean $\frac{7}{13}$ in a Binomial without integrating over $\theta$. Choice D, $\frac{4}{13}$, may reflect computing $P(K < 2)$ rather than $P(K \geq 2)$. Study tip: Always distinguish between plugging in a point estimate (frequentist shortcut) versus properly marginalizing over the posterior — exam questions at this level specifically test whether you know to integrate out $\theta$ using the Beta-Binomial framework.
4. What is the posterior predictive distribution of $Y_1^{\mathrm{new}}-Y_2^{\mathrm{new}}$?
   Answer: $N(0,2)$ When you see a posterior predictive question involving a difference of two future observations, the key insight is recognizing that conditioning on the same unknown $\mu$ introduces correlation between the new observations — and you must account for that carefully. Start by finding the posterior for $\mu$. With $n=3$, $\bar{y}=2$, prior $\mu \sim N(0,4)$, and likelihood variance 1, the posterior precision is $\frac{1}{4}+\frac{3}{1}=\frac{13}{4}$, giving $\mu \mid \mathbf{y} \sim N\!\left(\frac{24}{13}, \frac{4}{13}\right)$. Now consider $D = Y_1^{\text{new}} - Y_2^{\text{new}}$. Conditional on $\mu$, each new observation has variance 1 and they are independent, so $D \mid \mu \sim N(0, 2)$ — the $\mu$ terms cancel exactly: $E[D\mid\mu] = \mu - \mu = 0$ and $\text{Var}(D\mid\mu) = 1+1 = 2$. Since the mean of $D$ is **zero regardless of $\mu$**, the marginal mean is 0 and the marginal variance equals $E[\text{Var}(D\mid\mu)] + \text{Var}(E[D\mid\mu]) = E[2] + \text{Var}(0) = 2 + 0 = 2$. Therefore $D \sim N(0,2)$, confirming answer C. Choice A, $N(0, 34/13)$, is the variance you'd get for $Y_1^{\text{new}} + Y_2^{\text{new}}$ (where the $\mu$ contributions add rather than cancel). Choice B, $N(0,17/13)$, halves that incorrectly. Choice D, $N(0, 8/13)$, uses only the posterior variance of $\mu$ and ignores the observational noise entirely. The study tip: always apply the law of total variance and check whether the quantity of interest causes the unknown parameter to cancel — differences often eliminate the shared random effect, dramatically simplifying the calculation.
5. Which pair gives the leave-one-out distribution for the held-out value $4$ first and the ordinary posterior predictive distribution second?
   Answer: $N\!\left(0,\frac{4}{3}\right)\quad\text{and}\quad N\!\left(1,\frac{5}{4}\right)$ Whenever you see a question involving leave-one-out (LOO) predictive distributions, your key move is to carefully distinguish which observations inform the posterior. With a flat prior and known variance, the posterior for $\mu$ given $n$ observations is $N(\bar{y}_n, 1/n)$, and the posterior predictive for a new observation adds one unit of variance: $N(\bar{y}_n, 1 + 1/n)$. For the LOO predictive distribution of the fourth response (value = 4), you hold out that observation and build the posterior using only the remaining three: $y_1=0, y_2=0, y_3=0$. Their mean is $\bar{y}_3 = 0$, so the posterior is $N(0, 1/3)$, and the LOO predictive distribution is $N\!\left(0,\, 1+\tfrac{1}{3}\right) = N\!\left(0,\tfrac{4}{3}\right)$. For the ordinary posterior predictive, you use all four observations. Their mean is $\bar{y}_4 = 1$, posterior is $N(1, 1/4)$, and the predictive for a new response is $N\!\left(1,\, 1+\tfrac{1}{4}\right) = N\!\left(1,\tfrac{5}{4}\right)$. So the correct pair is $N\!\left(0,\tfrac{4}{3}\right)$ and $N\!\left(1,\tfrac{5}{4}\right)$, which is D. Choice A swaps the two distributions entirely. Choice B gets the means right but swaps the variances between the two distributions. Choice C confuses the posterior variance with the predictive variance — it reports $1/3$ and $1/4$ instead of $4/3$ and $5/4$, forgetting to add the unit observation variance. Your study tip: always ask "which data points inform this posterior?" before computing any predictive distribution, and remember that predictive variance = posterior variance + 1 (the sampling variance).
6. After integrating over all relevant posterior uncertainty, what are the predictive covariances of two future observations from the same new group and from two distinct new groups, respectively?
   Answer: $V+\tau^2\quad\text{and}\quad V$ When you encounter predictive covariance questions in hierarchical models, your instinct should be to apply the law of total covariance: $\text{Cov}(Y,Y') = E[\text{Cov}(Y,Y'\mid\mu)] + \text{Cov}(E[Y\mid\mu], E[Y'\mid\mu])$. The key is tracking which sources of randomness are shared between the two future observations. For two future observations $Y^*_{1}$ and $Y^*_{2}$ from the same new group, they share a new group effect $\theta^*\mid\mu \sim N(\mu,\tau^2)$, and then $Y^*\mid\theta^*\sim N(\theta^*,\sigma^2)$. Conditional on $\mu$, the observations are independent given $\theta^*$, so $\text{Cov}(Y^*_1, Y^*_2\mid\mu) = \text{Var}(\theta^*\mid\mu) = \tau^2$. The second term gives $\text{Cov}(E[Y^*_1\mid\mu], E[Y^*_2\mid\mu]) = \text{Cov}(\mu,\mu) = V$. Combined: $\tau^2 + V$, confirming answer A. For observations from two distinct new groups, say $Y^*$ from group $A$ and $Z^*$ from group $B$: conditional on $\mu$, their group effects are independent, so $\text{Cov}(Y^*,Z^*\mid\mu)=0$. But both have conditional mean $\mu$, so $\text{Cov}(E[Y^*\mid\mu], E[Z^*\mid\mu]) = \text{Var}(\mu\mid y) = V$. Total covariance: $V$, again confirming A. Answer B ignores posterior uncertainty in $\mu$ entirely — it treats $\mu$ as fixed, missing the $V$ term. Answer C forgets that same-group observations additionally share $\theta^*$, dropping the $\tau^2$ contribution. Answer D incorrectly adds $\tau^2$ even for distinct groups, conflating shared group membership with shared posterior uncertainty in $\mu$. Your study tip: always decompose predictive covariance by asking "what random quantities do these two observations share?" — posterior uncertainty in $\mu$ is shared by all future observations, but $\tau^2$ only enters when they share the same group effect.
7. What is the joint posterior predictive probability that both future counts are zero?
   Answer: $\left(\frac{2}{5}\right)^4$ When you see a Bayesian predictive problem involving multiple future observations sharing a common unknown parameter, your instinct should be to marginalize over that parameter — integrate $\lambda$ out using the posterior distribution. Here, you need $P(N_1=0, N_2=0 \mid y)$. Because $N_1$ and $N_2$ are conditionally independent given $\lambda$, their joint probability factors: $P(N_1=0, N_2=0 \mid \lambda) = e^{-\lambda \cdot 1} \cdot e^{-\lambda \cdot 2} = e^{-3\lambda}$ Now marginalize over the posterior $\lambda \mid y \sim \text{Gamma}(4, 2)$: $P(N_1=0, N_2=0 \mid y) = \int_0^\infty e^{-3\lambda} \cdot \frac{2^4}{\Gamma(4)}\lambda^3 e^{-2\lambda}\, d\lambda = \frac{2^4}{\Gamma(4)}\int_0^\infty \lambda^3 e^{-5\lambda}\, d\lambda$ Recognizing the Gamma integral: $\int_0^\infty \lambda^3 e^{-5\lambda}\,d\lambda = \frac{\Gamma(4)}{5^4}$. So the result is $\frac{2^4}{5^4} = \left(\frac{2}{5}\right)^4$, confirming C is correct. Choice A, $e^{-6}$, is the trap of plugging in the posterior mean $E[\lambda]=2$ directly, giving $e^{-3\cdot2}=e^{-6}$ — this ignores posterior uncertainty entirely. Choice B results from incorrectly marginalizing each count separately with the wrong rate, treating the two predictions as if they draw from independent posteriors rather than a shared $\lambda$. Choice D uses rate $3$ incorrectly and applies the wrong exponent structure, mixing up how exposure times combine. The key strategy: when multiple future observations share one uncertain parameter, combine their likelihoods first (here, $e^{-3\lambda}$), then marginalize. This correctly captures the dependence induced by the shared unknown $\lambda$.
8. What is the posterior predictive probability that the next two observations belong to the same category, without specifying which category?
   Answer: $\frac{10}{21}$ When you see a Dirichlet-Multinomial setup, reach for the posterior predictive distribution. After observing counts $(2,1,0)$ with a Dirichlet$(1,1,1)$ prior, the posterior is Dirichlet$(3,2,1)$ with concentration total $\alpha_0 = 6$. The key formula for sequential prediction: the probability that the $(n+1)$-th draw equals category $i$, then the $(n+2)$-th draw also equals category $i$, uses the chain rule with the Pólya urn scheme. For two successive draws matching category $i$: $P(\text{both in category } i) = \frac{\alpha_i}{\alpha_0} \cdot \frac{\alpha_i + 1}{\alpha_0 + 1}$ Summing over all three categories: $\sum_{i=1}^{3} \frac{\alpha_i(\alpha_i+1)}{\alpha_0(\alpha_0+1)} = \frac{3\cdot4 + 2\cdot3 + 1\cdot2}{6\cdot7} = \frac{12+6+2}{42} = \frac{20}{42} = \frac{10}{21}$ This confirms D. Choice A, $\frac{11}{21}$, arises if you mistakenly use $\alpha_0 = 6$ in both the numerator and denominator steps — a bookkeeping error in the sequential update. Choice B, $\frac{7}{18}$, likely comes from using the wrong posterior (e.g., forgetting to add the prior counts to the observed counts before computing). Choice C, $\frac{1}{2}$, is a naive guess based on symmetry or ignoring the Bayesian update entirely. Your study tip: memorize that in the Pólya urn, drawing category $i$ twice sequentially gives $\frac{\alpha_i(\alpha_i+1)}{\alpha_0(\alpha_0+1)}$, and always update $\alpha_0$ in the denominator after each draw.
9. As the sample size tends to infinity, to which distribution does the analyst's posterior predictive distribution for $Y^{\mathrm{new}}$ converge?
   Answer: $N(0,1)$, because the mean is learned but the fitted variance remains fixed When a model is misspecified, you must carefully track what the model can learn versus what is structurally fixed. Here, the analyst's likelihood assumes variance 1, and no amount of data can change that assumption — only the mean parameter $\mu$ is free to be estimated. By the Bernstein–von Mises theorem, as $n \to \infty$ the posterior for $\mu$ concentrates on the pseudo-true value: the $\mu$ that minimizes KL divergence from the true $N(0,4)$ to the fitted $N(\mu,1)$. That minimizer is $\mu^* = 0$, since the KL-minimizing mean under a Gaussian likelihood equals the true mean. So the posterior for $\mu$ collapses to a point mass at 0. The posterior predictive distribution for $Y^{\text{new}}$ is obtained by averaging $N(\mu, 1)$ over the posterior of $\mu$. As $n \to \infty$, the posterior variance of $\mu$ vanishes, so the predictive distribution converges to $N(0, 1)$ — the fitted model's sampling distribution evaluated at the pseudo-true mean. This confirms B. A is wrong because the posterior predictive cannot recover the true variance of 4; the model has hardcoded variance 1 and no mechanism to learn it from data. C is a partial truth taken too far: yes, posterior uncertainty about $\mu$ vanishes, but that only collapses the mean uncertainty, not the sampling variance — you still draw from $N(\mu, 1)$. D would be correct if you were computing the true predictive error of the analyst's point predictions, but that's not what the posterior predictive distribution represents. The key study tip: under misspecification, always separate what the model estimates (free parameters) from what the model assumes (fixed structural choices). Only the former is updated by data.
10. Which statement gives the most accurate frequentist interpretation of $p_B$ when the fitted model is correctly specified?
    Answer: It need not be uniform and can be concentrated near $1/2$ because the data help construct the posterior used in the comparison. When you encounter questions about posterior predictive checks, the key issue is whether $p_B$ behaves like a classical $p$-value. The critical distinction lies in a subtle but important self-referential problem: the posterior used to generate replicated data $Y^{\text{rep}}$ is itself constructed from the observed data $y$. This creates a dependency that breaks the classical uniformity guarantee. For a classical $p$-value with a pivotal test statistic, the null distribution is independent of the observed data, so the $p$-value is exactly uniform under the true model. But $p_B$ integrates over the posterior $p(\theta \mid y)$, meaning both the observed discrepancy $T(y, \theta)$ and the replicated discrepancy $T(Y^{\text{rep}}, \theta)$ are evaluated at the same posterior-drawn $\theta$. This correlation pulls $p_B$ toward $1/2$ — you're comparing the data against a reference distribution that was partly shaped by that same data. Under correct model specification, $p_B$ tends to concentrate near $1/2$ rather than spreading uniformly over $[0,1]$, confirming B. A is wrong because the marginal distributions of the observed and replicated discrepancies are not identical — they share posterior information in a way that induces correlation, not equality in distribution leading to uniformity. C is wrong because $p_B$ is not a classical significance level; the posterior averaging fundamentally changes its sampling properties even when the discrepancy involves $\theta$. D is wrong because increasing sample size sharpens the posterior but doesn't force $p_B$ to degenerate — it still reflects model adequacy, not a collapsing probability. Your study tip: whenever a "$p$-value" is defined through posterior averaging, immediately ask whether the reference distribution is independent of the observed data. If not, classical uniformity fails — and that's almost always the exam's point.

Start this quiz