Bias, Variance & MSE - Statistics Graduate Level

1. Two estimators of $\theta$ satisfy $E(T_1)-\theta=1$, $E(T_2)-\theta=-1$, $\operatorname{Var}(T_1)=4$, $\operatorname{Var}(T_2)=1$, and $\operatorname{Cov}(T_1,T_2)=1$. For $T_a=aT_1+(1-a)T_2$, where $a$ may be any real number, which value of $a$ minimizes the mean squared error of $T_a$?
   Answer: $a=2/7$, after accounting for both covariance and squared bias When minimizing mean squared error (MSE) for a linear combination of biased estimators, you must account for all three components: variance of each estimator, their covariance, and the squared bias of the combined estimator. Start by computing the bias of $T_a$. Since $E(T_1) = \theta + 1$ and $E(T_2) = \theta - 1$, the bias of $T_a = aT_1 + (1-a)T_2$ is $a(1) + (1-a)(-1) = 2a - 1$. Next, the variance of $T_a$ is $a^2\operatorname{Var}(T_1) + (1-a)^2\operatorname{Var}(T_2) + 2a(1-a)\operatorname{Cov}(T_1,T_2) = 4a^2 + (1-a)^2 + 2a(1-a)$, which simplifies to $3a^2 - 2a + 1 + 2a(1-a) = 3a^2 + 1 - 2a + 2a - 2a^2 = a^2 - 2a + 1 + 2a + 2 - 2a = ...$ Carefully expanding: $4a^2 + 1 - 2a + a^2 + 2a - 2a^2 = 3a^2 + 1$. So $\operatorname{MSE}(T_a) = 3a^2 + 1 + (2a-1)^2 = 3a^2 + 1 + 4a^2 - 4a + 1 = 7a^2 - 4a + 2$. Differentiating and setting equal to zero: $14a - 4 = 0 \Rightarrow a = 2/7$. This confirms answer B is correct. Choice A ignores that $T_2$'s smaller variance doesn't automatically make $a=0$ optimal once biases differ. Choice C ($a = 1/3$) drops the covariance term, treating the estimators as independent and arriving at the wrong minimum. Choice D ($a = 1/2$) achieves unbiasedness but ignores that MSE includes variance — unbiasedness alone is not optimality. Study tip: Always write out MSE = Variance + Bias² fully before differentiating. Forgetting the covariance term is the most common error on linear-combination optimization problems.
2. Which value of $c$ minimizes $MSE(T_c)$ as an estimator of $\sigma^2$?
   Answer: $c=1/(n+1)$, which accepts bias to reduce total MSE When estimating a parameter, minimizing MSE requires balancing bias and variance — and sometimes accepting bias actually pays off. Here, $T_c = c\sum(X_i - \bar{X})^2$. Since $\sum(X_i - \bar{X})^2/\sigma^2 \sim \chi^2_{n-1}$, we know $E\left[\sum(X_i-\bar{X})^2\right] = (n-1)\sigma^2$, so $E[T_c] = c(n-1)\sigma^2$. The bias is $[c(n-1)-1]\sigma^2$, and it can be shown that $\text{Var}(T_c) = 2c^2(n-1)\sigma^4$. Therefore: $\text{MSE}(T_c) = \text{Bias}^2 + \text{Var} = [c(n-1)-1]^2\sigma^4 + 2c^2(n-1)\sigma^4$ Taking $d(\text{MSE})/dc = 0$ and solving yields $c = \frac{1}{n+1}$, confirming C is correct. This value deliberately introduces a small downward bias, but the reduction in variance more than compensates, lowering total MSE. Choice A ($c = 1/(n-1)$) gives the unbiased estimator $S^2$, but zero bias doesn't mean minimum MSE — variance still inflates the MSE unnecessarily. Choice B ($c = 1/n$) is the MLE, which also fails to minimize MSE; the MLE optimizes likelihood, not squared-error loss. Choice D ($c = 1/(n+2)$) goes too far — overshooting the optimal bias-variance tradeoff actually increases MSE relative to $1/(n+1)$. Study tip: On MSE-minimization problems, always write out $\text{MSE} = \text{Bias}^2 + \text{Variance}$ explicitly, differentiate with respect to your free parameter, and solve. Never assume the unbiased or MLE estimator automatically minimizes MSE — that's the most common trap in this topic area.
3. For which condition does $\widetilde\beta_k$ have strictly smaller mean squared error than $\widehat\beta$ at the specified value of $\beta$?
   Answer: $\beta^2<[(1+k)/(1-k)]v$, because the variance reduction exceeds squared bias When comparing two estimators, the key tool is the bias-variance decomposition of MSE: $\text{MSE}(\tilde\beta_k) = \text{Bias}^2 + \text{Var}$. For $\widetilde\beta_k = k\widehat\beta$, since $E(\widehat\beta) = \beta$, the bias is $E(k\widehat\beta) - \beta = (k-1)\beta$, so $\text{Bias}^2 = (1-k)^2\beta^2$. The variance is $\text{Var}(k\widehat\beta) = k^2 v$. Thus $\text{MSE}(\widetilde\beta_k) = (1-k)^2\beta^2 + k^2 v$, compared to $\text{MSE}(\widehat\beta) = v$ (since OLS is unbiased). You want $\text{MSE}(\widetilde\beta_k) < v$, which means $(1-k)^2\beta^2 + k^2 v < v$. Rearranging: $(1-k)^2\beta^2 < v(1-k^2) = v(1-k)(1+k)$. Dividing both sides by $(1-k) > 0$ gives $(1-k)\beta^2 < v(1+k)$, and finally $\beta^2 < \frac{(1+k)}{(1-k)}v$. This confirms answer A is correct — shrinkage wins when the true coefficient is small relative to the variance, because the variance reduction outweighs the introduced bias. Answer B flips the ratio to $(1-k)/(1+k)$, which is strictly less than 1 for $0 < k < 1$, making the condition far too restrictive — this comes from an algebra error when dividing by $(1+k)$ instead of $(1-k)$. Answers C and D reverse the inequality direction entirely, claiming ridge improves for large $\beta$, which contradicts the intuition: shrinkage hurts most when the true signal is large, because bias grows while variance savings remain bounded. A useful pattern: whenever you see a shrinkage estimator, immediately write out $\text{MSE} = \text{Bias}^2 + \text{Var}$ and solve the inequality algebraically — don't rely on intuition alone to get the exact threshold ratio.
4. Ignoring integer restrictions initially, which allocation minimizes the mean squared error of the estimated treatment effect?
   Answer: $n_T=30$ and $n_C=10$, because the fixed bias does not affect variance-optimal allocation When you see a question mixing bias and variance in an estimation problem, your instinct should be to write out the mean squared error (MSE) explicitly: $\text{MSE} = \text{Variance} + \text{Bias}^2$. Here, the estimator is $\hat{\Delta} = \bar{Y}_T - \bar{Y}_C$. The bias equals $d$ (fixed, regardless of allocation), so $\text{Bias}^2 = d^2$ is a constant that plays no role in optimization. The variance is: $\text{Var}(\hat{\Delta}) = \frac{\sigma_T^2}{n_T} + \frac{\sigma_C^2}{n_C} = \frac{9}{n_T} + \frac{1}{n_C}$ To minimize this subject to $n_T + n_C = 40$, apply the Neyman allocation formula: $n_T \propto \sigma_T$ and $n_C \propto \sigma_C$. Since $\sigma_T = 3$ and $\sigma_C = 1$, the optimal split is $n_T : n_C = 3 : 1$, giving $n_T = 30$ and $n_C = 10$ — confirming D. A is wrong because equal allocation (20/20) ignores the unequal variances. Equal randomization controls confounding, but it does not minimize MSE when variances differ. B confuses the allocation rule — Neyman allocation is proportional to the standard deviation, not the variance. Proportional to variances would give 36/4, which is too extreme. C inverts the logic entirely; you should oversample the noisier arm (treatment, $\sigma=3$), not the quieter one. The key study tip: bias terms that are constant with respect to the design parameter vanish during optimization. Only the variance component drives the optimal allocation. Don't let the word "bias" in the problem distract you from a clean variance-minimization exercise.
5. Which statement correctly compares the mean squared errors of these estimators?
   Answer: $MSE(T_1)-MSE(T_2)=\sigma^4/n^2$, so the unbiased correction lowers MSE When comparing estimators, remember that $MSE = \text{Bias}^2 + \text{Variance}$. The key here is computing bias and variance for each estimator separately, then comparing. Since $\bar{X} \sim N(\mu, \sigma^2/n)$, we have $E[\bar{X}^2] = \text{Var}(\bar{X}) + (E[\bar{X}])^2 = \sigma^2/n + \mu^2$. So $T_1 = \bar{X}^2$ has bias $\sigma^2/n$, giving $\text{Bias}^2(T_1) = \sigma^4/n^2$. Meanwhile $T_2 = \bar{X}^2 - \sigma^2/n$ is unbiased for $\mu^2$, so $\text{Bias}^2(T_2) = 0$. Both estimators share the same variance term $\text{Var}(\bar{X}^2)$, since subtracting the constant $\sigma^2/n$ doesn't affect variance. Therefore $MSE(T_1) - MSE(T_2) = \sigma^4/n^2$, confirming A: the bias correction strictly lowers MSE. Choice B reverses the subtraction, claiming the correction raises MSE — but since $T_2$ eliminates the positive bias-squared term without touching variance, it must have lower MSE, not higher. Choice C is tempting because subtracting a constant truly does leave variance unchanged — but MSE includes bias, not just variance, and that's exactly where the difference lives. C confuses $\text{Var}(T)$ with $MSE(T)$. Choice D invokes a spurious "doubling" from normal fourth moments; while higher moments of the normal distribution do appear in $\text{Var}(\bar{X}^2)$, they cancel identically between $T_1$ and $T_2$, contributing nothing to their difference. The study tip: whenever you see an estimator formed by adding or subtracting a constant, remember that only the bias changes — variance is unaffected. Always decompose MSE into its two components before drawing conclusions.
6. Which comparison is correct to order $n^{-2}$?
   Answer: Bias correction raises MSE by $2/n^2$ because variance inflation exceeds the saved squared bias When comparing estimators via MSE, remember that $\text{MSE} = \text{Bias}^2 + \text{Variance}$. Bootstrap bias correction reduces squared bias but inflates variance — and you must track both effects carefully to the same order in $n$. For the original estimator $T_n$: $\text{Bias}^2 = (2/n)^2 = 4/n^2$ and $\text{Var} = 1/n$, so $\text{MSE}(T_n) = 1/n + 4/n^2 + o(n^{-2})$. For the bias-corrected estimator $T_n^{BC}$: $\text{Bias}^2 = o(n^{-2})$ (negligible) and $\text{Var} = 1/n + 6/n^2$, so $\text{MSE}(T_n^{BC}) = 1/n + 6/n^2 + o(n^{-2})$. The difference is $\text{MSE}(T_n^{BC}) - \text{MSE}(T_n) = 6/n^2 - 4/n^2 = +2/n^2$. Bias correction raises MSE by $2/n^2$ because the variance inflation of $6/n^2$ exceeds the recovered squared bias of $4/n^2$. That confirms B. Choice A gets the direction wrong — it claims MSE decreases, but the variance penalty outweighs the bias savings. Choice C is tempting because both estimators share the $1/n$ variance leading term, but the $n^{-2}$ terms differ critically — stopping at the leading variance term misses the entire comparison. Choice D correctly identifies that variance inflation drives the MSE increase, but assigns the wrong magnitude ($6/n^2$ instead of the net $2/n^2$), confusing gross variance increase with net MSE change. The strategy: always compute $\text{MSE} = \text{Bias}^2 + \text{Variance}$ term by term to the required order before drawing conclusions — never evaluate bias and variance in isolation.
7. For an estimator sequence $T_n$ of $\theta$, suppose $\operatorname{Bias}(T_n)=n^{-1/4}+o(n^{-1/4})$ and $\operatorname{Var}(T_n)=2n^{-1/2}+o(n^{-1/2})$. Which conclusion follows?
   Answer: $T_n$ is consistent, and $n^{1/2}MSE(T_n)\to 3$ When analyzing MSE asymptotics, your first instinct should be to decompose: $\text{MSE}(T_n) = \text{Bias}^2(T_n) + \text{Var}(T_n)$, then identify which term dominates. Here, $\text{Bias}^2(T_n) = (n^{-1/4} + o(n^{-1/4}))^2 = n^{-1/2} + o(n^{-1/2})$, and $\text{Var}(T_n) = 2n^{-1/2} + o(n^{-1/2})$. Adding them gives $\text{MSE}(T_n) = 3n^{-1/2} + o(n^{-1/2})$. Multiplying by $n^{1/2}$: $n^{1/2}\text{MSE}(T_n) \to 3$. Since $\text{MSE}(T_n) \to 0$, the estimator is consistent. This confirms B. A is the most tempting trap. It's true that bias shrinks at rate $n^{-1/4}$, slower than the familiar $n^{-1/2}$. But consistency only requires $\text{MSE}(T_n) \to 0$, which holds here since both the squared bias and variance vanish. Slow convergence ≠ inconsistency. C is a calculation error — students who forget to square the bias before adding it to the variance would see only the variance term contributing, getting a limit of 2 instead of 3. Always square the bias first. D would require $n \cdot \text{MSE}(T_n) \to L < \infty$, i.e., $\text{MSE} = O(n^{-1})$. But here $\text{MSE} \sim 3n^{-1/2}$, so $n \cdot \text{MSE} \sim 3n^{1/2} \to \infty$. Root-$n$ consistency fails. Study tip: Always square the bias before computing MSE limits — it changes the convergence rate and can flip which term dominates. This is the single most common arithmetic error in asymptotic MSE problems.
8. An estimator $T$ of $g(\theta)$ has finite second moment, and $U$ is sufficient for $\theta$. Define the Rao–Blackwellized estimator $T^*=E(T\mid U)$. Which statement about bias, variance, and mean squared error is necessarily correct?
   Answer: $T^*$ has the same bias as $T$ and no larger MSE, with equality if and only if $T=T^*$ almost surely Whenever you encounter a Rao–Blackwell question, anchor your thinking to two foundational tools: the law of total expectation and the law of total variance. These two identities fully determine what conditioning on a sufficient statistic does to bias, variance, and MSE. The law of total expectation gives $E(T^*) = E[E(T \mid U)] = E(T)$, so $T^*$ and $T$ share exactly the same bias — whatever it is. The law of total variance gives $\text{Var}(T) = \text{Var}(T^*) + E[\text{Var}(T \mid U)]$, meaning $\text{Var}(T^*) = \text{Var}(T) - E[\text{Var}(T \mid U)] \leq \text{Var}(T)$. Since bias is preserved and variance can only decrease, MSE satisfies $\text{MSE}(T^*) \leq \text{MSE}(T)$. Equality holds precisely when $E[\text{Var}(T \mid U)] = 0$, which happens if and only if $T = T^*$ almost surely. This is exactly what C states, making it the correct answer. A is wrong because bias is preserved by conditioning — $T^*$ is unbiased only if $T$ already is. Claiming $T^*$ corrects bias is a common misconception. B is wrong because it says MSE is strictly smaller whenever $T$ is not a constant. In fact, strict improvement requires $T \neq T^*$ almost surely — a deterministic (constant) $T$ could still equal $T^*$. D is wrong on both counts: the variance of $T^*$ never exceeds that of $T$, and absolute bias is not necessarily reduced (it is simply preserved). Your study tip: memorize that Rao–Blackwell preserves bias and never increases variance, with strict variance reduction if and only if $T \neq T^*$ a.s. These two facts together answer almost every Rao–Blackwell multiple-choice question.
9. Which value of $c$ minimizes the integrated mean squared error $E[(\delta_c(X)-\theta)^2]$, where expectation is over both $\theta$ and $X$?
   Answer: $c=\tau^2/(\tau^2+\sigma^2)$, balancing prior variation against sampling noise When you see a question involving a prior distribution on a parameter and a conditional likelihood, you're in Bayesian estimation territory. The key move is to compute the integrated (joint) MSE by expanding it using the law of total expectation. Write the loss as $E[(\delta_c(X)-\theta)^2] = E[(cX - \theta)^2]$. Since $X = \theta + \epsilon$ where $\epsilon \sim N(0,\sigma^2)$ independently of $\theta$, substitute: $cX - \theta = (c-1)\theta + c\epsilon$. Because $\theta$ and $\epsilon$ are independent with means zero, the MSE becomes $(c-1)^2\tau^2 + c^2\sigma^2$. Minimize over $c$ by differentiating and setting equal to zero: $2(c-1)\tau^2 + 2c\sigma^2 = 0$, giving $c(\tau^2 + \sigma^2) = \tau^2$, so $c = \tau^2/(\tau^2+\sigma^2)$. This is answer D, and it's exactly the Bayes estimator under squared error loss — the posterior mean $E[\theta \mid X]$ is linear in $X$ with this shrinkage coefficient. Choice A sets $c=1$, which is the MLE $\delta(X)=X$. It's conditionally unbiased for $\theta$, but ignores the prior, so it over-fits to noisy observations and inflates integrated MSE. Choice B, $c = \tau/(\tau+\sigma)$, looks plausible but mixes standard deviations rather than variances — the correct weighting involves squared quantities. Choice C flips the numerator and denominator, weighting by noise variance $\sigma^2$ instead of prior variance $\tau^2$; this would shrink more when the signal is stronger, which is backwards. As a study tip: whenever you minimize an integrated risk over a linear estimator, expand the squared error algebraically using independence, then differentiate. The answer will always involve ratios of variances, not standard deviations.
10. At $\theta=2$, what lower bound on $MSE(T)$ follows from the biased Cramér–Rao inequality?
    Answer: $5/16$, obtained by combining the biased variance bound and squared bias Whenever you see a biased estimator paired with the Cramér–Rao inequality, remember that the bound applies to variance, not MSE directly — but MSE includes both variance and squared bias, so you must combine them carefully. The biased Cramér–Rao lower bound states that for an estimator with bias $b(\theta)$, its variance satisfies: $\text{Var}(T) \geq \frac{(1 + b'(\theta))^2}{nI(\theta)}$ Here, $b(\theta) = -\theta/4$, so $b'(\theta) = -1/4$. With $n = 9$ and $I(\theta) = 1$: $\text{Var}(T) \geq \frac{(1 - 1/4)^2}{9 \cdot 1} = \frac{(3/4)^2}{9} = \frac{9/16}{9} = \frac{1}{16}$ Now add the squared bias at $\theta = 2$: $b(2) = -2/4 = -1/2$, so $b(2)^2 = 1/4$. The full MSE lower bound is: $\text{MSE}(T) \geq \frac{1}{16} + \frac{1}{4} = \frac{1}{16} + \frac{4}{16} = \frac{5}{16}$ This confirms C is correct. Choice A stops at the variance bound $(1/16)$ and forgets to add the squared bias — a classic incomplete application of MSE decomposition. Choice B keeps the squared bias $(1/4)$ but discards the variance contribution entirely, which is equally incomplete. Choice D uses $nI(\theta) = 9$ in the denominator without the bias-adjustment factor $(1+b')^2$, as if the estimator were unbiased, then awkwardly adds bias — mixing two incompatible frameworks. Study tip: Always write $\text{MSE} = \text{Var} + \text{Bias}^2$ first, then bound each piece separately. The biased CRB only touches the variance term — never skip the squared bias when the question asks about MSE.

Start this quiz