Monte Carlo Estimation - Statistics Graduate Level

1. Approximately how many post-burn-in iterations are needed for the Monte Carlo standard error of the sample mean to be at most $0.02$?
   Answer: $30{,}000$ iterations, using the full integrated autocorrelation time When estimating $E_\pi[h(X)]$ with a correlated Markov chain, the variance of the sample mean is not simply $\text{Var}_\pi(h)/n$. Instead, you must account for autocorrelation through the integrated autocorrelation time $\tau$, giving: $\text{Var}(\bar{h}_n) \approx \frac{\text{Var}_\pi(h)}{n} \cdot \tau, \quad \text{where} \quad \tau = 1 + 2\sum_{k=1}^{\infty} \rho_k$ With $\rho_k = 0.5^k$, the one-sided sum is $\sum_{k=1}^{\infty} 0.5^k = 1$, so $\tau = 1 + 2(1) = 3$. Setting the Monte Carlo standard error (MCSE) $\leq 0.02$: $\frac{\tau \cdot \text{Var}_\pi(h)}{n} \leq (0.02)^2 \implies n \geq \frac{3 \times 4}{0.0004} = 30{,}000$ This confirms D is correct. A is wrong because it ignores autocorrelation entirely, using $n \geq 4/0.0004 = 10{,}000$. This severely underestimates the required sample size — a classic trap when students forget that correlated draws carry less information than independent ones. B uses only the one-sided sum ($\tau = 1 + 1 = 2$), yielding $n = 20{,}000$. This forgets that the spectral variance formula requires doubling the lagged autocorrelation sum, so it underestimates $\tau$ by half. C applies some inflated doubling incorrectly, arriving at $\tau = 4$ and $n = 40{,}000$. There's no justification for replacing $\tau = 3$ with $4$ — this overcorrects without theoretical basis. Study tip: Always memorize $\tau = 1 + 2\sum_{k=1}^{\infty}\rho_k$ for geometric autocorrelations — the geometric series simplifies cleanly, making this a common exam calculation. Never treat MCMC draws as independent when autocorrelation is given.
2. Ignoring integer-rounding issues in the optimization, which allocation minimizes the estimator's variance?
   Answer: $n_1=200$ and $n_2=100$, balancing weighted variability against square-root cost When you see a stratified sampling question with both unequal variances and unequal costs, your first instinct should be optimal (cost-adjusted Neyman) allocation, not plain Neyman allocation. The optimal rule says allocate sample sizes proportional to $W_h \sigma_h / \sqrt{c_h}$, then scale to meet the budget. Compute the allocation weights: - Stratum 1: $W_1\sigma_1/\sqrt{c_1} = 0.8 \times 1 / \sqrt{1} = 0.8$ - Stratum 2: $W_2\sigma_2/\sqrt{c_2} = 0.2 \times 4 / \sqrt{4} = 0.4$ The ratio is $n_1 : n_2 = 0.8 : 0.4 = 2:1$. Now apply the budget constraint $n_1 + 4n_2 = 600$ with $n_1 = 2n_2$: substituting gives $2n_2 + 4n_2 = 6n_2 = 600$, so $n_2 = 100$ and $n_1 = 200$. This confirms answer A is correct. Answer B ($n_1=120, n_2=120$) uses plain Neyman allocation — proportional to $W_h\sigma_h$ — but ignores the cost differential entirely. This is a classic trap: Neyman allocation only minimizes variance under equal costs. Answer C ($n_1=300, n_2=75$) allocates purely by stratum weight (0.8 and 0.2), ignoring both within-stratum variance and cost. This is proportional allocation, which is even cruder than Neyman. Answer D ($n_1=400, n_2=50$) doesn't correspond to any standard optimality criterion — it overloads the cheaper stratum without a principled justification. Study tip: Always check whether costs are equal before applying Neyman allocation. If $c_h$ differs across strata, you must include the $1/\sqrt{c_h}$ factor — forgetting it is the single most common error on optimal allocation problems.
3. Which statement correctly compares the two estimators?
   Answer: The antithetic estimator has smaller variance because the two transformed outputs have negative covariance Variance reduction via antithetic variates is a core Monte Carlo technique worth mastering deeply. When you see a question comparing two estimators that both "use two uniform draws," immediately ask: are those draws independent or negatively correlated? That distinction drives everything. The antithetic estimator is $\hat{\mu}_A = \frac{e^U + e^{1-U}}{2}$. Since $1-U \sim \text{Uniform}(0,1)$, both terms have the correct marginal distribution, so $E[\hat{\mu}_A] = E[e^U] = e - 1$—the estimator is unbiased. Its variance is $\frac{1}{4}\left(\text{Var}(e^U) + \text{Var}(e^{1-U}) + 2\text{Cov}(e^U, e^{1-U})\right)$. The key insight: because $e^x$ is convex and monotone, when $U$ is large, $1-U$ is small, making $e^U$ and $e^{1-U}$ move in opposite directions. This produces a negative covariance, shrinking the variance below that of the independent-pair estimator $\hat{\mu}_I = \frac{e^{U_1}+e^{U_2}}{2}$, whose covariance term is zero. Answer C correctly identifies this mechanism. A is wrong because dependence does not imply bias—both $U$ and $1-U$ are marginally uniform, preserving unbiasedness. B is wrong because equal marginal distributions do not guarantee equal variances; the covariance structure differs critically between the two estimators. D is wrong because the inputs summing to 1 does not make the outputs constant—$e^U + e^{1-U}$ still varies with $U$ (it equals $2\cosh(U - 0.5)$), so variance is positive, just reduced. Study tip: For antithetic variates, always verify two things—same marginals (ensures unbiasedness) and negative covariance (ensures variance reduction). Monotone transformations of $U$ and $1-U$ always achieve the latter.
4. What is the variance of this paired estimator, and how does it compare with using independent random inputs for the two systems?
   Answer: $4/n$, which is smaller than the independent-input variance by a factor of $13/4$ When comparing two stochastic systems, the key question is how correlation between paired outputs affects the variance of their difference. Recall that for any two random variables, $\text{Var}(X - Y) = \text{Var}(X) + \text{Var}(Y) - 2\text{Cov}(X,Y)$, where $\text{Cov}(X,Y) = \rho\sqrt{\text{Var}(X)\cdot\text{Var}(Y)}$. Here, $\text{Var}(X) = 9$, $\text{Var}(Y) = 4$, and $\rho = 0.75$. So $\text{Cov}(X,Y) = 0.75\sqrt{9 \cdot 4} = 0.75 \times 6 = 4.5$. The variance of a single paired difference is $9 + 4 - 2(4.5) = 13 - 9 = 4$. Averaging over $n$ independent pairs gives variance $4/n$, confirming A is correct. Under independent inputs, the systems are uncorrelated, so the variance would be $(9 + 4)/n = 13/n$. The ratio is $13/4$, meaning the paired estimator is smaller by exactly that factor — the whole point of common random numbers (CRN) as a variance reduction technique. Choice B uses $9/n$, which would follow if you mistakenly dropped $\text{Var}(Y)$ from the calculation. Choice C gives $13/n$ — this is the independent-input variance, reflecting a failure to apply the covariance term at all; it's the benchmark you're comparing against, not the CRN result. Choice D claims variance $22/n$, which would arise from adding the covariance term instead of subtracting — a sign error that would only apply if you were computing $\text{Var}(X + Y)$. The key study tip: CRN works precisely because positive correlation reduces the variance of a difference. Whenever you see paired simulation experiments with positive correlation, expect the paired variance to be strictly less than the sum of marginals — and always subtract $2\text{Cov}$ for differences.
5. Using a normal approximation, what is the smallest total number of replications required for a two-sided $95\%$ Monte Carlo confidence interval to have half-width at most $0.10$?
   Answer: $3{,}458$ replications, because the half-width scales inversely with the square root of sample size Whenever you see a Monte Carlo confidence interval question, anchor yourself to the standard formula: the half-width of a two-sided 95% interval is $z_{\alpha/2} \cdot \frac{\sigma}{\sqrt{n}}$, where $z_{0.025} = 1.96$, $\sigma^2$ is the planning variance, and $n$ is the number of replications. Your goal is to solve for the smallest $n$ such that this half-width is at most the target. Here, the pilot study gives $\hat{\sigma}^2 = 9$, so $\hat{\sigma} = 3$. Setting the half-width condition: $1.96 \cdot \frac{3}{\sqrt{n}} \leq 0.10$ Solving: $\sqrt{n} \geq \frac{1.96 \times 3}{0.10} = 58.8$, so $n \geq 58.8^2 = 3{,}457.44$. Rounding up gives $n = 3{,}458$, confirming C is correct. The half-width shrinks at rate $1/\sqrt{n}$, not $1/n$, which is the critical insight. Choice A confuses the half-width with the standard error itself — it skips multiplying by $z_{0.025} = 1.96$ and sets $\sigma/\sqrt{n} \leq 0.10$, yielding only $n \geq 900$. That ignores the confidence coefficient entirely. Choice B applies an incorrect $1/n$ scaling relationship instead of $1/\sqrt{n}$, producing a nonsensical intermediate formula. Choice D doubles the correct answer as if both endpoints each independently require the full half-width precision, which is a misunderstanding — the half-width already accounts for both directions symmetrically. Your study tip: always write the half-width formula explicitly before solving. The most common trap on precision-planning problems is forgetting either the $z$-multiplier (leading to A) or using $n$ instead of $\sqrt{n}$ as the denominator (leading to B or D).
6. Assuming the proposal covers the target support and the needed first moments are finite, which statement is generally correct?
   Answer: $\widehat\mu$ is generally biased for finite $n$ but consistent for $E_p[h(X)]$ as $n$ increases Whenever you see a question involving ratio estimators in importance sampling, your instinct should be to think carefully about the distinction between exact unbiasedness and asymptotic unbiasedness (consistency). The self-normalized estimator $\widehat\mu = \frac{\sum_i w_i h(X_i)}{\sum_i w_i}$ is a ratio of two random quantities. Even though the unknown normalizing constant $Z$ cancels algebraically, the ratio of two unbiased estimators is not generally unbiased — this is a classic result in statistics. Formally, $E\left[\frac{A}{B}\right] \neq \frac{E[A]}{E[B]}$ unless $A$ and $B$ are independent or the ratio is deterministic. However, by the Law of Large Numbers, both $\frac{1}{n}\sum_i w_i h(X_i) \xrightarrow{p} E_q[w(X)h(X)]$ and $\frac{1}{n}\sum_i w_i \xrightarrow{p} E_q[w(X)] = Z$, so their ratio converges in probability to $E_p[h(X)]$, confirming B is correct. A is wrong because cancellation of $Z$ does not eliminate the finite-sample bias introduced by taking the expectation of a ratio. The bias is $O(1/n)$ and vanishes only asymptotically. C is wrong on two counts: rescaling weights to have mean 1 is algebraically equivalent to $\widehat\mu$ already written, and the estimator is consistent without any such additional requirement. D is wrong because $\widehat\mu$ converges to $E_p[h(X)]$ regardless of the choice of proposal $q$ (provided $q$ covers the support of $p$), not to an expectation under $q$. Study tip: On graduate-level exams, always flag ratio estimators — they are virtually never exactly unbiased, but they are often consistent. The phrase "bias-variance tradeoff" and "$O(1/n)$ bias" are your signals that self-normalization is the topic at hand.
7. Which proposal gives the smallest variance among the choices, and what is that variance per replication?
   Answer: $q=(4/13,9/13)$, with variance $0$ because the weighted output is constant Importance sampling works by rewriting $E_p[h(X)] = E_q\left[\frac{h(X)p(X)}{q(X)}\right]$, so the estimator's variance depends entirely on how much the importance weights $w(x) = h(x)p(x)/q(x)$ vary across draws. The holy grail is a proposal that makes those weights constant, because a constant random variable has zero variance. The optimal proposal is proportional to $|h(x)|p(x)$. Here, $h(0)p(0) = 1 \times 0.8 = 0.8$ and $h(1)p(1) = 9 \times 0.2 = 1.8$, giving a normalizing constant of $0.8 + 1.8 = 2.6 = 13/5$. This yields $q^*(0) = 0.8/2.6 = 4/13$ and $q^*(1) = 1.8/2.6 = 9/13$. Under this proposal, every weight equals $h(x)p(x)/q^*(x) = 2.6$ regardless of which $x$ is drawn — the estimator is constant, so its variance is exactly 0. This confirms C is correct. Choice A uses the target itself as the proposal, so weights are all 1 and the estimator simply averages $h(X)$. The variance is $E_p[h^2] - (E_p[h])^2 = (0.8 \cdot 1 + 0.2 \cdot 81) - (0.8 + 1.8)^2 = 17 - 6.76 = 10.24$. This is valid but far from optimal. Choice B uses uniform weights; you can verify the weighted outputs are $1.6$ and $3.6$ with mean $2.6$ and variance $1$, better than A but not zero. Choice D inverts the sensible direction — piling mass on the rare, high-value state without the precise optimal ratio still leaves weight variability, yielding variance $0.64$. Your takeaway: whenever you see an importance-sampling variance question, immediately check whether any proposal makes $h(x)p(x)/q(x)$ constant. That proposal, $q \propto h(x)p(x)$, always achieves zero variance and is the theoretical benchmark you should recognize on sight.
8. Which quantity should be averaged over independent draws of $X$ to obtain the conditional estimator, and what variance guarantee does it satisfy?
   Answer: $I(X\geq 2)+e^{-(2-X)}\,I(X<2)$, with variance no larger than the crude estimator by the law of total variance Conditional Monte Carlo is a variance reduction technique based on the law of total variance: $\text{Var}(Z) = \text{Var}(E[Z|X]) + E[\text{Var}(Z|X)]$, which guarantees $\text{Var}(E[Z|X]) \leq \text{Var}(Z)$ always. The strategy is to replace a noisy indicator with its conditional expectation given observed information. For $Z = I(X+Y>2)$ with $Y \sim \text{Exp}(1)$ independent of $X$, compute $E[Z \mid X]= P(Y > 2-X \mid X)$. When $X \geq 2$, the event is certain: $P(Y > 2-X) = 1$. When $X < 2$, you need $P(Y > 2-X) = e^{-(2-X)}$ by the exponential survival function. This gives the conditional estimator $I(X \geq 2) + e^{-(2-X)}I(X < 2)$, averaged over draws of $X$ — exactly answer B. The law of total variance then guarantees variance no larger than the crude estimator. Choice A omits the $I(X \geq 2)$ term entirely, so it's a biased estimator — it assigns zero when $X \geq 2$ even though the event is guaranteed. Its variance claim of "strictly smaller for every integrand" is also too strong; the law of total variance only guarantees non-strict inequality. Choice C uses $e^{-(2+X)}$ instead of $e^{-(2-X)}$, which is simply the wrong survival probability — a sign error inside the exponent. The variance claim about iterated expectation is also a non sequitur. Choice D replaces $e^{-(2-X)}$ with the constant $e^{-2}$, as if substituting $X=0$, which is neither the conditional expectation nor an unbiased quantity. Your study tip: whenever you condition on a simulated variable, always write out $E[I(\text{event}) \mid \text{observed}]$ explicitly — identify each case (certain, impossible, or in-between) before writing the formula.
9. Which coefficient is variance minimizing, and what is the resulting variance per replication before division by the number of replications?
   Answer: $\beta=1.50$ and variance $16$, using covariance divided by the control variance Whenever you encounter a control variate problem, your instinct should be to minimize the variance of the adjusted estimator $\overline{Y} - \beta(\overline{C} - \mu_C)$. Since replications are independent, the variance of this estimator (per replication, before dividing by $n$) is: $\operatorname{Var}(Y - \beta C) = \operatorname{Var}(Y) - 2\beta\operatorname{Cov}(Y,C) + \beta^2\operatorname{Var}(C)$ To minimize, take the derivative with respect to $\beta$ and set it to zero: $-2\operatorname{Cov}(Y,C) + 2\beta\operatorname{Var}(C) = 0 \implies \beta^* = \frac{\operatorname{Cov}(Y,C)}{\operatorname{Var}(C)}$ Now plug in the given values. Since $\operatorname{Corr}(Y,C) = 0.6$, $\operatorname{Var}(Y)=25$, $\operatorname{Var}(C)=4$, we get $\operatorname{Cov}(Y,C) = 0.6\sqrt{25}\sqrt{4} = 0.6 \times 5 \times 2 = 6$. Therefore $\beta^* = 6/4 = 1.50$. The minimized variance is: $25 - \frac{[\operatorname{Cov}(Y,C)]^2}{\operatorname{Var}(C)} = 25 - \frac{36}{4} = 25 - 9 = 16$ This confirms B is correct. Choice A misuses the formula by dividing correlation by the control standard deviation ($0.6/2 = 0.3 \times 5 = 0.75$) — that's not the covariance-to-variance ratio. Choice C incorrectly flips the sign of $\beta$; the subtraction in the estimator is already absorbed into the optimization setup. Choice D uses a ratio of standard deviations ($5/2 = 2.50$... or $\sqrt{25}/\sqrt{4}$), which is related to regression slope only if correlation were 1. Study tip: Always memorize $\beta^* = \operatorname{Cov}(Y,C)/\operatorname{Var}(C)$ and the resulting variance reduction formula $\operatorname{Var}(Y)(1 - \rho^2)$ — a quick sanity check here gives $25(1-0.36)=16$ ✓.
10. Which sample average is an unbiased importance-sampling estimator of the rare-event probability?
    Answer: $n^{-1}\sum_{i=1}^n I(X_i>5)\exp(-5X_i+12.5)$ Whenever you see an importance sampling problem, your first instinct should be to write out the likelihood ratio (the Radon-Nikodym derivative) between the target and proposal distributions. This ratio is the engine of the estimator. You want $P(Z > 5) = E_f[I(Z > 5)]$ where $f$ is the standard normal density. Importance sampling rewrites this as $E_g\!\left[I(X > 5)\frac{f(X)}{g(X)}\right]$, where $g$ is the proposal density. Here $f = N(0,1)$ and $g = N(5,1)$, so the likelihood ratio is: $\frac{f(x)}{g(x)} = \frac{\exp(-x^2/2)}{\exp(-(x-5)^2/2)} = \exp\!\left(-\frac{x^2}{2} + \frac{(x-5)^2}{2}\right) = \exp(-5x + 12.5)$ The unbiased estimator is therefore $n^{-1}\sum_{i=1}^n I(X_i > 5)\exp(-5X_i + 12.5)$, which is answer A. B flips the sign of the exponent, giving $\exp(5X_i - 12.5)$. This would correspond to the ratio $g/f$ rather than $f/g$ — it's the reciprocal of the correct weight and produces a heavily biased, explosive estimator. C uses the wrong indicator $I(X_i > 0)$ instead of $I(X_i > 5)$, so it estimates the wrong probability entirely — $P(Z > 0) = 0.5$, not the rare-event probability. D uses $\exp(-5X_i - 12.5)$, with the wrong sign on the constant 12.5; this comes from misexpanding $(x-5)^2$ and loses the cross-term correction. Study tip: Always expand the exponent in $f(x)/g(x)$ term-by-term and double-check the sign of every piece — sign errors in the weight are the most common trap in importance sampling questions.

Start this quiz