Simulation for Validation - Statistics Graduate Level

1. Which simulation design most directly assesses the finite-sample validity of the proposed asymptotic rejection rule under the stated null model?
   Answer: Generate observations from $\log X_i\sim N(-1/2,1)$, recompute $S$ in every sample, and record how often $|T_n|>1.96$. When evaluating whether an asymptotic test achieves its nominal size at a specific finite sample, you need a Monte Carlo study that perfectly mirrors the stated null model — same distribution, same parameters, same statistic, same critical value. Any deviation introduces a confound that makes it impossible to isolate the finite-sample behavior of the proposed rule. Option D does exactly this: it draws $X_i$ with $\log X_i \sim N(-1/2, 1)$, which guarantees $E[X_i]=1$ (the null is true by construction), recomputes $S$ fresh in every replicate just as the test requires, and checks whether $|T_n| > 1.96$. The empirical rejection frequency across many replicates estimates the true Type I error at $n=40$, directly answering whether the asymptotic $z$-approximation is valid here. Option A substitutes normal data for lognormal data. Even though the variance can be matched, the shape is completely different — lognormal is right-skewed, which is precisely why asymptotic approximation might struggle. Replacing the distribution defeats the purpose of the study. Option B describes a bootstrap procedure on observed data. This assesses variability of the statistic given a particular dataset, not the null-sampling behavior of the test across hypothetical repetitions of the experiment under the true null. Option C introduces two errors simultaneously: parameters are estimated without imposing the null (so the true mean may not equal 1), and the $t$ critical value replaces the proposed 1.96 threshold. You can't assess the validity of a specific rule by changing both the data-generating model and the decision boundary. Strategy tip: For simulation-validity questions, ask yourself whether the design faithfully reproduces every element of the stated problem — null distribution, test statistic formula, and critical value. Any substitution is a red flag.
2. Which conclusion best reflects a valid simulation assessment of the proposed asymptotic interval procedure?
   Answer: The reported $94\%$ is conditional coverage only; unconditional performance requires a prespecified rule for failed fits and inclusion of those outcomes. When evaluating any statistical procedure by simulation, you must think carefully about the unconditional coverage probability — the probability that the interval covers the true parameter across all samples the procedure will encounter in practice, not just the "nice" ones where estimation succeeds. The key insight is that unconditional coverage is a weighted average over every possible sample outcome: $P(\text{cover}) = P(\text{cover} \mid \text{no sep.}) \cdot P(\text{no sep.}) + P(\text{cover} \mid \text{sep.}) \cdot P(\text{sep.})$ Here, $P(\text{no sep.}) = 0.92$ and the conditional coverage among those samples is $0.94$. But what happens in the $8\%$ of separated samples? The problem doesn't say — the analyst never specified a fallback rule. Without knowing $P(\text{cover} \mid \text{sep.})$, the unconditional coverage is simply unknown. This is exactly what D captures: the $94\%$ figure is conditional coverage only, and completing the assessment requires a prespecified rule (e.g., report no interval, use a penalized estimator) that determines outcomes for failed fits. A is wrong because separation is not merely a "computational nuisance" — those samples are drawn from the same data-generating process and their coverage outcomes count toward the unconditional rate. Ignoring them inflates the apparent coverage. B is tempting because it attempts an unconditional calculation, but it incorrectly assumes separated samples always fail to cover (assigning them coverage probability zero). That assumption is unjustified without a specified rule. C is the most seductive distractor: conditional coverage being near nominal does not validate unconditional performance, because the conditioning event itself (non-separation) is sample-dependent. A useful habit: whenever a simulation reports coverage "among converged samples," immediately ask what happened to the rest — those outcomes are never free to ignore.
3. Which design most directly assesses whether the chi-square approximation is accurate for the stated procedure?
   Answer: Simulate null errors with autocorrelation $0.7$, rerun the bandwidth selection, and recompute the Wald statistic separately in every replicate. When assessing whether a chi-square approximation is accurate for a specific inferential procedure, you must replicate every stochastic component of that procedure — not a simplified or modified version of it. The question describes a Wald statistic whose covariance estimator uses a data-driven bandwidth. That bandwidth selection is itself a random variable, so its variability is part of the procedure's finite-sample behavior. Choice A is correct because it faithfully reproduces the full procedure under the null: errors are generated with the specified $\text{AR}(1)$ autocorrelation of $0.7$, bandwidth selection is rerun on each replicate, and the Wald statistic is recomputed fresh. The empirical distribution of these statistics can then be compared against the $\chi^2$ reference to assess approximation quality. This is the only design that matches what would happen in practice. Choice B fails because simulating independent errors ignores the autocorrelation structure entirely. This tests a different scenario — one where HAC correction may be unnecessary — so any size distortion found (or not found) is irrelevant to the stated procedure. Choice C is flawed in two ways: permuting residuals cannot reproduce the correct temporal dependence structure, and reusing a single covariance matrix eliminates the variability from estimation, artificially understating uncertainty in the test statistic's distribution. Choice D undermines the assessment by replacing the HAC estimator with the exact known covariance. This removes estimation error from the covariance matrix, which is central to whether the chi-square approximation holds for the actual procedure. You'd be evaluating a cleaner, oracle version that no practitioner could use. The key strategy: whenever a procedure involves adaptive or data-driven components (like bandwidth selection), a valid simulation study must re-execute those components in every replicate — otherwise you're evaluating a phantom procedure, not the real one.
4. Which interpretation best accounts for Monte Carlo uncertainty?
   Answer: There is evidence of modest size inflation because the Monte Carlo standard error is about $0.0016$, placing $0.05$ just below a two-sided $95\%$ interval. When you simulate a null hypothesis to assess test size, your observed rejection rate is itself a random variable with its own sampling distribution. The key framework here is treating the proportion of rejections across $B$ simulations as a binomial estimate, and applying standard error logic to determine whether any deviation from nominal size is statistically meaningful. Under the null, each simulation replicate is a Bernoulli trial with probability $p_0 = 0.05$. The Monte Carlo standard error of the estimated rejection rate is $\sqrt{p_0(1-p_0)/B} = \sqrt{(0.05)(0.95)/20{,}000} \approx 0.00154$, roughly $0.0016$. A two-sided 95% confidence interval around the observed rate of $0.0540$ is approximately $0.0540 \pm 1.96(0.0016)$, or about $(0.0509,\ 0.0571)$. Since $0.05$ falls just below this interval, there is modest but detectable evidence of size inflation — confirming B as the best answer. A is wrong because declaring any estimate within one percentage point "indistinguishable" ignores the actual precision of the simulation. With 20,000 replicates, you have enough resolution to detect deviations far smaller than one percentage point. C is wrong because the relevant standard error for evaluating the simulated rejection rate depends on the number of simulation replicates, not the original data sample size. These are completely separate sources of variability. D is wrong because computing $0.054/0.05 = 1.08$ without accounting for Monte Carlo uncertainty overstates precision. The ratio is an estimate, not an exact measurement, and calling the distortion "exactly 8%" ignores simulation noise. Study tip: Whenever a question gives you a simulated rejection rate and a number of replicates, immediately compute $\sqrt{\hat{p}(1-\hat{p})/B}$ — the Monte Carlo SE is the lens through which all conclusions must pass.
5. Which assessment of this claim is most appropriate?
   Answer: The claim is not established; assessing a convergence rate requires examining several sample sizes and accounting for Monte Carlo uncertainty in each estimated rejection probability. When evaluating claims about asymptotic convergence rates in simulation studies, you need to think carefully about what constitutes statistical evidence versus a plausible-looking pattern. A single pair of data points is almost never sufficient to establish a rate of convergence. Here's why C is correct: the analyst observes the excess rejection probability dropping from $0.030$ to $0.010$ when $n$ goes from $100$ to $400$. An $n^{-1/2}$ rate would predict the excess to scale by $\sqrt{100}/\sqrt{400} = 1/2$, giving $0.015$, not $0.010$. The observed drop is actually larger than what $n^{-1/2}$ predicts — more consistent with $n^{-1}$. But even setting that aside, two simulation estimates cannot establish a rate. Each estimated rejection probability carries Monte Carlo standard error (roughly $\sqrt{\hat{p}(1-\hat{p})/R}$), so the difference of two estimates inherits compounded uncertainty. You need multiple sample sizes to fit a trend and confidence intervals around each estimate to make a credible rate claim. A is wrong because quadrupling $n$ halves an $n^{-1/2}$ error only in the limit, and this "law" doesn't apply exactly at finite $n$ — nor does two points establish adherence to any asymptotic law. B is wrong because it misidentifies what should halve. Under $n^{-1/2}$ distortion, the excess over the nominal level scales as $n^{-1/2}$, not the rejection probability itself, so B's criterion is simply incorrect. D is wrong because equal replication controls one source of variance but doesn't resolve the fundamental problem: two data points are insufficient to establish a convergence rate regardless of replication balance. Study tip: When a question asks you to assess a convergence rate claim, always ask: how many data points were used, and was Monte Carlo uncertainty quantified? One pair of simulations is never enough.
6. Which conclusion is most consistent with the simulations and the boundary-limit theory?
   Answer: The simulations support the mixture limit because its tail probability above $3.84$ is approximately $\tfrac12(0.05)=0.025$. Whenever a likelihood-ratio test involves a parameter on the boundary of its parameter space (here, a variance component constrained to be ≥ 0), the standard $\chi^2_1$ null distribution no longer applies. Instead, the limiting null distribution is the mixture $\tfrac{1}{2}\chi^2_0 + \tfrac{1}{2}\chi^2_1$, where $\chi^2_0$ places a point mass of 1 at zero. Intuitively, about half the time the MLE hits the boundary (zero), producing a test statistic of exactly 0, and half the time it moves interior, producing a $\chi^2_1$ statistic. The key calculation: the tail probability of this mixture above the $\chi^2_1$ critical value $3.84$ is $\tfrac{1}{2} \cdot P(\chi^2_0 > 3.84) + \tfrac{1}{2} \cdot P(\chi^2_1 > 3.84) = \tfrac{1}{2}(0) + \tfrac{1}{2}(0.05) = 0.025$. The simulated rejection rate of ≈ 0.026 matches this almost exactly, confirming C is correct. A is wrong because calling 0.026 "reasonably close" to 0.05 misses the point — the discrepancy is systematic and theoretically predicted, not random noise. The test is anti-conservative by exactly the factor the mixture theory explains. B is wrong because it misreads the mixture. $\chi^2_0$ places all its mass at zero, so it contributes nothing to the upper tail — the mixture does not assign probability one-half above every positive value. D is wrong because the correct limit isn't purely $\chi^2_0$; it's the half-half mixture. Observing that half the mass concentrates at zero is only part of the picture. Study tip: On boundary-parameter problems, always check whether a mixture distribution applies — the rejection rate under the standard critical value will be exactly half the nominal level, a telltale signature you should recognize immediately.
7. What is the strongest generally justified conclusion from close agreement between the simulated and $\chi^2_1$ distributions?
   Answer: The chi-square approximation appears pointwise accurate near the fitted value of $\eta$, but poor behavior elsewhere in the nuisance space remains possible. When validating asymptotic approximations through simulation, you must think carefully about what the simulation actually varies versus what it holds fixed. Here, the analyst fixes $\hat{\eta}$ — a single estimated value of the nuisance parameter — and generates all simulated samples from that one point in the nuisance space. The simulation is essentially a pointwise check: "Does the $\chi^2_1$ approximation work when $\eta = \hat{\eta}$?" This makes C the strongest justified conclusion. Close agreement tells you the approximation is accurate near the fitted value, but because the entire simulation is anchored to one estimate of $\eta$, you have no information about how the approximation behaves at other values of $\eta$. The nuisance-parameter space can be large, and the LRT's finite-sample behavior can vary substantially across it — a phenomenon called non-uniform size distortion. A is wrong because using a consistent estimate of $\eta$ does not imply uniformity over the nuisance space. Consistency only guarantees convergence to the true value; it says nothing about approximation quality at other parameter points you never simulated. B is wrong because "exact at every sample size" is far too strong. The $\chi^2$ approximation is asymptotic — it improves with $n$ but is not exact at finite samples even under perfect conditions. Imposing the null restriction in simulation ensures you're sampling from the null, but that doesn't grant exactness. D is wrong because the question concerns approximation accuracy under the null (Type I error), not under alternatives. The nuisance parameter being estimated under the null is the setup, not a limitation on null-distribution validation. Study tip: Whenever a simulation study fixes nuisance parameters at a single estimated value, immediately recognize it as a pointwise validation — and flag any answer claiming uniform or global validity as overreach.
8. Which simulation scheme best validates the chi-square approximation for the procedure conditional on the observed design?
   Answer: Impose $R\beta=r$, generate heteroskedastic errors at the fixed design points, refit the model, and recompute the sandwich covariance in every replicate. When validating an asymptotic reference distribution via simulation, the golden rule is: your simulation must mimic exactly the conditions under which the statistic is derived. For a sandwich-covariance Wald statistic testing $H_0: R\beta = r$, this means three things must hold simultaneously — the null must be true, the error structure must match reality, and the covariance estimator must be the one actually used. Option A satisfies all three. By imposing $R\beta = r$, you ensure the simulated data genuinely come from the null distribution, so any rejection is a false positive — exactly what you're calibrating. By generating heteroskedastic errors at the fixed design points, you respect both the conditional-on-design assumption and the real error structure the sandwich estimator was built to handle. Recomputing the sandwich covariance in every replicate ensures the statistic is constructed identically to how it would be in practice. The resulting empirical distribution of the Wald statistic should then track $\chi^2$ if the approximation is valid. Option B fails because homoskedastic errors violate the very heteroskedasticity the test is designed for; you'd be validating the wrong scenario and potentially finding chi-square agreement that disappears in practice. Option C resamples the design matrix, breaking the conditional-on-design framework — the simulation no longer reflects the fixed-$X$ setting the asymptotic theory assumes. Option D keeps responses fixed (eliminating all Monte Carlo variation in the outcome) and uses a misspecified homoskedastic covariance, making it doubly invalid. Your study tip: whenever a simulation question involves hypothesis testing, ask yourself — "Is the null imposed? Is the data-generating process realistic? Is the estimator recomputed fresh?" All three must say yes.
9. Which next step would most efficiently provide a defensible simulation-based validation of the extreme-tail approximation?
   Answer: Use an importance-sampling distribution that makes exceedances more common and weight each replicate by the appropriate likelihood ratio. Whenever you see a question about validating an extreme-tail probability via simulation, your central challenge is variance: with a true probability of $p = 10^{-5}$, a naive simulation needs millions of replicates just to observe a handful of exceedances. The question is really asking which method makes rare-event estimation statistically credible and efficient. Importance sampling (D) is the correct tool here. You shift the sampling distribution toward the tail so that exceedances occur far more frequently, then correct each observation by multiplying by the likelihood ratio $w_i = dP_0/dQ(x_i)$, where $P_0$ is the null and $Q$ is the proposal. This yields an unbiased estimator with dramatically lower variance, giving you a defensible confidence interval around your tail probability estimate with far fewer replicates. A is wrong because a single exceedance out of 100,000 trials produces an estimate with enormous relative uncertainty — the 95% confidence interval spans roughly an order of magnitude. "Matching the asymptotic order of magnitude" with $n=1$ event is not statistical validation. B is wrong because antithetic variates reduce variance by exploiting negative correlation between pairs of replicates. This works well for smooth integrands but provides little to no guaranteed variance reduction for a rare binary indicator, since both antithetic samples typically equal zero simultaneously. C is wrong because retaining only exceedances and using their unweighted frequency is just naive Monte Carlo restricted to observed events — it does nothing to increase the rate at which rare events are generated, so it solves nothing. Your study tip: whenever $p \ll 1/n$, naive simulation is unreliable — reach immediately for importance sampling or other variance-reduction techniques designed specifically for rare events.
10. Which simulation procedure estimates the coverage of the interval actually being proposed?
    Answer: Generate many data sets at $\theta_0$, recompute both the estimate and its standard error, and record whether each resulting interval contains $\theta_0$. Whenever you see a question about coverage estimation, anchor yourself to this definition: the coverage probability of a confidence interval procedure is the long-run frequency with which the procedure — applied fresh to new data — captures the true parameter. Every ingredient of the procedure must be recomputed from scratch on each simulated dataset. B is correct because it faithfully mirrors the actual proposal. You fix $\theta_0$, generate many independent datasets from that model, and on each one you apply the full procedure: recompute $\widehat{\theta}$, recompute $\widehat{\operatorname{se}}(\widehat{\theta})$ using the same complex data-dependent method, form the interval $\widehat{\theta} \pm 1.96\,\widehat{\operatorname{se}}$, and check containment of $\theta_0$. The proportion of successes estimates finite-sample coverage — exactly what was asked. A describes a bootstrap resampling diagnostic applied to a single observed dataset. It approximates the sampling distribution of $\widehat{\theta}$ via resampling but never generates new data from $\theta_0$, so it cannot estimate coverage in the frequentist sense. C uses the true standard deviation and the Monte Carlo mean — neither of which is available in practice or part of the proposed interval. You would be evaluating an idealized procedure, not the one actually proposed. D fixes the standard error at its observed-data value instead of recomputing it on each new dataset. This ignores variability in $\widehat{\operatorname{se}}$, which is a key source of finite-sample coverage distortion — precisely what the statistician wants to study. Strategy tip: When evaluating a simulation procedure, mentally check a checklist — correct distribution, correct parameter value, and all procedure components recomputed independently. Any shortcut (fixed SE, single dataset, oracle quantities) means you're evaluating a different procedure than the one proposed.

Start this quiz