Permutation Tests - Statistics Graduate Level

Which procedure most faithfully constructs the design-based randomization distribution while retaining the serial structure of the observed outcomes?
Answer: Evaluate the statistic at each eligible implementation date and weight the values uniformly over those dates. Whenever you see a question about randomization-based inference, anchor yourself to one core principle: the randomization distribution must mirror exactly how randomization was actually conducted in the study design — nothing more, nothing less. Here, the implementation date was drawn uniformly from 20 prespecified eligible dates. That single randomization act is the only source of design-based uncertainty. Under the sharp null, every daily outcome is fixed — what varies is only which date was chosen. The correct procedure, then, is to ask: "What statistic value would we have observed had each of the other 19 eligible dates been chosen instead?" Answer D does precisely this — it evaluates the before/after difference statistic at every eligible date using the unchanged observed outcome series, then weights all 20 values equally. This perfectly reconstructs the design's randomization distribution and naturally preserves serial correlation because the outcome sequence is never touched. Answer A is fatally flawed because it shuffles the outcome values themselves, destroying the serial dependence structure and inventing a distribution that was never part of the study design. Answer B similarly manipulates the outcomes (via residual permutation), which alters the temporal autocorrelation pattern and departs from the actual randomization mechanism. Answer C reorders outcomes within the pre- and post-windows separately, which also breaks the serial structure and, more critically, does not correspond to any step in the original randomization — the date was randomized, not the outcomes around a fixed date. The key study tip: in design-based inference, ask "what did the randomization actually vary?" Then build your reference distribution by re-running the statistic over those randomized assignments only, leaving outcomes untouched.
Which procedure gives the appropriate randomization-test null distribution for the sharp null hypothesis of no treatment effect on any unit?
Answer: Enumerate admissible treatment sets and weight each statistic by that set's probability under the actual assignment design. Whenever you see a question about randomization (permutation) tests, anchor yourself to this core principle: the null distribution must mirror the actual probability mechanism used to assign treatments. The sharp null hypothesis of no effect means every unit's outcome is fixed regardless of assignment — so the test statistic's distribution comes entirely from re-randomizing according to the true design. Here, the design is non-uniform: different treatment sets carry different probabilities based on baseline risk scores. Under the sharp null, the correct null distribution weights each possible treatment set's statistic by the probability that set would actually be selected — exactly what A prescribes. This preserves the integrity of the design and produces a valid $p$ -value as $p = \sum_{\mathbf{z} \in \mathcal{Z}} \mathbf{1}[T(\mathbf{z}) \geq T_{\text{obs}}] \cdot P(\mathbf{Z} = \mathbf{z}).$ B is the classic trap for non-uniform designs. Equal weighting is only valid when the assignment mechanism is itself uniform — collapsing all sets to equal weight discards the actual design and yields a test that is neither valid nor properly sized. C confuses randomization inference with something closer to a label-shuffling permutation test. Holding treatment indicators fixed while permuting outcomes doesn't enumerate the reference distribution generated by the assignment mechanism, and it conflates the roles of outcomes and assignments. D describes a bootstrap procedure, which relies on sampling assumptions and large-sample approximations. Randomization tests make no distributional assumptions — they derive exact finite-sample distributions from the design itself, so resampling with replacement is conceptually mismatched here. The key study takeaway: always ask "what was the actual assignment mechanism?" and let that — not convenience or symmetry — define your reference distribution.
Which permutation strategy most directly uses the stated exchangeability assumption?
Answer: Permute exposure labels separately within each level of $X$ , preserving each stratum's exposed count. When a question mentions exchangeability conditional on a covariate, your first instinct should be to ask: "Where exactly does the randomization argument apply?" Conditional exchangeability means that within strata defined by $X$ , the exposed and unexposed units are interchangeable under the null — not across the entire sample. This is the core of stratified permutation testing. Choice C is correct because it operationalizes the assumption precisely: by permuting exposure labels separately within each level of $X$ , you respect the fact that $X$ is a confounder that must be held fixed. Within each stratum, the exposed count is preserved, reflecting the conditional null hypothesis that exposure assignment within that stratum is arbitrary. The resulting reference distribution accounts for $X$ 's influence on both exposure and outcome. Choice A is the classic trap: permuting labels across all subjects ignores the confounding structure of $X$ . Because exposure probability varies substantially by $X$ , unrestricted permutation mixes strata and produces a reference distribution contaminated by $X$ 's effect, inflating or deflating the test statistic under the null. Choice B permutes outcomes rather than exposure labels, which addresses a different kind of exchangeability — it would be relevant if you were testing whether the outcome distribution differs by $X$ , not whether exposure causes the outcome. Choice D permutes $X$ itself, which destroys the covariate structure entirely and is unrelated to any standard inference strategy described in the passage. The study tip: whenever you see "exchangeability conditional on $X$ ," that phrase is a direct signal to stratify your permutation by $X$ . Unconditional permutation tests ignore confounders; conditional (stratified) permutation tests control for them by design.
Under the sharp null, which randomization procedure is appropriate for inference based on the trial's design?
Answer: Permute intervention labels among clinics, preserving seven treated clinics, and recompute the regression statistic. When a trial randomizes clusters (here, clinics) rather than individuals, the unit of randomization governs what a valid permutation test must mimic. The sharp null says the intervention had no effect on any outcome — but to test it honestly, you must rerandomize the data exactly as the original randomization was conducted. Since the investigators randomized clinics to treatment or control (not individual patients), the permutation distribution must be built by shuffling treatment labels among clinics, not among patients. Choice C is correct because it respects this structure: permute the intervention label across the fourteen clinics (keeping seven treated), recompute the regression-adjusted statistic each time, and compare the observed statistic to that reference distribution. This faithfully replicates how chance operated in the actual design. Choice A is wrong because it permutes treatment at the patient level, ignoring the cluster structure entirely. This inflates the effective sample size and produces a reference distribution that is far too narrow — leading to anticonservative inference. Patients within the same clinic share common exposures and are not exchangeable across clinics under the design. Choice B is wrong in a different way: permuting outcomes within clinics while holding clinic assignments fixed tests nothing about the treatment effect — it destroys only within-clinic variation while preserving the clinic-level treatment contrast that actually matters. Choice D is wrong because shuffling clinic labels among patients scrambles cluster membership, not treatment assignment, and doesn't correspond to any step in the randomization process. The key strategy: always match the permutation scheme to the unit of randomization. In cluster-randomized trials, that unit is the cluster — full stop.
Under the sharp null hypothesis of no treatment effect, which resampling scheme correctly reflects the assignment design?
Answer: Independently reverse the treatment and control labels within each pair with probability $1/2$ . When conducting a randomization (permutation) test, the resampling scheme must mirror the exact random mechanism used in the original assignment — nothing more, nothing less. In a matched-pairs design, randomization happened within each pair independently: a fair coin determined which unit got treatment and which got control. Under the sharp null (no treatment effect whatsoever), the two outcomes within each pair are exchangeable, meaning flipping the labels inside any pair is equally likely and valid. This is precisely what B does: for each of the twelve pairs, independently flip the sign of the paired difference with probability $1/2$ . Over all $2^{12} = 4096$ equally likely sign combinations, you reconstruct the full randomization distribution of the test statistic — exactly as the design dictates. A is wrong because pooling all 24 outcomes and reshuffling ignores the pairing structure entirely. This destroys the within-pair correlation that matching was designed to exploit, producing an invalid reference distribution. C is wrong because bootstrapping is an inference tool for estimating sampling variability, not for conducting a randomization test. Bootstrapping doesn't respect the original assignment mechanism and introduces resampling noise unrelated to the null distribution. D applies a single sign to all twelve differences simultaneously — equivalent to just two possible outcomes (+all, −all) instead of $2^{12}$ . This collapses the reference distribution to two points, making it nearly useless and completely misrepresenting the independent within-pair randomizations. As a study tip: always ask yourself, "What was the actual random mechanism?" and build your resampling scheme to replicate it precisely — pair by pair, unit by unit, as independence requires.
Which reported Monte Carlo permutation value uses the standard finite-simulation correction that avoids assigning a zero value and yields a valid randomized-test calibration?
Answer: $5/100$ , because one is added to both the extreme count and total count. Whenever you see a question about permutation tests and Monte Carlo p-values, focus on a foundational principle: the observed test statistic itself represents one valid realization under the null hypothesis — namely, the original (unpermuted) data arrangement. Ignoring it creates a p-value that can equal zero, which is statistically invalid and philosophically incoherent (it would claim the data are impossible under the null). The standard finite-simulation correction, formalized by Davison & Hinkley and others, defines the Monte Carlo p-value as $\hat{p} = \frac{b + 1}{B + 1}$ , where $b$ is the number of simulated permutation statistics at least as extreme as the observed statistic, and $B$ is the number of simulated permutations. Here, $b = 4$ and $B = 99$ , giving $\frac{4+1}{99+1} = \frac{5}{100} = 0.05$ . This is answer C, and it's correct because adding 1 to both numerator and denominator effectively includes the observed arrangement as an additional "permutation" that trivially equals itself — guaranteeing $\hat{p} \geq 1/(B+1) > 0$ . Answer A ( $4/99$ ) omits the observed arrangement entirely, risks underestimating the true p-value, and can produce zero in other scenarios. Answer B ( $4/100$ ) adds 1 only to the denominator, inflating it without crediting the observed statistic as an extreme case, which underestimates the p-value. Answer D ( $5/99$ ) adds 1 only to the numerator while keeping the old denominator, which overcounts relative to the actual reference set and yields an inconsistent estimator. Your study tip: memorize the formula $\hat{p} = (b+1)/(B+1)$ and remember the justification — both counts shift by 1 because the observed data is one legitimate member of the full permutation reference set.
Which statement most accurately compares a raw difference-in-means statistic with a studentized difference-in-means statistic?
Answer: The studentized statistic can be asymptotically valid, whereas the raw-statistic permutation test can have incorrect size. When you see a permutation test applied under unequal variances, your first instinct should be to ask: what does permuting labels actually assume? A permutation test is exact only when observations are exchangeable under the null — meaning the joint distribution is truly invariant to label swaps. Equal means alone does not guarantee exchangeability when variances differ. The raw difference-in-means statistic, $\bar{X}_1 - \bar{X}_2$ , is the correct choice to scrutinize here. Under permutation, you're redistributing observations as if they came from a single common population. But if the two groups have different variances, this redistribution produces a null distribution that doesn't match the true sampling distribution of the statistic — leading to inflated or deflated Type I error rates. The test loses its size guarantee asymptotically. The studentized statistic, $T = \frac{\bar{X}_1 - \bar{X}_2}{\sqrt{\hat{\sigma}_1^2/n_1 + \hat{\sigma}_2^2/n_2}},$ rescales by group-specific variance estimates. A key theoretical result (Chung & Romano, and related work) shows that permutation of the studentized statistic is asymptotically valid even under unequal variances, because the studentization accounts for the heteroskedasticity and the permutation null distribution converges correctly. This makes D the right answer. A is wrong because equal means alone doesn't imply exchangeability — you need identical distributions for that. B has it exactly backwards: studentization fixes the heteroskedasticity problem rather than breaking it. C is overly restrictive; studentization allows asymptotic validity without requiring identical distributions. Study tip: On permutation test questions, always ask whether the null implies full exchangeability. When it doesn't (e.g., unequal variances), look for studentization as the asymptotic fix.
Which procedure best describes the Freedman-Lane approach to testing $H_0$ ?
Answer: Fit the reduced model using only $Z$ , permute its residuals, add them to the reduced-model fitted values to form a null response, and refit the full model containing both $Z$ and $X$ . When testing a coefficient of interest in the presence of nuisance covariates, the key challenge is preserving the null distribution while respecting the correlation structure between $X$ and $Z$ . Simply permuting $X$ or raw residuals can distort that structure and produce invalid tests. The Freedman-Lane procedure handles this elegantly. First, fit the reduced model $Y = Z\gamma + \varepsilon$ and collect its residuals $\hat{\varepsilon}$ . These residuals represent variation in $Y$ unexplained by $Z$ — exactly the component that should be exchangeable under $H_0: \beta = 0$ . Permute those residuals, then add them back to the reduced-model fitted values $\hat{Y}_Z = Z\hat{\gamma}$ to construct a synthetic null response $Y^* = Z\hat{\gamma} + \pi(\hat{\varepsilon})$ . Finally, refit the full model with both $Z$ and $X$ to $Y^*$ and record the test statistic. Repeating this yields the reference distribution under the null. This is answer C. Answer A permutes $X$ directly, which destroys its correlation with $Z$ and inflates the Type I error rate — the permuted $X$ no longer reflects the covariate structure observed in the data. Answer B permutes residuals from the full model, which already absorbs the signal from $\beta$ . This tends to be overly conservative and does not correctly isolate the null component. Answer D describes the Manly or residualization approach — permuting residuals of both $Y$ and $X$ after partialing out $Z$ — a different (and less standard) strategy that conflates two separate residualization steps. Remember: Freedman-Lane always uses reduced-model residuals added back to reduced-model fitted values, then refits the full model — that three-step rhythm is the signature of this method.
Which conclusion is justified by this result?
Answer: The sharp null is rejected, but a zero average treatment effect may still be compatible with heterogeneous effects. Whenever you see a question involving Fisher's randomization test, your first instinct should be to identify exactly which null hypothesis is being tested — because the conclusion is strictly limited to that hypothesis. Fisher's randomization test evaluates the sharp null hypothesis: that every unit's treatment effect is exactly zero, i.e., $Y_i(1) = Y_i(0)$ for all $i$ . Under this hypothesis, all potential outcomes are fully observed (the treated units' control outcomes are imputed as equal to what was observed), making the test statistic's exact distribution computable via rerandomization. When the test rejects at level 0.05, you have evidence against this specific sharp null — nothing more, nothing less. B is correct because it precisely captures this logic. Rejecting the sharp null tells you it is not the case that every unit has a zero effect. However, a zero average treatment effect (ATE) is still compatible with the data: some units could have large positive effects and others large negative effects, canceling out to an average of zero. Heterogeneity in effects is the key nuance. A is wrong because the sharp null and the null of zero average treatment effect are not equivalent. The sharp null is strictly stronger — it implies zero ATE, but zero ATE does not imply the sharp null. C is wrong because rejecting the sharp null doesn't mean every unit has a nonzero effect; it only means not all effects are simultaneously zero. D is wrong because the sharp null rejection does carry information — it is the sharp null, not the ATE null, that is directly tested. Study tip: Always distinguish the sharp null (unit-level) from the weak null (average-level) — these appear repeatedly in causal inference questions and require different testing frameworks.
What additional condition most directly supports that strong-control claim for a single-step max-statistic permutation procedure?
Answer: Subset pivotality, so the joint null distribution for any subset of true null hypotheses is unaffected by which other null hypotheses happen to be false. When a multiple-testing procedure must control the familywise error rate (FWER) strongly — meaning under any configuration of true and false null hypotheses, not just the global null — the key question is whether the permutation distribution of your test statistic remains valid even when some hypotheses are false. This is exactly what subset pivotality guarantees. Subset pivotality states that the joint distribution of the test statistics corresponding to the true null hypotheses is the same regardless of which other hypotheses are false. For a max-statistic permutation procedure, this means the permutation distribution of $M = \max_j |T_j|$ over the true nulls isn't distorted by treatment effects on the false nulls. Without it, false nulls could contaminate the permutation reference distribution, making your critical value invalid and destroying strong FWER control. With it, you can apply a single-step procedure across all endpoints and trust that the familywise rejection threshold is simultaneously valid for every partial null configuration. Answer A captures this precisely. Answer B is wrong because pairwise independence is far too strong a requirement and also unnecessary — permutation procedures exploit the joint dependence structure naturally. Factoring into marginal distributions isn't what you need. Answer C is wrong because equality of effect estimates is not a theoretical requirement for FWER control; it's a symmetry convenience that has no bearing on validity under partial nulls. Answer D is wrong because permutation procedures are nonparametric by design — you never need marginal normality or convergence to a parametric form; the permutation distribution is empirically derived. Study tip: Whenever you see "strong FWER control" with a max-statistic, immediately think subset pivotality — it's the standard sufficient condition in the Westfall–Young framework and a frequent exam target.

Start this quiz