- Which conclusion is best supported by this experiment?
Answer: The estimated effect is causally valid for volunteers, but applying it to all customers requires additional assumptions about generalizability.
Whenever you see an experiment with both self-selection and random assignment, you need to separate two distinct questions: internal validity (did the treatment actually cause the observed effect?) and external validity (does that effect generalize beyond the study sample?).
Here, volunteers were randomly assigned to either receive the tool or serve as controls. That randomization eliminates confounding within the volunteer pool — any differences in outcomes between the two groups can be attributed to the tool itself, not to pre-existing differences between participants. So the causal claim is solid, but only for the kind of person who opted in. Because volunteers are younger and more mobile-banking-savvy than the broader customer base, you cannot automatically assume the same effect size would appear across all customers. Extending the finding requires additional assumptions — for example, that the mechanism driving overdraft reduction isn't specific to tech-comfortable users. This is precisely what A captures, making it the correct answer.
B is wrong because self-selection into the invitation pool doesn't undermine causality within that pool. Random assignment after opt-in is what controls for confounding, regardless of how participants came to be eligible.
C is wrong because random assignment only handles internal validity. It does not resolve external validity — the sample's demographic skew is a real limitation that randomization cannot fix.
D is wrong because it arbitrarily singles out age as the sole moderating variable. Mobile banking usage and other unobserved traits may matter equally, so restricting the conclusion to age alone is unjustified.
Study tip: On experiment-design questions, always ask two separate questions — "Is the effect causal?" and "Who does it apply to?" Conflating these two is the most common trap.
- What per-comparison significance threshold should the analyst use?
Answer: 0.0125, because the prespecified family contains four comparisons with the current layout.
Whenever you see a Bonferroni correction question, your first job is to define the family of comparisons correctly — not just count every possible pair, but count only the comparisons that were prespecified as decision-relevant.
The Bonferroni adjustment divides the target family-wise error rate by the number of tests in the family: αper=mαfamily. The critical step is determining m. Here, management explicitly stated that only comparisons between each new layout and the current layout matter — four new layouts versus one control gives exactly m=4 prespecified tests. Therefore: αper=40.05=0.0125, confirming answer C.
Choice A is a common trap: it counts all pairwise combinations among five layouts ((25)=10), which would be appropriate if every head-to-head comparison were of interest. But since the analyst doesn't care about new-layout-vs.-new-layout comparisons, including those ten pairs inflates m unnecessarily and produces an overly conservative threshold of 0.005.
Choice B divides by 5 (the total number of layouts) rather than the number of actual comparisons, which is a category error — you count tests, not groups.
Choice D misreads the logic entirely: the fact that all tests share the same control does not collapse them into a single test. Four separate hypothesis tests still create four opportunities for a type I error.
The strategy to remember: define the family before doing the math. The family size equals the number of prespecified hypothesis tests the analyst will actually conduct, not the number of groups or all possible pairs. - What is the approximate probability that the analyst stops for a false positive during the five weeks?
Answer: 22.6 percent, because the probability is 1−(0.95)5.
Whenever you see a question involving repeated significance tests, think about the multiple comparisons problem: each independent test carries its own false positive risk, and those risks compound across tests.
Under a true null effect, each individual test has a 1−0.05=0.95 probability of not producing a false positive. Because the five weekly batches are approximately independent, the probability of never stopping on a false positive across all five weeks is (0.95)5≈0.7738. The complement — the probability of stopping at least once — is 1−(0.95)5≈0.2262, or about 22.6 percent. This confirms C is correct.
Choice A makes the classic error of treating repeated testing as if it were a single test. Just because you use α=0.05 each week doesn't mean your overall false positive rate stays at 5%. The threshold governs each individual decision, not the family of decisions. Choice B arbitrarily excludes the first test from the compounding calculation — there's no statistical justification for treating the first test differently; each test independently contributes to the cumulative error rate. Choice D suggests the five probabilities can always be added exactly (5×0.05=0.25). Simple addition assumes the events are mutually exclusive, but false positives across tests are not mutually exclusive — you could technically observe them in multiple weeks. The correct approach uses the complement rule, not direct addition.
As a study tip: anytime you see independent repeated tests at the same threshold, reach for 1−(1−α)k to find the family-wise error rate. It almost always appears in A/B testing and sequential analysis questions. - What is the most important problem with the analyst's conclusion?
Answer: Reaching the payment page is a post-treatment selection criterion, so the restricted groups need not remain comparable despite initial randomization.
Whenever you see a question about experimental design, ask yourself: does the analysis preserve the randomization? Random assignment guarantees that treatment and control groups are comparable at the start, but that guarantee evaporates the moment you filter participants based on something that happens after treatment begins.
Here, free shipping was the treatment, and it directly caused more visitors to reach the payment page. That means "reached the payment page" is a post-treatment outcome, not a stable background characteristic. When the analyst restricts the comparison to only those visitors, she's no longer comparing randomly assigned groups — she's comparing two self-selected subgroups that differ in ways beyond the treatment itself. For example, visitors who reached the payment page despite not getting free shipping may be unusually motivated buyers. The groups are no longer apples-to-apples, so any conclusion about purchase rates is confounded. This is precisely what A describes: a post-treatment selection criterion that destroys comparability.
B is wrong because it misclassifies "reaching the payment page" as a pretreatment measure. It isn't — it's caused by the treatment, so stratifying on it introduces the same bias, not less.
C is wrong because binary outcomes are perfectly valid in randomized experiments. You don't need to abandon randomization and switch to matched observational data just because the dependent variable is 0/1.
D is wrong because it invents a constraint that doesn't exist — there's no rule requiring monetization adjustments before using conversion rates as an experimental outcome.
The study tip here: watch for analyses that filter or condition on variables measured after treatment. This is called collider bias or post-treatment conditioning, and it's a classic trap in both experimental design and causal inference questions.
- Which change would most directly preserve the validity of the retailer's claimed significance level while still permitting early stopping?
Answer: Use a prespecified group-sequential boundary that allocates the type I error rate across the planned interim looks.
Whenever you see a question about A/B testing and early stopping, focus on Type I error inflation. Every time you peek at a p-value and apply the same α=0.05 threshold, you increase the probability of a false positive beyond 5%. After 28 daily looks, the actual false-positive rate can exceed 20%. The core problem is that the fixed-horizon p-value was never designed for repeated testing — you need a framework that formally accounts for multiple looks.
Group-sequential testing (answer B) solves this directly. It pre-specifies how the total α=0.05 budget is distributed across each planned interim look using boundaries like O'Brien-Fleming or Pocock. Each individual check uses a stricter threshold, so the cumulative Type I error stays at exactly 5%. This preserves the claimed significance level while explicitly permitting early stopping — a perfect match to the question's requirement.
Answer A is wrong because requiring 100 conversions per group only addresses sample size adequacy, not the repeated-testing problem. The threshold remains unadjusted, so Type I error inflation continues unchecked.
Answer C is wrong because reporting a confidence interval alongside an already-inflated p-value doesn't fix the underlying error rate problem. The CI shares the same flawed threshold and provides no multiplicity correction.
Answer D is the most dangerous trap. Reporting the minimum p-value observed over 28 days is pure data dredging — it guarantees you're selecting the most extreme result, which maximally inflates Type I error rather than controlling it.
Study tip: When a question combines "early stopping" with "valid significance level," the answer almost always involves a sequential testing boundary — the mechanism specifically designed for that tradeoff. - What is the strongest next step before accepting the segment-specific recommendation?
Answer: Test whether the segment's treatment effect differs from other segments, account for the search, and validate it in new data.
Whenever you see a question involving post-hoc subgroup analysis, your alarm bells should ring about multiple comparisons and data dredging. The core issue here is that searching through segments after the primary outcome fails is a classic way to find spurious patterns.
When an analyst scans 12 segments, the probability of finding at least one false positive at α=0.05 by chance alone is approximately 1−(0.95)12≈0.46. A raw p=0.03 from one of those segments is therefore far less impressive than it looks — it hasn't been corrected for the search. The right response, captured in B, is a three-step discipline: (1) formally test whether that segment's effect differs from the others using an interaction test, (2) apply a multiple-comparisons correction (e.g., Bonferroni or Benjamini-Hochberg) to account for the 12-segment search, and (3) validate the finding in a fresh, pre-registered experiment before acting on it.
A is wrong because a subgroup result found post-hoc is exploratory, not confirmatory, regardless of how the company chooses to deploy. Reframing the launch doesn't change the statistical validity of the discovery.
C contains a real principle — interaction effects should generally be detected before acting on subgroups — but the claim that subgroup effects cannot exist without a significant overall effect is false. Simpson's Paradox and opposing subgroup effects can mask each other in aggregate results.
D is a distraction. Arbitrarily merging segments to match a sample-size target manipulates the data structure and introduces its own biases without addressing the core validity problem.
Your study tip: whenever a question involves subgroup or segment analysis, ask yourself — was this pre-specified, or found by searching? If found by searching, correction and replication are always required. - Which interpretation most appropriately addresses selection and regression to the mean?
Answer: Randomization supports a causal effect for selected stores, although generalization beyond low-growth stores remains uncertain.
Whenever you see a question combining randomization, sample selection, and regression to the mean, your job is to carefully separate what each concept actually controls for — and what it doesn't.
Here's the key insight: regression to the mean occurs when units are selected based on an extreme performance, then naturally drift back toward average. In this scenario, low-growth stores were deliberately selected, so both groups will likely improve simply due to mean reversion. However, because stores were randomly assigned to treatment and control within that selected group, both groups face identical regression-to-mean pressure. The treatment group's significantly larger improvement therefore can't be explained away by mean reversion alone — it suggests the forecasting tool caused the additional gain. This is precisely what D captures: randomization supports a causal interpretation, but only for the population of low-growth stores actually studied.
A is wrong because it overcorrects. Yes, both groups improve naturally, but randomization equalizes that natural improvement across groups, making the difference between them meaningful. Selection doesn't invalidate the comparison — it just limits generalizability.
B is wrong because randomization doesn't eliminate the consequences of sample selection on external validity. You can draw valid causal conclusions within your sample, but you cannot automatically generalize to all stores.
C is wrong because regression to the mean applies equally to both groups. If mean reversion were the sole driver, both groups would improve similarly — it cannot explain why the treatment group improved significantly more.
Study tip: Remember that randomization fixes internal validity (causal inference within your sample) but never fixes external validity (generalization). Exam questions frequently conflate these two — keep them separate.
- Which analysis would best address the principal bias introduced by the exclusions?
Answer: Use the server-side purchase outcome for all randomized users and analyze them according to their original treatment assignments.
Whenever you see a question about experimental bias from missing data, think about survivorship bias and the integrity of your original randomization. When participants are excluded non-randomly — especially in a way caused by the treatment itself — the remaining sample is no longer comparable between groups, and your causal estimate becomes unreliable.
Here, the redesigned checkout causes more crashes, which means users in the treatment group are systematically more likely to have incomplete analytics records. When the analyst drops incomplete records, the surviving treatment group skews toward users who somehow completed the flow despite the crashes — likely more persistent, tech-savvy, or on more stable devices. This makes the groups fundamentally different, undermining the random assignment. The correct fix is D: use the independent server-side purchase outcome, which captures every randomized user regardless of whether the analytics event fired. Analyzing all users under their original assignments preserves the randomization — this is the intention-to-treat (ITT) principle — and the server-side data provides the outcome without the crash-induced missingness.
A is wrong because balancing sample sizes by removing control records doesn't fix the selection bias; it just makes the surviving bias symmetric in size, not in composition. B is wrong because adding device type as a covariate only adjusts for one potential confounder — it doesn't recover the excluded users or fix the non-random dropout problem. C is wrong because restricting to crash-free users reintroduces the exact selection bias you're trying to eliminate; "crash-free" is itself a post-treatment outcome, not a neutral filter.
When treatment itself causes missingness, your go-to solution is an outcome source unaffected by the mechanism creating the missing data, analyzed on the full randomized sample.
- Approximately what is the probability that the team declares success even though the page affects none of the KPIs?
Answer: 26.5 percent, because the probability is 1−(0.95)6.
When a team runs multiple independent hypothesis tests simultaneously without any correction, the risk of at least one false positive grows with each additional test — this is the multiple comparisons problem, and it's a classic trap in business analytics.
Here's the core logic: if each test has a 5% chance of a false positive, it also has a 95% chance of not producing one. For six independent tests, the probability that none of them produce a false positive is (0.95)6≈0.735. That means the probability of at least one false positive — declaring success when the page actually does nothing — is 1−(0.95)6≈0.265, or about 26.5%. That makes B correct.
A is wrong because it confuses the per-test error rate with the family-wise error rate. A 5% significance level applies to each individual test in isolation, but the moment you run six tests and accept any one positive result as success, you've dramatically inflated your overall error rate. C applies faulty arithmetic — you cannot simply add probabilities across tests unless the events are mutually exclusive, which false positives across independent tests are not. D misreads the formula: (0.95)6 is the probability of zero false positives, not at least one; D inverts the interpretation entirely.
As a study tip, remember this formula: **Family-Wise Error Rate = 1−(1−α)k**, where α is the per-test significance level and k is the number of tests. Any question mentioning multiple KPIs, no correction method, and "at least one" significant result should trigger this calculation immediately. - Which statement best describes the resulting estimate and the most appropriate response?
Answer: The estimate is generally too high because selection favors positive noise; confirm the winner on an independent holdout sample.
Whenever you see a question involving selecting the best performer from multiple simultaneous tests, your instinct should jump to the winner's curse and multiple comparisons bias — two closely related ideas that cause reported results to mislead decision-makers.
Here's the core logic: even if all 20 messages have identical true effects, random sampling noise will cause some to appear higher and some lower than reality. By construction, you pick the one with the largest observed lift — which is also the one that benefited most from favorable noise. That noise gets baked into your reported estimate, inflating it beyond the true effect. The more candidates you test, the worse this problem gets, because you're effectively taking the maximum of 20 noisy draws. Reporting that inflated number as your "expected gain" guarantees disappointment at deployment. The correct remedy is to validate the winner on a fresh, independent holdout sample, which hasn't been touched by the selection process and will give you an unbiased read on actual performance. This makes A the right answer.
B is wrong because the problem isn't about reusing control observations — it's about selecting the maximum across treatments, which inflates estimates upward, not downward.
C is wrong because random assignment protects within-test comparisons from confounding, but it does nothing to correct the upward bias introduced by selecting the largest of many noisy estimates.
D is wrong on both counts: statistical significance doesn't eliminate selection bias, and loosening thresholds when no winner emerges only compounds the multiple-comparisons problem.
Study tip: Anytime a question describes "pick the best from N simultaneous tests," expect selection bias to inflate the winner's observed effect — and remember that holdout validation is the standard fix.
