Business Analytics Quiz: A B Testing Pitfalls
10 questions · exam conditions
0:00
A B Testing PitfallsQuestion 1 of 10

An e-commerce company randomly assigns visitors to a standard checkout or a checkout offering free shipping. The free-shipping offer increases the fraction of visitors who reach the payment page. An analyst then compares purchase rates only among visitors who reached the payment page and concludes that free shipping has no effect.

What is the most important problem with the analyst's conclusion?

Reaching the payment page is a post-treatment selection criterion, so the restricted groups need not remain comparable despite initial randomization.
Reaching the payment page is a pretreatment engagement measure, so the analysis should stratify visitors by that measure before testing.
Purchase rate is a binary outcome, so a causal comparison requires replacing random assignment with matched observational groups.
Free shipping changes the treatment cost, so purchase conversion cannot be used as an experimental outcome without monetization.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: A B Testing Pitfalls

Practice A B Testing Pitfalls in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on A B Testing Pitfalls, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

An e-commerce company randomly assigns visitors to a standard checkout or a checkout offering free shipping. The free-shipping offer increases the fraction of visitors who reach the payment page. An analyst then compares purchase rates only among visitors who reached the payment page and concludes that free shipping has no effect.

What is the most important problem with the analyst's conclusion?

  1. Reaching the payment page is a post-treatment selection criterion, so the restricted groups need not remain comparable despite initial randomization. (correct answer)
  2. Reaching the payment page is a pretreatment engagement measure, so the analysis should stratify visitors by that measure before testing.
  3. Purchase rate is a binary outcome, so a causal comparison requires replacing random assignment with matched observational groups.
  4. Free shipping changes the treatment cost, so purchase conversion cannot be used as an experimental outcome without monetization.
Explanation: Whenever you see a question about experimental design, ask yourself: does the analysis preserve the randomization? Random assignment guarantees that treatment and control groups are comparable at the start, but that guarantee evaporates the moment you filter participants based on something that happens after treatment begins. Here, free shipping was the treatment, and it directly caused more visitors to reach the payment page. That means "reached the payment page" is a post-treatment outcome, not a stable background characteristic. When the analyst restricts the comparison to only those visitors, she's no longer comparing randomly assigned groups — she's comparing two self-selected subgroups that differ in ways beyond the treatment itself. For example, visitors who reached the payment page despite not getting free shipping may be unusually motivated buyers. The groups are no longer apples-to-apples, so any conclusion about purchase rates is confounded. This is precisely what A describes: a post-treatment selection criterion that destroys comparability. B is wrong because it misclassifies "reaching the payment page" as a pretreatment measure. It isn't — it's caused by the treatment, so stratifying on it introduces the same bias, not less. C is wrong because binary outcomes are perfectly valid in randomized experiments. You don't need to abandon randomization and switch to matched observational data just because the dependent variable is 0/1. D is wrong because it invents a constraint that doesn't exist — there's no rule requiring monetization adjustments before using conversion rates as an experimental outcome. The study tip here: watch for analyses that filter or condition on variables measured after treatment. This is called collider bias or post-treatment conditioning, and it's a classic trap in both experimental design and causal inference questions.

Question 2

A software company runs an A/B test whose prespecified primary outcome shows no significant overall effect. After reviewing the data, the analyst examines 1212 customer segments and finds that one segment has an estimated lift with p=0.03p=0.03. The analyst recommends launching the feature only to that segment.

What is the strongest next step before accepting the segment-specific recommendation?

  1. Treat the segment result as confirmatory because targeting only that segment avoids relying on the nonsignificant overall result.
  2. Test whether the segment's treatment effect differs from other segments, account for the search, and validate it in new data. (correct answer)
  3. Replace the segment p-value with the overall p-value because subgroup effects cannot exist without a significant overall effect.
  4. Combine the segment with a similar segment until its sample size matches the full experiment's average group size.
Explanation: Whenever you see a question involving post-hoc subgroup analysis, your alarm bells should ring about multiple comparisons and data dredging. The core issue here is that searching through segments after the primary outcome fails is a classic way to find spurious patterns. When an analyst scans 1212 segments, the probability of finding at least one false positive at α=0.05\alpha = 0.05 by chance alone is approximately 1−(0.95)12≈0.461 - (0.95)^{12} \approx 0.46. A raw p=0.03p = 0.03 from one of those segments is therefore far less impressive than it looks — it hasn't been corrected for the search. The right response, captured in B, is a three-step discipline: (1) formally test whether that segment's effect differs from the others using an interaction test, (2) apply a multiple-comparisons correction (e.g., Bonferroni or Benjamini-Hochberg) to account for the 12-segment search, and (3) validate the finding in a fresh, pre-registered experiment before acting on it. A is wrong because a subgroup result found post-hoc is exploratory, not confirmatory, regardless of how the company chooses to deploy. Reframing the launch doesn't change the statistical validity of the discovery. C contains a real principle — interaction effects should generally be detected before acting on subgroups — but the claim that subgroup effects cannot exist without a significant overall effect is false. Simpson's Paradox and opposing subgroup effects can mask each other in aggregate results. D is a distraction. Arbitrarily merging segments to match a sample-size target manipulates the data structure and introduces its own biases without addressing the core validity problem. Your study tip: whenever a question involves subgroup or segment analysis, ask yourself — was this pre-specified, or found by searching? If found by searching, correction and replication are always required.

Question 3

Under a true null effect, an analyst performs five approximately independent significance tests on five nonoverlapping weekly batches from the same A/B rollout. Each week, the analyst uses a threshold of 0.050.05 and stops the rollout permanently if that week's result is significant.

What is the approximate probability that the analyst stops for a false positive during the five weeks?

  1. 5.05.0 percent, because the stopping decision uses the same nominal threshold each week.
  2. 20.020.0 percent, because only the four tests after the first count as repeated looks.
  3. 22.622.6 percent, because the probability is 1−(0.95)51-(0.95)^5. (correct answer)
  4. 25.025.0 percent, because five weekly error probabilities can always be added exactly.
Explanation: Whenever you see a question involving repeated significance tests, think about the multiple comparisons problem: each independent test carries its own false positive risk, and those risks compound across tests. Under a true null effect, each individual test has a 1−0.05=0.951 - 0.05 = 0.95 probability of not producing a false positive. Because the five weekly batches are approximately independent, the probability of never stopping on a false positive across all five weeks is (0.95)5≈0.7738(0.95)^5 \approx 0.7738. The complement — the probability of stopping at least once — is 1−(0.95)5≈0.22621 - (0.95)^5 \approx 0.2262, or about 22.6 percent. This confirms C is correct. Choice A makes the classic error of treating repeated testing as if it were a single test. Just because you use α=0.05\alpha = 0.05 each week doesn't mean your overall false positive rate stays at 5%. The threshold governs each individual decision, not the family of decisions. Choice B arbitrarily excludes the first test from the compounding calculation — there's no statistical justification for treating the first test differently; each test independently contributes to the cumulative error rate. Choice D suggests the five probabilities can always be added exactly (5×0.05=0.255 \times 0.05 = 0.25). Simple addition assumes the events are mutually exclusive, but false positives across tests are not mutually exclusive — you could technically observe them in multiple weeks. The correct approach uses the complement rule, not direct addition. As a study tip: anytime you see independent repeated tests at the same threshold, reach for 1−(1−α)k1 - (1 - \alpha)^k to find the family-wise error rate. It almost always appears in A/B testing and sequential analysis questions.

Question 4

A mobile application randomly assigns users to an old or redesigned checkout. The redesign causes more application crashes before the usual analytics purchase event is recorded. The analyst excludes every user without a complete analytics record and compares purchase rates among the remaining users. Purchase completion is also available from an independent server-side order system.

Which analysis would best address the principal bias introduced by the exclusions?

  1. Remove an equal number of complete control records so that both analyzed groups have the same final sample size.
  2. Retain only complete analytics records and add device type as a covariate in a purchase regression to control for hardware differences.
  3. Compare only crash-free users across groups, since they received sufficient treatment exposure to support a fair efficacy estimate.
  4. Use the server-side purchase outcome for all randomized users and analyze them according to their original treatment assignments. (correct answer)
Explanation: Whenever you see a question about experimental bias from missing data, think about survivorship bias and the integrity of your original randomization. When participants are excluded non-randomly — especially in a way caused by the treatment itself — the remaining sample is no longer comparable between groups, and your causal estimate becomes unreliable. Here, the redesigned checkout causes more crashes, which means users in the treatment group are systematically more likely to have incomplete analytics records. When the analyst drops incomplete records, the surviving treatment group skews toward users who somehow completed the flow despite the crashes — likely more persistent, tech-savvy, or on more stable devices. This makes the groups fundamentally different, undermining the random assignment. The correct fix is D: use the independent server-side purchase outcome, which captures every randomized user regardless of whether the analytics event fired. Analyzing all users under their original assignments preserves the randomization — this is the intention-to-treat (ITT) principle — and the server-side data provides the outcome without the crash-induced missingness. A is wrong because balancing sample sizes by removing control records doesn't fix the selection bias; it just makes the surviving bias symmetric in size, not in composition. B is wrong because adding device type as a covariate only adjusts for one potential confounder — it doesn't recover the excluded users or fix the non-random dropout problem. C is wrong because restricting to crash-free users reintroduces the exact selection bias you're trying to eliminate; "crash-free" is itself a post-treatment outcome, not a neutral filter. When treatment itself causes missingness, your go-to solution is an outcome source unaffected by the mechanism creating the missing data, analyzed on the full randomized sample.

Question 5

A retailer simultaneously tests 2020 promotional messages against a control. It selects the message with the largest observed revenue lift and reports that observed lift as the expected gain from deployment. Even if every message has the same small true effect, sampling noise influences which message is selected.

Which statement best describes the resulting estimate and the most appropriate response?

  1. The estimate is generally too high because selection favors positive noise; confirm the winner on an independent holdout sample. (correct answer)
  2. The estimate is generally too low because control performance is reused; pool all treatment observations before deployment.
  3. The estimate is unbiased because random assignment protects estimates after selecting the largest treatment effect.
  4. The estimate is unbiased if the winner is significant; lower the significance threshold only when no message wins.
Explanation: Whenever you see a question involving selecting the best performer from multiple simultaneous tests, your instinct should jump to the winner's curse and multiple comparisons bias — two closely related ideas that cause reported results to mislead decision-makers. Here's the core logic: even if all 20 messages have identical true effects, random sampling noise will cause some to appear higher and some lower than reality. By construction, you pick the one with the largest observed lift — which is also the one that benefited most from favorable noise. That noise gets baked into your reported estimate, inflating it beyond the true effect. The more candidates you test, the worse this problem gets, because you're effectively taking the maximum of 20 noisy draws. Reporting that inflated number as your "expected gain" guarantees disappointment at deployment. The correct remedy is to validate the winner on a fresh, independent holdout sample, which hasn't been touched by the selection process and will give you an unbiased read on actual performance. This makes A the right answer. B is wrong because the problem isn't about reusing control observations — it's about selecting the maximum across treatments, which inflates estimates upward, not downward. C is wrong because random assignment protects within-test comparisons from confounding, but it does nothing to correct the upward bias introduced by selecting the largest of many noisy estimates. D is wrong on both counts: statistical significance doesn't eliminate selection bias, and loosening thresholds when no winner emerges only compounds the multiple-comparisons problem. Study tip: Anytime a question describes "pick the best from N simultaneous tests," expect selection bias to inflate the winner's observed effect — and remember that holdout validation is the standard fix.

Question 6

An online retailer plans to run an A/B test for 2828 days. Beginning on day 33, the analyst checks the ordinary fixed-horizon p-value every morning and will stop the experiment as soon as p<0.05p<0.05. No adjustment is made for repeated looks.

Which change would most directly preserve the validity of the retailer's claimed significance level while still permitting early stopping?

  1. Require a minimum of 100100 conversions per group before using the same 0.050.05 threshold at every subsequent look.
  2. Use a prespecified group-sequential boundary that allocates the type I error rate across the planned interim looks. (correct answer)
  3. Report a 9595 percent confidence interval whenever the ordinary p-value first falls below 0.050.05.
  4. Continue collecting data through day 2828 but report the smallest p-value observed during the experiment.
Explanation: Whenever you see a question about A/B testing and early stopping, focus on Type I error inflation. Every time you peek at a p-value and apply the same α=0.05\alpha = 0.05 threshold, you increase the probability of a false positive beyond 5%5\%. After 28 daily looks, the actual false-positive rate can exceed 20%20\%. The core problem is that the fixed-horizon p-value was never designed for repeated testing — you need a framework that formally accounts for multiple looks. Group-sequential testing (answer B) solves this directly. It pre-specifies how the total α=0.05\alpha = 0.05 budget is distributed across each planned interim look using boundaries like O'Brien-Fleming or Pocock. Each individual check uses a stricter threshold, so the cumulative Type I error stays at exactly 5%5\%. This preserves the claimed significance level while explicitly permitting early stopping — a perfect match to the question's requirement. Answer A is wrong because requiring 100 conversions per group only addresses sample size adequacy, not the repeated-testing problem. The threshold remains unadjusted, so Type I error inflation continues unchecked. Answer C is wrong because reporting a confidence interval alongside an already-inflated p-value doesn't fix the underlying error rate problem. The CI shares the same flawed threshold and provides no multiplicity correction. Answer D is the most dangerous trap. Reporting the minimum p-value observed over 28 days is pure data dredging — it guarantees you're selecting the most extreme result, which maximally inflates Type I error rather than controlling it. Study tip: When a question combines "early stopping" with "valid significance level," the answer almost always involves a sequential testing boundary — the mechanism specifically designed for that tradeoff.

Question 7

A marketing team tests one new landing page and evaluates six independent KPIs. Under the global null hypothesis, each KPI is tested at significance level 0.050.05. The team plans to call the page successful if at least one KPI is significant, without applying any correction.

Approximately what is the probability that the team declares success even though the page affects none of the KPIs?

  1. 5.05.0 percent, because every individual test uses a significance level of 0.050.05.
  2. 26.526.5 percent, because the probability is 1−(0.95)61-(0.95)^6. (correct answer)
  3. 30.030.0 percent, because the six significance levels should be added directly.
  4. 73.573.5 percent, because (0.95)6(0.95)^6 is the probability of at least one false positive across six independent tests.
Explanation: When a team runs multiple independent hypothesis tests simultaneously without any correction, the risk of at least one false positive grows with each additional test — this is the multiple comparisons problem, and it's a classic trap in business analytics. Here's the core logic: if each test has a 5% chance of a false positive, it also has a 95% chance of not producing one. For six independent tests, the probability that none of them produce a false positive is (0.95)6≈0.735(0.95)^6 \approx 0.735. That means the probability of at least one false positive — declaring success when the page actually does nothing — is 1−(0.95)6≈0.2651 - (0.95)^6 \approx 0.265, or about 26.5%. That makes B correct. A is wrong because it confuses the per-test error rate with the family-wise error rate. A 5% significance level applies to each individual test in isolation, but the moment you run six tests and accept any one positive result as success, you've dramatically inflated your overall error rate. C applies faulty arithmetic — you cannot simply add probabilities across tests unless the events are mutually exclusive, which false positives across independent tests are not. D misreads the formula: (0.95)6(0.95)^6 is the probability of zero false positives, not at least one; D inverts the interpretation entirely. As a study tip, remember this formula: Family-Wise Error Rate = 1−(1−α)k1-(1-\alpha)^k, where α\alpha is the per-test significance level and kk is the number of tests. Any question mentioning multiple KPIs, no correction method, and "at least one" significant result should trigger this calculation immediately.

Question 8

A bank invites all customers to volunteer for a trial of a new budgeting tool. Among the customers who opt in, the bank randomly assigns half to receive the tool and half to a control group. The tool significantly reduces overdraft fees among participants. Volunteers are younger and use mobile banking more often than the bank's overall customer base.

Which conclusion is best supported by this experiment?

  1. The estimated effect is causally valid for volunteers, but applying it to all customers requires additional assumptions about generalizability. (correct answer)
  2. The estimated effect is noncausal even for volunteers because customers selected themselves into the original invitation pool.
  3. The estimated effect is causally valid for all customers because random assignment removes both internal and external selection bias.
  4. The estimated effect applies only to younger customers because age is the observed characteristic that differs between the populations.
Explanation: Whenever you see an experiment with both self-selection and random assignment, you need to separate two distinct questions: internal validity (did the treatment actually cause the observed effect?) and external validity (does that effect generalize beyond the study sample?). Here, volunteers were randomly assigned to either receive the tool or serve as controls. That randomization eliminates confounding within the volunteer pool — any differences in outcomes between the two groups can be attributed to the tool itself, not to pre-existing differences between participants. So the causal claim is solid, but only for the kind of person who opted in. Because volunteers are younger and more mobile-banking-savvy than the broader customer base, you cannot automatically assume the same effect size would appear across all customers. Extending the finding requires additional assumptions — for example, that the mechanism driving overdraft reduction isn't specific to tech-comfortable users. This is precisely what A captures, making it the correct answer. B is wrong because self-selection into the invitation pool doesn't undermine causality within that pool. Random assignment after opt-in is what controls for confounding, regardless of how participants came to be eligible. C is wrong because random assignment only handles internal validity. It does not resolve external validity — the sample's demographic skew is a real limitation that randomization cannot fix. D is wrong because it arbitrarily singles out age as the sole moderating variable. Mobile banking usage and other unobserved traits may matter equally, so restricting the conclusion to age alone is unjustified. Study tip: On experiment-design questions, always ask two separate questions — "Is the effect causal?" and "Who does it apply to?" Conflating these two is the most common trap.

Question 9

A retailer identifies stores in the lowest quartile of sales growth during the previous month. Within this selected group, it randomly assigns half to receive a new forecasting tool and half to continue using the existing process. Sales growth improves in both groups during the test month, but the treatment group's improvement is significantly larger.

Which interpretation most appropriately addresses selection and regression to the mean?

  1. Selecting low-growth stores invalidates the treatment comparison because both groups were expected to improve naturally.
  2. The result supports a causal effect for every store because randomization eliminates the consequences of sample selection.
  3. The larger treatment improvement must be regression to the mean because assignment used the prior month's performance.
  4. Randomization supports a causal effect for selected stores, although generalization beyond low-growth stores remains uncertain. (correct answer)
Explanation: Whenever you see a question combining randomization, sample selection, and regression to the mean, your job is to carefully separate what each concept actually controls for — and what it doesn't. Here's the key insight: regression to the mean occurs when units are selected based on an extreme performance, then naturally drift back toward average. In this scenario, low-growth stores were deliberately selected, so both groups will likely improve simply due to mean reversion. However, because stores were randomly assigned to treatment and control within that selected group, both groups face identical regression-to-mean pressure. The treatment group's significantly larger improvement therefore can't be explained away by mean reversion alone — it suggests the forecasting tool caused the additional gain. This is precisely what D captures: randomization supports a causal interpretation, but only for the population of low-growth stores actually studied. A is wrong because it overcorrects. Yes, both groups improve naturally, but randomization equalizes that natural improvement across groups, making the difference between them meaningful. Selection doesn't invalidate the comparison — it just limits generalizability. B is wrong because randomization doesn't eliminate the consequences of sample selection on external validity. You can draw valid causal conclusions within your sample, but you cannot automatically generalize to all stores. C is wrong because regression to the mean applies equally to both groups. If mean reversion were the sole driver, both groups would improve similarly — it cannot explain why the treatment group improved significantly more. Study tip: Remember that randomization fixes internal validity (causal inference within your sample) but never fixes external validity (generalization). Exam questions frequently conflate these two — keep them separate.

Question 10

A subscription company compares four new checkout layouts with its current layout. Before the experiment, management states that it cares only about whether each new layout outperforms the current layout; comparisons among the four new layouts are not decision-relevant. The analyst wants family-wise type I error no greater than 0.050.05 and will use a Bonferroni adjustment.

What per-comparison significance threshold should the analyst use?

  1. 0.0050.005, because all ten pairwise comparisons among the five layouts must be counted.
  2. 0.0100.010, because the family consists of the current layout and four new layouts.
  3. 0.01250.0125, because the prespecified family contains four comparisons with the current layout. (correct answer)
  4. 0.0500.050, because every new layout is tested against the same control layout.
Explanation: Whenever you see a Bonferroni correction question, your first job is to define the family of comparisons correctly — not just count every possible pair, but count only the comparisons that were prespecified as decision-relevant. The Bonferroni adjustment divides the target family-wise error rate by the number of tests in the family: αper=αfamilym\alpha_{per} = \frac{\alpha_{family}}{m}. The critical step is determining mm. Here, management explicitly stated that only comparisons between each new layout and the current layout matter — four new layouts versus one control gives exactly m=4m = 4 prespecified tests. Therefore: αper=0.054=0.0125\alpha_{per} = \frac{0.05}{4} = 0.0125, confirming answer C. Choice A is a common trap: it counts all pairwise combinations among five layouts ((52)=10\binom{5}{2} = 10), which would be appropriate if every head-to-head comparison were of interest. But since the analyst doesn't care about new-layout-vs.-new-layout comparisons, including those ten pairs inflates mm unnecessarily and produces an overly conservative threshold of 0.0050.005. Choice B divides by 5 (the total number of layouts) rather than the number of actual comparisons, which is a category error — you count tests, not groups. Choice D misreads the logic entirely: the fact that all tests share the same control does not collapse them into a single test. Four separate hypothesis tests still create four opportunities for a type I error. The strategy to remember: define the family before doing the math. The family size equals the number of prespecified hypothesis tests the analyst will actually conduct, not the number of groups or all possible pairs.