- Which metric specification provides the most defensible test of the hypothesis?
Answer: Purchases per assigned account, paired with unsubscribes per assigned account as a guardrail, regardless of delivery or opening.
When evaluating A/B test metrics, your guiding principle should be intention-to-treat (ITT): measure outcomes relative to the group as randomly assigned, not as filtered by downstream behavior. This preserves the integrity of randomization and avoids selection bias.
Answer B is correct because both metrics — purchases and unsubscribes — are calculated per assigned account, the true unit of randomization. This captures the full causal effect of sending the redesigned email, including deliverability problems. If the redesign reaches fewer inboxes due to image-heavy filtering, that's part of the treatment effect and should count against it. Using assigned accounts as the denominator for both metrics keeps the comparison clean and statistically defensible.
Answer A introduces severe selection bias by conditioning on opens. Customers who open emails aren't random — they're systematically different between groups, especially if deliverability differs. Comparing purchase rates among openers is no longer a causal comparison between treatments.
Answer C conditions on successful delivery, which excludes accounts harmed by the redesign's deliverability issues. This artificially inflates the redesign's apparent performance by removing the very cases where it underperformed.
Answer D uses two incompatible denominators: revenue per purchasing account (a highly filtered subgroup) paired with raw unsubscribe counts. Mixing these denominators makes the guardrail metric uninterpretable and the primary metric biased toward customers who were already going to buy.
Strategy tip: On any A/B testing question, ask yourself, "Is the denominator the randomized unit?" If the denominator filters on post-assignment behavior (opens, clicks, deliveries, purchases), selection bias has been introduced — that answer is almost certainly wrong.
- Which experimental hypothesis best tests whether the new model should be deployed?
Answer: Using the new model's targeting policy will increase retained contribution margin per eligible account, with offer cost and complaints included as guardrails.
When evaluating whether a predictive model should actually be deployed, you need to shift from thinking like a data scientist to thinking like a business decision-maker. A model's statistical performance (like AUC) only matters insofar as it translates into better real-world outcomes. The right experimental hypothesis should test the full business value chain: targeting quality, economic returns, and unintended consequences.
Option D is correct because it frames the hypothesis around the metric that actually matters — retained contribution margin per eligible account — while incorporating offer costs and complaint rates as guardrails. This design captures both upside (revenue retained) and downside risks (cost overruns, customer friction), which is exactly what a deployment decision requires. A properly structured business experiment must connect model outputs to financial outcomes, not just predictive statistics.
Option A is a trap: a higher observed churn rate among targeted accounts would actually suggest worse targeting, not better. Even if this were reversed, raw churn rates ignore offer costs and don't reflect net economic value.
Option B limits analysis to customers who accepted an offer, which introduces severe survivorship bias. You're no longer measuring targeting effectiveness — you're measuring something about self-selected acceptors, which tells you nothing about the model's deployment value.
Option C simply re-evaluates AUC on the experiment population. But the passage already tells you the new model has better AUC. Re-confirming a statistical metric doesn't tell you whether the model improves business outcomes, which is the entire question.
The broader lesson: whenever a question asks whether a model should be deployed, look for the answer that ties model outputs to economic impact with appropriate constraints — not the one that celebrates better statistics.
- Which interpretation is most consistent with the predefined success criterion?
Answer: The evidence is insufficient to declare success because the interval includes effects below both zero and the $0.20 target.
Whenever you see a question combining a confidence interval with a predefined success threshold, resist the temptation to focus only on the point estimate. You need to ask: does the entire confidence interval support the claim?
Here, success requires a contribution margin lift of at least $0.20. The point estimate of $0.25 looks promising, but the 95% confidence interval runs from −$0.05 to $0.55. That interval spans a wide range of plausible true effects — some below zero, many below the $0.20 threshold. Because the data are compatible with effects that would not meet the success criterion, you cannot confidently declare success. That's exactly what C captures: the interval includes values below both zero and the $0.20 target, making the evidence insufficient.
A is the classic trap: cherry-picking the point estimate while ignoring uncertainty. A single number like $0.25 is your best guess, not a guaranteed outcome — the interval tells you how much that guess could be off. B makes a logical error in two ways: containing zero doesn't prove no effect (absence of evidence isn't evidence of absence), and the standard for this test isn't simply "above zero" — it's "above $0.20." D misuses the upper bound; that $0.55 ceiling represents an optimistic extreme, not a reliable outcome to celebrate.
Study tip: When a question defines a minimum meaningful effect, treat that threshold exactly like zero in a standard significance test — success requires the entire confidence interval to fall above it, not just the point estimate. - Under the preregistered criteria, what conclusion should the analyst report?
Answer: The test succeeds because contribution per assigned visitor rises by about 3.9% and the return-rate increase remains within the guardrail.
Whenever you see an A/B test question with preregistered success criteria, your job is to evaluate the exact metrics specified beforehand — not whatever looks impressive after the fact. Here, success requires (1) at least a 3% lift in contribution per assigned visitor and (2) no more than a 1.0-percentage-point increase in return rate.
Start with the primary metric. Control contribution per assigned visitor: 2000240×$30=$3.60. Treatment: 2000220×$34=$3.74. The lift is 3.603.74−3.60≈3.9%, which clears the 3% threshold. Next, check the guardrail: the return rate rises from 4.0% to 4.6%, an increase of 0.6 percentage points — within the 1.0-point limit. Both criteria are satisfied, making B the correct conclusion.
A is wrong because it fixates on the conversion count dropping (220 vs. 240) without computing the actual prespecified metric. Fewer purchasers paying more can still improve contribution per assigned visitor — and it does here.
C confuses the primary metric with contribution per purchaser. Yes, $30$34−$30≈13.3%, but that was never the success criterion. Applying the wrong formula to the right data is a classic trap.
D is wrong because it ignores the guardrail entirely. Preregistered criteria exist precisely to prevent cherry-picking favorable metrics after results come in.
The key habit: always anchor your analysis to the prespecified metrics, not whichever numbers look best post-hoc. If a guardrail is listed, it is binding — treat it as a hard constraint, not a suggestion. - Which decision rule most appropriately operationalizes this hypothesis?
Answer: Deploy if evidence supports at least a 15% cost reduction and the lower confidence bound for the bot-minus-control satisfaction difference exceeds −2 points.
When translating business constraints into statistical decision rules, you need to match each requirement to the right inferential tool. Here, the bank has two separate thresholds: a minimum benefit (cost reduction ≥ 15%) and a maximum harm (satisfaction loss ≤ 2 points). Each threshold demands a different type of evidence.
Answer C correctly handles both. For the cost reduction, requiring that evidence "supports at least 15%" means the confidence interval for cost savings should sit above 15% — you need the lower bound to clear the threshold, not just the point estimate. For satisfaction, the concern is harm, so you need the lower confidence bound of the bot-minus-control difference to exceed −2. If the worst plausible outcome is better than −2 points, you've ruled out unacceptable harm. This is the logic of non-inferiority testing applied to a business gate.
Answer A is flawed because "statistically different from zero" only tells you the effect isn't zero — a 1% cost reduction could be significant but useless. The satisfaction criterion is also nonsensical as written.
Answer B relies solely on a point estimate for cost, ignoring uncertainty. A 15% estimate with wide confidence intervals could easily be below the threshold in reality. Using non-significance for satisfaction also conflates "no detectable harm" with "acceptable harm."
Answer D checks the upper bound for cost reduction, which is the optimistic end — it can exceed 15% even when the true effect is far lower. That's the wrong tail to examine for a minimum-benefit requirement.
The study tip: whenever a question sets a minimum required effect, check the lower confidence bound; for a maximum tolerated harm, also check the lower bound of the difference. The conservative end of the interval is what protects the decision. - Which primary hypothesis and metric are most appropriate?
Answer: The new flow will increase ninety-day contribution margin per assigned user, with seven-day activation treated as a secondary leading indicator.
When designing an experiment around a business objective, you need to align your primary hypothesis and metric directly with what the business actually cares about — not a convenient proxy. Here, the stated goal is customer value, not short-term engagement, which immediately tells you the primary metric must reflect long-term economic outcomes.
Option C is correct because it anchors the hypothesis to ninety-day contribution margin per assigned user — a true measure of customer value that captures both revenue and cost, over a window long enough to detect whether early engagement translates into genuine retention. Critically, it includes all assigned users (not just completers), which prevents survivorship bias from inflating results. Seven-day activation is kept as a secondary leading indicator, giving you an early signal without letting it drive the conclusion.
Option A fails because day-one screen views measure superficial activity, and restricting measurement to step-completers introduces selection bias — you're no longer measuring the treatment's true effect on the population.
Option B makes a causal overreach: seven-day activation is a leading indicator, not proof that long-term value has increased. The passage explicitly warns that simplified flows can attract low-intent users who don't persist, so short-term activation alone is insufficient evidence.
Option D compounds two errors: session duration during week one is a weak proxy for value, and making ninety-day retention conditional on activation declining means you'd only measure what matters when things go wrong — you'd miss the full picture by design.
Your takeaway: when a passage defines the business objective as long-term value, immediately eliminate any answer whose primary metric is short-term engagement. Match metric horizon to goal horizon.
- Which hypothesis and success-metric design best addresses the likely interference?
Answer: Randomize restaurant markets and test whether repeat-purchase rate among preidentified eligible customers increases, analyzing outcomes with market-level assignment.
When an experiment risks interference (also called spillover or SUTVA violation), individual-level randomization breaks down because treated units can influence control units. In a referral program, a referred friend who visits a different location "infects" the control group with the treatment effect, making your estimates unreliable. The classic fix is cluster randomization — assigning entire geographic units (markets, cities, stores) so that treated and control customers are unlikely to interact.
Option B gets this right on both dimensions. By randomizing at the market level, it prevents cross-contamination between referred friends across groups. It then measures repeat-purchase rate among preidentified eligible customers — a clean behavioral outcome — and correctly analyzes results at the market level (matching the unit of assignment). This is proper cluster-randomized trial design.
Option A fails because transaction-level randomization is even more granular than customer-level, making spillover nearly inevitable. "Reward-coded transactions" also measures revenue per visit, not the causal effect of the referral program on loyalty.
Option C randomizes individual customers, which still allows referred friends to contaminate the control pool. Worse, it compares successful referrers to nonreferrers — that's a self-selection comparison, not a randomized contrast. Customers motivated enough to refer are simply different people.
Option D uses market-level randomization (good), but restricts analysis to customers who enroll after assignment. Post-assignment enrollment is endogenous — it's an outcome of the treatment, not a baseline characteristic — introducing selection bias.
Study tip: When you see interference risk, immediately look for cluster-level randomization and verify that the analysis unit matches the assignment unit. Mismatches between these two are a common trap.
- Which success-metric plan best limits ambiguity and outcome shopping?
Answer: Predefine paid conversion per assigned trial as primary, activation and invitations as secondary, and support tickets per assigned trial as a guardrail.
When designing experiments, the biggest threats to valid conclusions are ambiguity (unclear success criteria) and outcome shopping (cherry-picking whichever metric looks best after seeing the data). The antidote is a pre-registered metric hierarchy established before the experiment runs.
Option D is the right approach because it does exactly this. Paid conversion per assigned trial is declared the primary metric upfront — it directly reflects the business goal leadership cares about. Activation and invitations are designated secondary metrics, explaining how the tour drives conversion. Support tickets serve as a guardrail, flagging whether gains come at the cost of excessive support burden. This structure leaves no room for post-hoc rationalization: success or failure is defined before a single data point is collected.
Option A sounds balanced, but equally weighting all four metrics creates ambiguity about what "better" actually means. A tour that tanks paid conversion but spikes invitations could still score well on the composite index — a misleading outcome for leadership's actual decision.
Option B is a classic multiple comparisons trap. Testing three metrics and calling the experiment a success if any one is significant inflates your false-positive rate dramatically. You're essentially shopping for a win.
Option C is the most explicit form of outcome shopping — you're literally choosing your primary metric after seeing the p-values, which invalidates the statistical test entirely. Framing support tickets as "contextual" doesn't rescue the approach.
The exam pattern to remember: whenever a question asks about experimental rigor, look for the answer that predefines a single primary metric with a clear business rationale. That's D here, and it will almost always be the right structure in business analytics.
- Which preregistered hypothesis and success criteria best align with the business objective?
Answer: One-click checkout will increase completed orders per assigned checkout session by at least 3%, while increasing the seven-day cancellation rate by no more than 0.5 percentage points.
When evaluating A/B test hypotheses in business analytics, ask yourself three questions: Does the metric directly measure the business goal? Is the control for harm realistic and actionable? And does the design avoid selection bias?
The business objective here is twofold — increase completed purchases while controlling for accidental-order cancellations. Answer A nails both. It measures completed orders per assigned checkout session (capturing all users randomized into the treatment, not just those who clicked), sets a concrete lift threshold of 3%, and bounds the acceptable harm with a +0.5 percentage-point cancellation ceiling. This is exactly what a well-formed, preregistered hypothesis looks like: a primary metric tied to revenue and a guardrail metric tied to the specific risk management identified.
Answer B is tempting because button clicks sound like checkout activity, but clicks are a proxy metric — they don't confirm a purchase was completed. Basket value is also irrelevant to the stated concern about cancellations. Answer C compounds this error by focusing on basket value among completed orders, which excludes abandoned sessions and says nothing about cancellations — the very risk management flagged. Answer D introduces survivorship bias: measuring only customers "who use the new button" excludes users who were assigned the treatment but didn't engage, making the lift estimate artificially optimistic and invalidating the randomized comparison.
Study tip: On experiment-design questions, always check whether the hypothesis metric matches the business outcome (not a proxy) and whether the sample is defined by assignment, not by behavior after assignment — the latter almost always introduces selection bias. - Which hypothesis and success metrics best support a deployment decision?
Answer: The new policy will increase contribution per assigned store after holding and markdown costs, while limiting any decline in product availability.
When evaluating a prescriptive analytics pilot, you need a hypothesis that captures the net business outcome — not just one operational metric in isolation. The passage explicitly flags a trade-off: potential sales gains versus higher holding costs and markdown losses. A good deployment hypothesis must reflect that full profit picture, not just one side of it.
Choice D does exactly this. It defines success as increased contribution margin per store — meaning gross sales minus holding and markdown costs — while also guarding against unacceptable drops in product availability. This aligns the hypothesis with real business value and acknowledges the specific risks the passage raised. It's the only option that measures whether the policy is worth deploying, not just whether it moves one dial.
Choice A focuses on reducing forecast error, which is a modeling quality metric, not a business outcome. A policy could have lower forecast error and still hurt profitability — the retailer cares about what happens at the register and the bottom line, not the RMSE.
Choice B is tempting because it mentions gross sales and caps inventory growth, but it ignores markdown costs entirely. A policy that holds inventory flat but triggers heavy end-of-season markdowns could still destroy margin — this hypothesis would miss that failure mode.
Choice C inverts the priority: it optimizes for lower inventory and treats stockouts as merely explanatory. Reducing inventory is only good if it doesn't erode sales and contribution — making it the primary metric misaligns the goal.
When you see pilot evaluation questions, look for the answer that measures net value with guardrails, not a single input metric. That structure — primary profit outcome plus a constraint — is the hallmark of a deployment-ready hypothesis.
