Business Analytics Quiz: Hypotheses And Success Metrics
10 questions · exam conditions
0:00
Hypotheses And Success MetricsQuestion 1 of 10

An online grocer is testing a one-click checkout against its standard multistep checkout. Management wants more completed purchases but is concerned that accidental orders may increase cancellations.

Which preregistered hypothesis and success criteria best align with the business objective?

One-click checkout will increase completed orders per assigned checkout session by at least 3%3\%, while increasing the seven-day cancellation rate by no more than 0.50.5 percentage points.
One-click checkout will increase clicks on the purchase button by at least 3%3\%, while keeping average basket value statistically unchanged during the experiment.
One-click checkout will increase average basket value among completed orders by at least 3%3\%, while keeping the number of checkout page views unchanged.
One-click checkout will increase completed orders among customers who use the new button by at least 3%3\%, while keeping their cancellation rate below the historical average.
← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Hypotheses And Success Metrics

Practice Hypotheses And Success Metrics in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Hypotheses And Success Metrics, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

An online grocer is testing a one-click checkout against its standard multistep checkout. Management wants more completed purchases but is concerned that accidental orders may increase cancellations.

Which preregistered hypothesis and success criteria best align with the business objective?

  1. One-click checkout will increase completed orders per assigned checkout session by at least 3%3\%, while increasing the seven-day cancellation rate by no more than 0.50.5 percentage points. (correct answer)
  2. One-click checkout will increase clicks on the purchase button by at least 3%3\%, while keeping average basket value statistically unchanged during the experiment.
  3. One-click checkout will increase average basket value among completed orders by at least 3%3\%, while keeping the number of checkout page views unchanged.
  4. One-click checkout will increase completed orders among customers who use the new button by at least 3%3\%, while keeping their cancellation rate below the historical average.
Explanation: When evaluating A/B test hypotheses in business analytics, ask yourself three questions: Does the metric directly measure the business goal? Is the control for harm realistic and actionable? And does the design avoid selection bias? The business objective here is twofold — increase completed purchases while controlling for accidental-order cancellations. Answer A nails both. It measures completed orders per assigned checkout session (capturing all users randomized into the treatment, not just those who clicked), sets a concrete lift threshold of 3%3\%, and bounds the acceptable harm with a +0.5+0.5 percentage-point cancellation ceiling. This is exactly what a well-formed, preregistered hypothesis looks like: a primary metric tied to revenue and a guardrail metric tied to the specific risk management identified. Answer B is tempting because button clicks sound like checkout activity, but clicks are a proxy metric — they don't confirm a purchase was completed. Basket value is also irrelevant to the stated concern about cancellations. Answer C compounds this error by focusing on basket value among completed orders, which excludes abandoned sessions and says nothing about cancellations — the very risk management flagged. Answer D introduces survivorship bias: measuring only customers "who use the new button" excludes users who were assigned the treatment but didn't engage, making the lift estimate artificially optimistic and invalidating the randomized comparison. Study tip: On experiment-design questions, always check whether the hypothesis metric matches the business outcome (not a proxy) and whether the sample is defined by assignment, not by behavior after assignment — the latter almost always introduces selection bias.

Question 2

A retailer randomly assigns customer accounts to receive either its existing promotional email or a redesigned email. The redesign contains more images and may affect deliverability. Management wants to know whether sending the redesign causes more purchases without increasing unsubscribes.

Which metric specification provides the most defensible test of the hypothesis?

  1. Purchase rate among customers who open the email, paired with unsubscribes as a percentage of customers who click at least one link.
  2. Purchases per assigned account, paired with unsubscribes per assigned account as a guardrail, regardless of delivery or opening. (correct answer)
  3. Purchases per successfully delivered email, paired with unsubscribes per delivered email after excluding invalid and filtered addresses.
  4. Revenue per purchasing account, paired with the total number of unsubscribes observed among recipients of the redesigned email.
Explanation: When evaluating A/B test metrics, your guiding principle should be intention-to-treat (ITT): measure outcomes relative to the group as randomly assigned, not as filtered by downstream behavior. This preserves the integrity of randomization and avoids selection bias. Answer B is correct because both metrics — purchases and unsubscribes — are calculated per assigned account, the true unit of randomization. This captures the full causal effect of sending the redesigned email, including deliverability problems. If the redesign reaches fewer inboxes due to image-heavy filtering, that's part of the treatment effect and should count against it. Using assigned accounts as the denominator for both metrics keeps the comparison clean and statistically defensible. Answer A introduces severe selection bias by conditioning on opens. Customers who open emails aren't random — they're systematically different between groups, especially if deliverability differs. Comparing purchase rates among openers is no longer a causal comparison between treatments. Answer C conditions on successful delivery, which excludes accounts harmed by the redesign's deliverability issues. This artificially inflates the redesign's apparent performance by removing the very cases where it underperformed. Answer D uses two incompatible denominators: revenue per purchasing account (a highly filtered subgroup) paired with raw unsubscribe counts. Mixing these denominators makes the guardrail metric uninterpretable and the primary metric biased toward customers who were already going to buy. Strategy tip: On any A/B testing question, ask yourself, "Is the denominator the randomized unit?" If the denominator filters on post-assignment behavior (opens, clicks, deliveries, purchases), selection bias has been introduced — that answer is almost certainly wrong.

Question 3

A bank is considering an automated support bot that is expected to lower service cost. The bot is acceptable only if it reduces cost by at least 15%15\% and does not lower customer satisfaction by more than 22 points on a 100100-point scale.

Which decision rule most appropriately operationalizes this hypothesis?

  1. Deploy if the cost reduction is statistically different from zero and the bot's average satisfaction score is at least 22 points above zero.
  2. Deploy if the point estimate shows a cost reduction of at least 15%15\% and the satisfaction difference is not statistically significant.
  3. Deploy if evidence supports at least a 15%15\% cost reduction and the lower confidence bound for the bot-minus-control satisfaction difference exceeds −2-2 points. (correct answer)
  4. Deploy if the upper confidence bound for cost reduction exceeds 15%15\% and average bot satisfaction is within 22 points of the historical mean.
Explanation: When translating business constraints into statistical decision rules, you need to match each requirement to the right inferential tool. Here, the bank has two separate thresholds: a minimum benefit (cost reduction ≥ 15%15\%) and a maximum harm (satisfaction loss ≤ 22 points). Each threshold demands a different type of evidence. Answer C correctly handles both. For the cost reduction, requiring that evidence "supports at least 15%15\%" means the confidence interval for cost savings should sit above 15%15\% — you need the lower bound to clear the threshold, not just the point estimate. For satisfaction, the concern is harm, so you need the lower confidence bound of the bot-minus-control difference to exceed −2-2. If the worst plausible outcome is better than −2-2 points, you've ruled out unacceptable harm. This is the logic of non-inferiority testing applied to a business gate. Answer A is flawed because "statistically different from zero" only tells you the effect isn't zero — a 1%1\% cost reduction could be significant but useless. The satisfaction criterion is also nonsensical as written. Answer B relies solely on a point estimate for cost, ignoring uncertainty. A 15%15\% estimate with wide confidence intervals could easily be below the threshold in reality. Using non-significance for satisfaction also conflates "no detectable harm" with "acceptable harm." Answer D checks the upper bound for cost reduction, which is the optimistic end — it can exceed 15%15\% even when the true effect is far lower. That's the wrong tail to examine for a minimum-benefit requirement. The study tip: whenever a question sets a minimum required effect, check the lower confidence bound; for a maximum tolerated harm, also check the lower bound of the difference. The conservative end of the interval is what protects the decision.

Question 4

A telecommunications company has developed a new predictive churn model with a higher cross-validated area under the ROC curve than its current model. The model will create value only if using its scores to target retention offers improves economic outcomes.

Which experimental hypothesis best tests whether the new model should be deployed?

  1. Accounts scored by the new model will have a higher observed churn rate than accounts scored by the current model.
  2. The new model will retain more contacted accounts than the old model when analysis is limited to customers who accept an offer.
  3. The new model will achieve a significantly higher area under the ROC curve on customers included in the deployment experiment.
  4. Using the new model's targeting policy will increase retained contribution margin per eligible account, with offer cost and complaints included as guardrails. (correct answer)
Explanation: When evaluating whether a predictive model should actually be deployed, you need to shift from thinking like a data scientist to thinking like a business decision-maker. A model's statistical performance (like AUC) only matters insofar as it translates into better real-world outcomes. The right experimental hypothesis should test the full business value chain: targeting quality, economic returns, and unintended consequences. Option D is correct because it frames the hypothesis around the metric that actually matters — retained contribution margin per eligible account — while incorporating offer costs and complaint rates as guardrails. This design captures both upside (revenue retained) and downside risks (cost overruns, customer friction), which is exactly what a deployment decision requires. A properly structured business experiment must connect model outputs to financial outcomes, not just predictive statistics. Option A is a trap: a higher observed churn rate among targeted accounts would actually suggest worse targeting, not better. Even if this were reversed, raw churn rates ignore offer costs and don't reflect net economic value. Option B limits analysis to customers who accepted an offer, which introduces severe survivorship bias. You're no longer measuring targeting effectiveness — you're measuring something about self-selected acceptors, which tells you nothing about the model's deployment value. Option C simply re-evaluates AUC on the experiment population. But the passage already tells you the new model has better AUC. Re-confirming a statistical metric doesn't tell you whether the model improves business outcomes, which is the entire question. The broader lesson: whenever a question asks whether a model should be deployed, look for the answer that ties model outputs to economic impact with appropriate constraints — not the one that celebrates better statistics.

Question 5

Before testing a new pricing page, a company defines success as an increase of at least $0.20\$0.20 in contribution margin per assigned visitor. The estimated treatment effect is $0.25\$0.25, with a 95%95\% confidence interval from −$0.05-\$0.05 to $0.55\$0.55.

Which interpretation is most consistent with the predefined success criterion?

  1. The test is successful because the point estimate exceeds the minimum economically meaningful increase of $0.20\$0.20.
  2. The test is unsuccessful because the confidence interval contains zero, proving that the pricing page has no positive effect.
  3. The evidence is insufficient to declare success because the interval includes effects below both zero and the $0.20\$0.20 target. (correct answer)
  4. The test is successful because the upper confidence bound exceeds the target by $0.35\$0.35 per assigned visitor.
Explanation: Whenever you see a question combining a confidence interval with a predefined success threshold, resist the temptation to focus only on the point estimate. You need to ask: does the entire confidence interval support the claim? Here, success requires a contribution margin lift of at least $0.20\$0.20. The point estimate of $0.25\$0.25 looks promising, but the 95%95\% confidence interval runs from −$0.05-\$0.05 to $0.55\$0.55. That interval spans a wide range of plausible true effects — some below zero, many below the $0.20\$0.20 threshold. Because the data are compatible with effects that would not meet the success criterion, you cannot confidently declare success. That's exactly what C captures: the interval includes values below both zero and the $0.20\$0.20 target, making the evidence insufficient. A is the classic trap: cherry-picking the point estimate while ignoring uncertainty. A single number like $0.25\$0.25 is your best guess, not a guaranteed outcome — the interval tells you how much that guess could be off. B makes a logical error in two ways: containing zero doesn't prove no effect (absence of evidence isn't evidence of absence), and the standard for this test isn't simply "above zero" — it's "above $0.20\$0.20." D misuses the upper bound; that $0.55\$0.55 ceiling represents an optimistic extreme, not a reliable outcome to celebrate. Study tip: When a question defines a minimum meaningful effect, treat that threshold exactly like zero in a standard significance test — success requires the entire confidence interval to fall above it, not just the point estimate.

Question 6

A subscription company is testing a simplified onboarding flow. Prior tests showed that simplified flows can increase first-week activity but sometimes attract low-intent usage that does not persist. The business objective is to improve customer value rather than short-term engagement alone.

Which primary hypothesis and metric are most appropriate?

  1. The new flow will increase day-one screen views per active user, measured only among users who complete every onboarding step.
  2. The new flow will increase seven-day activation, which is sufficient evidence that long-term customer value has causally increased.
  3. The new flow will increase ninety-day contribution margin per assigned user, with seven-day activation treated as a secondary leading indicator. (correct answer)
  4. The new flow will increase average session duration during the first week, with ninety-day retention reported only if activation declines.
Explanation: When designing an experiment around a business objective, you need to align your primary hypothesis and metric directly with what the business actually cares about — not a convenient proxy. Here, the stated goal is customer value, not short-term engagement, which immediately tells you the primary metric must reflect long-term economic outcomes. Option C is correct because it anchors the hypothesis to ninety-day contribution margin per assigned user — a true measure of customer value that captures both revenue and cost, over a window long enough to detect whether early engagement translates into genuine retention. Critically, it includes all assigned users (not just completers), which prevents survivorship bias from inflating results. Seven-day activation is kept as a secondary leading indicator, giving you an early signal without letting it drive the conclusion. Option A fails because day-one screen views measure superficial activity, and restricting measurement to step-completers introduces selection bias — you're no longer measuring the treatment's true effect on the population. Option B makes a causal overreach: seven-day activation is a leading indicator, not proof that long-term value has increased. The passage explicitly warns that simplified flows can attract low-intent users who don't persist, so short-term activation alone is insufficient evidence. Option D compounds two errors: session duration during week one is a weak proxy for value, and making ninety-day retention conditional on activation declining means you'd only measure what matters when things go wrong — you'd miss the full picture by design. Your takeaway: when a passage defines the business objective as long-term value, immediately eliminate any answer whose primary metric is short-term engagement. Match metric horizon to goal horizon.

Question 7

A restaurant chain is testing a referral reward. Customers can refer friends who visit different locations, so individually assigned customers may expose control customers to the offer. Management wants to estimate the reward's effect on repeat purchasing.

Which hypothesis and success-metric design best addresses the likely interference?

  1. Randomize individual transactions and test whether reward-coded transactions have higher average revenue than transactions without a reward code.
  2. Randomize restaurant markets and test whether repeat-purchase rate among preidentified eligible customers increases, analyzing outcomes with market-level assignment. (correct answer)
  3. Randomize individual customers and test whether customers who successfully refer a friend have a higher repeat-purchase rate than nonreferrers.
  4. Randomize restaurant markets and test whether total transactions increase among customers who enroll in the referral program after assignment.
Explanation: When an experiment risks interference (also called spillover or SUTVA violation), individual-level randomization breaks down because treated units can influence control units. In a referral program, a referred friend who visits a different location "infects" the control group with the treatment effect, making your estimates unreliable. The classic fix is cluster randomization — assigning entire geographic units (markets, cities, stores) so that treated and control customers are unlikely to interact. Option B gets this right on both dimensions. By randomizing at the market level, it prevents cross-contamination between referred friends across groups. It then measures repeat-purchase rate among preidentified eligible customers — a clean behavioral outcome — and correctly analyzes results at the market level (matching the unit of assignment). This is proper cluster-randomized trial design. Option A fails because transaction-level randomization is even more granular than customer-level, making spillover nearly inevitable. "Reward-coded transactions" also measures revenue per visit, not the causal effect of the referral program on loyalty. Option C randomizes individual customers, which still allows referred friends to contaminate the control pool. Worse, it compares successful referrers to nonreferrers — that's a self-selection comparison, not a randomized contrast. Customers motivated enough to refer are simply different people. Option D uses market-level randomization (good), but restricts analysis to customers who enroll after assignment. Post-assignment enrollment is endogenous — it's an outcome of the treatment, not a baseline characteristic — introducing selection bias. Study tip: When you see interference risk, immediately look for cluster-level randomization and verify that the analysis unit matches the assignment unit. Mismatches between these two are a common trap.

Question 8

A retailer is piloting a prescriptive inventory policy that uses demand forecasts to recommend store-level replenishment. The policy may increase sales by reducing stockouts, but it may also increase holding costs and end-of-season markdowns.

Which hypothesis and success metrics best support a deployment decision?

  1. The new policy will reduce forecast error by at least 5%5\%, with total sales reported as a secondary descriptive outcome.
  2. The new policy will increase gross sales per store, provided that average on-hand inventory does not increase during the pilot.
  3. The new policy will reduce average inventory units per store, with stockout frequency used only to explain any sales changes.
  4. The new policy will increase contribution per assigned store after holding and markdown costs, while limiting any decline in product availability. (correct answer)
Explanation: When evaluating a prescriptive analytics pilot, you need a hypothesis that captures the net business outcome — not just one operational metric in isolation. The passage explicitly flags a trade-off: potential sales gains versus higher holding costs and markdown losses. A good deployment hypothesis must reflect that full profit picture, not just one side of it. Choice D does exactly this. It defines success as increased contribution margin per store — meaning gross sales minus holding and markdown costs — while also guarding against unacceptable drops in product availability. This aligns the hypothesis with real business value and acknowledges the specific risks the passage raised. It's the only option that measures whether the policy is worth deploying, not just whether it moves one dial. Choice A focuses on reducing forecast error, which is a modeling quality metric, not a business outcome. A policy could have lower forecast error and still hurt profitability — the retailer cares about what happens at the register and the bottom line, not the RMSE. Choice B is tempting because it mentions gross sales and caps inventory growth, but it ignores markdown costs entirely. A policy that holds inventory flat but triggers heavy end-of-season markdowns could still destroy margin — this hypothesis would miss that failure mode. Choice C inverts the priority: it optimizes for lower inventory and treats stockouts as merely explanatory. Reducing inventory is only good if it doesn't erode sales and contribution — making it the primary metric misaligns the goal. When you see pilot evaluation questions, look for the answer that measures net value with guardrails, not a single input metric. That structure — primary profit outcome plus a constraint — is the hallmark of a deployment-ready hypothesis.

Question 9

A business-to-business software company is testing a guided product tour. Analysts can measure feature activation, coworker invitations, support tickets, and conversion from free trial to a paid contract. Leadership will deploy the tour only if it improves paid acquisition without creating excessive support demand.

Which success-metric plan best limits ambiguity and outcome shopping?

  1. Define an equally weighted index of all four measures, treating increases in every component as beneficial to the business.
  2. Declare the experiment successful if any of activation, invitations, or paid conversion has a statistically significant positive result.
  3. Select the primary metric after observing which measure has the smallest p-value, then use support tickets as contextual evidence.
  4. Predefine paid conversion per assigned trial as primary, activation and invitations as secondary, and support tickets per assigned trial as a guardrail. (correct answer)
Explanation: When designing experiments, the biggest threats to valid conclusions are ambiguity (unclear success criteria) and outcome shopping (cherry-picking whichever metric looks best after seeing the data). The antidote is a pre-registered metric hierarchy established before the experiment runs. Option D is the right approach because it does exactly this. Paid conversion per assigned trial is declared the primary metric upfront — it directly reflects the business goal leadership cares about. Activation and invitations are designated secondary metrics, explaining how the tour drives conversion. Support tickets serve as a guardrail, flagging whether gains come at the cost of excessive support burden. This structure leaves no room for post-hoc rationalization: success or failure is defined before a single data point is collected. Option A sounds balanced, but equally weighting all four metrics creates ambiguity about what "better" actually means. A tour that tanks paid conversion but spikes invitations could still score well on the composite index — a misleading outcome for leadership's actual decision. Option B is a classic multiple comparisons trap. Testing three metrics and calling the experiment a success if any one is significant inflates your false-positive rate dramatically. You're essentially shopping for a win. Option C is the most explicit form of outcome shopping — you're literally choosing your primary metric after seeing the p-values, which invalidates the statistical test entirely. Framing support tickets as "contextual" doesn't rescue the approach. The exam pattern to remember: whenever a question asks about experimental rigor, look for the answer that predefines a single primary metric with a clear business rationale. That's D here, and it will almost always be the right structure in business analytics.

Question 10

A retailer tests a personalized offer on 2,0002{,}000 assigned visitors per group. In control, 240240 visitors purchase and average contribution is $30\$30 per purchaser. In treatment, 220220 visitors purchase and average contribution is $34\$34 per purchaser. The return rate rises from 4.0%4.0\% to 4.6%4.6\%. Before the test, success was defined as at least a 3%3\% increase in contribution per assigned visitor, with no more than a 1.01.0-percentage-point increase in return rate.

Under the preregistered criteria, what conclusion should the analyst report?

  1. The test fails because treatment conversion is lower, even though contribution per assigned visitor and the return-rate guardrail both improve.
  2. The test succeeds because contribution per assigned visitor rises by about 3.9%3.9\% and the return-rate increase remains within the guardrail. (correct answer)
  3. The test fails because contribution per purchaser rises by about 13.3%13.3\%, which does not match the required primary-metric threshold.
  4. The test succeeds because total treatment contribution exceeds control contribution, regardless of the prespecified return-rate guardrail.
Explanation: Whenever you see an A/B test question with preregistered success criteria, your job is to evaluate the exact metrics specified beforehand — not whatever looks impressive after the fact. Here, success requires (1) at least a 3% lift in contribution per assigned visitor and (2) no more than a 1.0-percentage-point increase in return rate. Start with the primary metric. Control contribution per assigned visitor: 240×$302000=$3.60\frac{240 \times \$30}{2000} = \$3.60. Treatment: 220×$342000=$3.74\frac{220 \times \$34}{2000} = \$3.74. The lift is 3.74−3.603.60≈3.9%\frac{3.74 - 3.60}{3.60} \approx 3.9\%, which clears the 3% threshold. Next, check the guardrail: the return rate rises from 4.0% to 4.6%, an increase of 0.6 percentage points — within the 1.0-point limit. Both criteria are satisfied, making B the correct conclusion. A is wrong because it fixates on the conversion count dropping (220 vs. 240) without computing the actual prespecified metric. Fewer purchasers paying more can still improve contribution per assigned visitor — and it does here. C confuses the primary metric with contribution per purchaser. Yes, $34−$30$30≈13.3%\frac{\$34 - \$30}{\$30} \approx 13.3\%, but that was never the success criterion. Applying the wrong formula to the right data is a classic trap. D is wrong because it ignores the guardrail entirely. Preregistered criteria exist precisely to prevent cherry-picking favorable metrics after results come in. The key habit: always anchor your analysis to the prespecified metrics, not whichever numbers look best post-hoc. If a guardrail is listed, it is binding — treat it as a hard constraint, not a suggestion.