- A company runs two simultaneous but independent A/B tests. Test 1, a change to the homepage headline, shows a 10% lift in sign-ups. Test 2, a change to the 'Sign Up' button color, shows a 5% lift. Both are statistically significant. The company launches both changes. Three months later, the overall sign-up rate has increased by only 8%, less than the expected 15.5% (1.10 * 1.05 - 1). What is the most likely statistical explanation for this discrepancy?
Answer: A negative interaction effect occurred, where the combined impact of the two changes is less than the sum of their individual impacts.
When two changes are tested independently, their effects are measured in isolation. When deployed together, they can interact. A negative interaction effect means the two changes interfere with each other, or are redundant, making their combined effect smaller than what would be predicted by simply adding or multiplying their individual lifts. For example, a great headline might convince a user to sign up, making the button color less relevant. While novelty effect (A) or regression to the mean (D) could cause a general drop, the interaction effect (B) specifically explains why the combination underperforms relative to the sum of the parts. Type I error (C) is a possibility for any test, but interaction effects are a common reason for over-optimistic forecasts based on independent A/B tests.
- An analyst is planning an A/B test to detect a 2% relative uplift in conversion rate from a baseline of 10%. She calculates the required sample size using a power of 80% and a significance level of 5%. The marketing team then states that, due to implementation costs, only a 5% or greater relative uplift would be considered a business success. How should the analyst adjust her experimental plan based on this new information?
Answer: Recalculate the required sample size using the larger minimum detectable effect (MDE), which will result in a smaller sample size requirement.
The minimum detectable effect (MDE) is the smallest effect size that is of business interest. A key relationship in sample size calculation is that, all else being equal, a larger MDE requires a smaller sample size. It is easier to statistically detect a large effect than a small one. The marketing team's feedback increased the MDE from 2% to 5%. Therefore, the analyst should recalculate the sample size, which will now be smaller than the original calculation. Distractor A is incorrect; a larger effect requires a smaller, not larger, sample. Distractor B is poor practice as it increases the Type I error rate. Distractor D describes a consequence of keeping the sample size fixed with a larger MDE, but the correct action is to recalculate the sample size based on the new MDE.
- An e-commerce platform is running an A/B test on product recommendation algorithms. They plan to run the test until they achieve statistical significance, checking p-values daily. The current sample size is 50,000 users per group with a conversion rate difference of 2.1% (p = 0.08). Marketing pressure exists to either declare victory or expand the test scope to include additional metrics. What is the most appropriate guardrail strategy?
Answer: Implement a pre-planned sequential testing procedure with defined stopping rules to control for multiple testing while allowing early stopping for significant results
Option B correctly identifies that sequential testing with pre-planned stopping rules is needed to control Type I error when conducting multiple interim analyses. Option A describes 'peeking' which inflates Type I error rates and violates statistical assumptions. Option C introduces multiple testing problems and moves away from the primary metric without proper statistical planning. Option D ignores the lack of statistical evidence and could lead to implementing an ineffective change.
- An A/B test is conducted, and the p-value for the difference in the primary metric is 0.25. The team concludes there is no difference between the variant and the control. However, a post-hoc power analysis reveals the test only had 30% power to detect the pre-specified minimum detectable effect (MDE). What is the most accurate interpretation of the test results?
Answer: The test was inconclusive; there was a 70% chance of failing to detect a real effect of the MDE size (a Type II error), so a true effect may well exist.
Statistical power is the probability of detecting an effect of a certain size, if one truly exists. A power of 30% means that even if the variant had a real effect equal to the MDE, there was only a 30% chance of getting a statistically significant result. Conversely, there was a 70% chance of a Type II error (failing to reject the null when it is false). A high p-value combined with low power means the test was underpowered and therefore inconclusive. It does not prove the null hypothesis is true (A). While re-running with more power (likely from a larger sample) is a good idea, the interpretation of the current result is that it's inconclusive. Changing alpha (C) is not the standard way to fix power issues. There is no evidence to suggest a negative effect (D).
- A fintech company runs an A/B test on loan approval criteria, with the treatment group using a new machine learning model. After 6 weeks, the new model shows 22% higher approval rates and 8% higher default rates. The product team wants to implement the new model, arguing that the increased volume will offset the higher defaults. However, the risk management team notices that 73% of the additional approvals went to applicants from historically underrepresented demographic groups. What should be the primary focus of the post-test analysis?
Answer: Perform disparate impact analysis to ensure the new model doesn't create unfair lending practices while also evaluating whether the higher default rates are concentrated in specific demographic groups
When you encounter A/B testing scenarios in business statistics that involve demographic disparities and regulatory concerns, you need to prioritize compliance and fairness analysis over pure financial metrics. This is especially critical in regulated industries like financial services, where lending practices are subject to fair lending laws.
The correct answer is A because this situation presents clear red flags for potential discriminatory lending practices. The fact that 73% of additional approvals went to historically underrepresented groups, combined with 8% higher default rates, requires immediate disparate impact analysis. You need to determine whether the model is making fair decisions or inadvertently creating biased outcomes that could violate fair lending regulations. Additionally, analyzing whether default rate increases are concentrated in specific demographics helps identify if the model is making systematically poor decisions for certain groups.
Option B is wrong because focusing solely on profitability ignores serious regulatory and ethical concerns that could result in legal penalties far exceeding any revenue gains. Option C misses the point—this isn't about model accuracy but about fairness and compliance. The demographic patterns suggest the model may be working as designed but creating biased outcomes. Option D is incorrect because waiting longer doesn't address the immediate compliance concerns, and seasonal variations won't resolve fundamental fairness issues.
Remember: In regulated industries, compliance and fairness analysis must come before financial optimization. When you see demographic disparities in A/B tests involving protected characteristics, always prioritize bias detection and regulatory compliance in your analysis framework.
- A company tests a new, simplified sign-up form. The primary metric, sign-up completion rate, shows a significant 10% improvement overall. However, a segment analysis reveals that for mobile users the completion rate improved by 15%, while for desktop users it decreased by 5% (a statistically significant drop). The company has a roughly 50/50 split between mobile and desktop users. What is the most critical conclusion from this result?
Answer: The overall positive result is misleading because it masks a significant negative impact on a large user segment that must be investigated.
A key principle of A/B testing is to 'do no harm.' While the overall result is positive, this is an aggregate figure that hides a significant problem. Launching the feature would actively harm the experience for half the user base (desktop users). The correct interpretation is that the overall 'win' is misleading. The critical next step is to understand why the experience degraded for desktop users before making any launch decision. Distractor A is a classic error of relying only on aggregate metrics. Distractor B is a potential long-term solution, but it creates technical debt and a fragmented user experience; the immediate step is investigation, not a partial launch. Distractor D correctly identifies the paradoxical nature of the result but incorrectly calls the experiment invalid; the data is valid and provides a crucial insight that requires further analysis.
- A SaaS company conducts an A/B test on email subject lines with 10,000 users per group. The test shows a 12% increase in open rates (from 25% to 28%, p = 0.03). However, the company notices that 15% of users in the treatment group never received emails due to a server configuration error, while only 2% of control group users experienced delivery failures. How should this impact the interpretation and next steps?
Answer: The delivery failure represents a confounding variable that invalidates the test, requiring a complete restart with proper infrastructure monitoring
The differential delivery failure rates (15% vs 2%) create a systematic bias that confounds the treatment effect with infrastructure issues. This isn't measuring the true effect of subject line changes but rather a combination of subject line and delivery problems. Option A incorrectly applies intention-to-treat when the randomization integrity was compromised. Option B suggests per-protocol analysis but doesn't address that the remaining samples may not be representative. Option D proposes a statistical fix for what is fundamentally a design and implementation problem.
- A marketing team tests five new email subject lines (B, C, D, E, F) against the current one (A) in a single experiment. They want to maintain a family-wise error rate (FWER) of at most 5%. If they use the Bonferroni correction to adjust for multiple comparisons, what significance level (α) should they use for each of the five individual hypothesis tests (A vs. B, A vs. C, etc.)?
Answer: 0.01
The Bonferroni correction is a method to control the FWER when performing multiple hypothesis tests. It works by dividing the desired overall alpha level by the number of comparisons being made. In this scenario, there are five new subject lines being compared to the control, resulting in 5 comparisons. To maintain a FWER of 0.05, the significance level for each individual test should be α′=kα=50.05=0.01. Distractor A fails to make any correction. Distractor C incorrectly divides by the total number of groups (6) instead of the number of comparisons (5). Distractor D confuses the logic of two-tailed tests with the Bonferroni correction method. - A company is conducting an A/B test on their checkout page to measure conversion rate improvements. The test shows a statistically significant 15% increase in conversion rate (p < 0.05) for the treatment group. However, when they segment the results by device type, they discover that mobile users show a 25% decrease in conversion rate while desktop users show a 45% increase. What statistical phenomenon does this illustrate, and what should be the primary concern for implementing the treatment?
Answer: This demonstrates interaction effects, and the primary concern should be developing separate treatments for mobile and desktop users rather than implementing a universal solution
This scenario demonstrates interaction effects between the treatment and device type, where the treatment works differently for different subgroups. The primary concern should be developing device-specific solutions rather than implementing a one-size-fits-all treatment that harms mobile users. Option A incorrectly identifies this as Simpson's paradox and focuses on traffic volume rather than user experience. Option C misidentifies the issue as selection bias when proper randomization was likely conducted. Option D incorrectly suggests this is a multiple testing issue requiring statistical correction.
- An online marketplace tests two different search ranking algorithms (A and B) by randomly assigning users to each algorithm for 30 days. Algorithm A shows 6% higher click-through rates but Algorithm B shows 15% higher purchase conversion rates. Both differences are statistically significant. The business team argues for Algorithm A because clicks are easier to measure and show immediate impact. What is the most critical consideration for the final decision?
Answer: Analyze the relationship between click-through and conversion rates to determine if Algorithm A's clicks represent higher-quality traffic or just increased browsing behavior
The key issue is understanding whether Algorithm A's higher clicks represent meaningful user intent or just increased browsing without purchase intent. Algorithm A might be showing users products they click on but don't want to buy, while Algorithm B might show fewer but more relevant results. Option A assumes clicks will translate to conversions without evidence. Option B focuses only on the conversion metric without understanding the user journey. Option D proposes an implementation that wasn't tested and could create inconsistent user experiences.
- A streaming service tests a new recommendation algorithm (Variant B). The primary metric, 'average daily watch time per user,' increases by 5% with statistical significance (p=0.01). However, a critical guardrail metric, 'rate of user-initiated content buffering events,' also increases significantly by 15% (p=0.005). What is the most prudent course of action for the company?
Answer: Do not launch Variant B, as the significant degradation in a key user experience metric likely outweighs the benefit and could lead to long-term churn.
Guardrail metrics are designed to catch unintended negative consequences. A significant negative change in a critical user experience metric like buffering can lead to user frustration and attrition over the long term, even if a primary engagement metric improves in the short term. The most prudent decision is to prioritize the long-term health of the user base and not launch a change that significantly worsens their experience. Distractor A exhibits 'metric fixation' by ignoring the negative guardrail. Distractor C is unnecessary as the negative effect is already statistically significant. Distractor D is risky; the quantitative data already provides a strong warning signal that should be heeded before exposing more users to a poor experience.
- A B2B software company starts an A/B test on its pricing page on a Thursday morning. By Friday afternoon, the new page (Variant B) shows a 30% lift in demo requests with a p-value of 0.01. A product manager argues to stop the test and ship the change immediately. What is the most significant flaw in this reasoning?
Answer: The test has not run for a full weekly business cycle, and user behavior and intent may differ significantly between weekdays and weekends.
A fundamental rule for experiment duration is to run the test long enough to capture the natural cyclicality of the business. For a B2B company, user profiles and purchasing intent can vary dramatically between weekdays (when employees are working) and weekends. Making a decision based on data from only two weekdays is premature because it is not representative of the behavior of all users across a full week. While small sample size (A) and novelty effect (C) are also valid concerns, failing to account for the business cycle is the most critical and definite flaw in this scenario.
- A streaming platform is testing a new content recommendation algorithm. They randomly assign 100,000 users to treatment and control groups. After two weeks, they find that treatment group users watch 8% more content (statistically significant), but they also notice that 12% of treatment users and 8% of control users have canceled their subscriptions during the test period. What does this pattern suggest about the algorithm's impact, and what analysis approach would be most informative?
Answer: The algorithm may be recommending more addictive but lower-quality content, leading to short-term engagement gains but long-term satisfaction decline requiring cohort retention analysis
The combination of increased short-term engagement but higher cancellation rates suggests the algorithm may be optimizing for immediate engagement at the expense of long-term satisfaction. Cohort retention analysis would reveal if this pattern continues over time. Option A incorrectly dismisses the cancellation pattern as unrelated when it's likely connected to the algorithm change. Option C minimizes a 50% relative increase in cancellation rates (from 8% to 12%). Option D suggests bimodal responses but doesn't address the concerning trend of higher cancellations despite increased engagement.
- Before launching a series of A/B tests, a data science team runs an A/A test where both user groups see the exact same version of the website. After collecting data from 50,000 users, they find that the conversion rate for Group A is 5.2% and for Group B is 5.8%, with a p-value of 0.03. What is the most important implication of this A/A test result?
Answer: The experimentation platform has a systematic bias, possibly in the randomization algorithm or data collection, that must be investigated and fixed.
The purpose of an A/A test is to validate the testing system. The null hypothesis is that there is no difference between the groups. In a well-calibrated system, we expect the p-value to be greater than 0.05 about 95% of the time. Finding a statistically significant difference (p=0.03) is a strong indication that something is wrong with the testing infrastructure, such as biased user assignment or faulty data logging. This is a critical failure that must be fixed before any real A/B tests can be trusted. Distractor A misattributes a systematic issue to random noise. Distractor B misunderstands the purpose of an A/A test; a significant result is a failure, not a success. Distractor D suggests a workaround that doesn't address the root cause of the problem.
- A ride-sharing company tests a new driver incentive in a city by randomizing drivers into control and variant groups. The metric is 'average trips per driver.' The test shows a significant lift for the variant group. However, the incentive might cause variant drivers to cluster in high-demand areas, taking rides that control-group drivers might have otherwise gotten. This phenomenon, where the treatment applied to one unit affects the outcome for another, is a violation of the Stable Unit Treatment Value Assumption (SUTVA). What is the most likely consequence of this violation on the test's results?
Answer: The measured effect size is likely an overestimate of the true lift, because the variant group's gain came at the control group's expense.
SUTVA assumes that a unit's outcome is only affected by its own treatment status, not the treatment status of others. Here, that assumption is violated by interference (or network effects). Variant drivers' success (from the incentive) directly reduces the opportunities available to control drivers. This artificially inflates the performance of the variant group while artificially depressing the performance of the control group, making the measured difference between them larger than the true effect of the incentive. Therefore, the effect size is likely overestimated. While changing the randomization unit (B) is a way to solve this, the question asks for the consequence of the violation. While independence assumptions (C) are related, the most direct business consequence is the mis-estimation of the effect size (A).
- An e-commerce company tests a new checkout button design (Variant B) against the current design (Variant A). They monitor 10 key metrics. The primary metric, overall conversion rate, shows no statistically significant change. However, a secondary metric, 'average session duration,' shows a statistically significant increase with a p-value of 0.04. What is the most appropriate conclusion for the product manager to draw?
Answer: The significant result for session duration is likely a false positive due to the multiple testing problem, and it should be treated with skepticism unless confirmed by a follow-up experiment.
When multiple metrics are tested simultaneously, the probability of observing at least one significant result by random chance (a Type I error) increases. With 10 metrics and α = 0.05, the probability of at least one false positive is approximately 1 - (0.95)10 ≈ 40%. Therefore, a single significant result on a secondary metric, especially when the primary metric shows no effect, is suspect. The most appropriate conclusion is that this might be a statistical anomaly due to multiple comparisons, not a true effect. Distractor A ignores the primary metric and the multiple testing problem. Distractor B is too general and doesn't identify the specific statistical issue. Distractor D suggests changing the significance level after seeing the results, which is improper statistical practice; adjustments for multiple testing like Bonferroni correction should be planned in advance. - A food delivery platform implements an A/B test where treatment users receive personalized restaurant recommendations based on machine learning, while control users see the standard list sorted by delivery time. After 5 weeks with 80,000 users per group, treatment users place 11% more orders (p = 0.01) and have 7% higher average order values (p = 0.04). However, restaurant partners in the treatment group report 23% fewer orders on average compared to control group restaurants. What does this result pattern most likely indicate about the recommendation algorithm?
Answer: The algorithm creates a concentration effect, heavily favoring popular restaurants and potentially creating long-term marketplace imbalances that could reduce restaurant diversity
When analyzing A/B test results for recommendation systems, you need to examine outcomes across all stakeholders—users, the platform, and supply-side partners. Conflicting signals often reveal deeper algorithmic behaviors that aren't immediately apparent from user metrics alone.
The data shows users are happier (11% more orders, 7% higher order values), but restaurants report 23% fewer orders on average. This pattern indicates the algorithm is concentrating demand among fewer restaurants rather than distributing it evenly. Popular restaurants likely receive disproportionate recommendations, while less popular ones see significant order drops—a classic concentration effect.
Answer A incorrectly assumes this is temporary. A 23% decrease across restaurant partners after 5 weeks suggests a systematic issue, not an adjustment period. Answer B misinterprets the restaurant feedback as selection bias from unpopular establishments, but the magnitude (23% average decrease) indicates widespread impact beyond just struggling restaurants. Answer C focuses on data quality issues around availability or capacity, but the user satisfaction metrics contradict this—if restaurants couldn't fulfill orders due to capacity problems, users wouldn't be ordering more with higher values.
Answer D correctly identifies the core issue: the algorithm optimizes for user preferences by heavily recommending popular restaurants, creating marketplace concentration that reduces opportunities for diverse restaurant partners.
Study tip: In marketplace A/B tests, always analyze multi-sided effects. Strong user metrics paired with negative supplier metrics often signal concentration effects that can threaten long-term marketplace health, even when short-term user satisfaction improves.
- An e-commerce company tests a new feature on its product pages that recommends similar, but less expensive, alternative products. The primary metric, 'conversion rate on the main product,' decreases significantly. However, a key secondary metric, 'overall revenue per visitor,' shows a statistically significant increase. What is the best interpretation of this outcome?
Answer: The feature successfully increased overall business value by guiding users to alternatives they purchased, despite cannibalizing sales of the initial product.
This scenario highlights the importance of choosing the right Overall Evaluation Criterion (OEC). While the feature 'failed' on the narrow primary metric (conversion of the main product), it 'succeeded' on the broader, more important business metric (overall revenue per visitor). The feature caused cannibalization—it moved sales from one product to another—but the net effect was positive. The correct interpretation is that this cannibalization was beneficial, leading to an overall lift. Distractor A focuses on the wrong metric. Distractor B fails to see that the results are not contradictory but rather tell a complete story about a business trade-off. Distractor D proposes a future action but is not the best interpretation of the current results, which show a net positive effect overall.
- An A/B test on advertising copy results in a 95% confidence interval for the difference in conversion rates (Variant B - Variant A) of [+0.5%, +3.5%]. Which of the following statements is an incorrect interpretation of this result?
Answer: We are 95% confident that the true difference in conversion rates between Variant B and Variant A is between 0.5% and 3.5%.
This is a subtle but critical point in frequentist statistics. The confidence level (95%) refers to the reliability of the procedure used to create the interval, not the probability that a specific interval contains the true parameter. After an interval is calculated (e.g., [0.5%, 3.5%]), the true parameter is either in it or it is not. The probability is either 0 or 1. Therefore, stating there is a 95% probability the true value lies within this specific interval is incorrect. Choice C provides the correct frequentist interpretation. Choice B is correct; if a 95% CI for a difference does not include 0, the result is significant at the 5% level. Choice D is a correct and practical interpretation of the range of plausible values for the true effect.
- A data analyst is running an A/B test on a new feature, with a pre-calculated required sample size of 20,000 users per variant. After only 5,000 users have been assigned, the analyst checks the results and finds a p-value of 0.04 for the primary metric. They decide to stop the test and declare a winner. This practice of 'peeking' is statistically problematic primarily because it:
Answer: inflates the Type I error rate, increasing the probability of concluding there is an effect when one does not actually exist.
Repeatedly checking results and stopping a test as soon as the p-value crosses the significance threshold (e.g., 0.05) dramatically increases the Type I error rate. Each check is an opportunity for random fluctuations to yield a 'significant' result by chance. By stopping early, the analyst capitalizes on this randomness. The correct procedure is to commit to a sample size in advance and only analyze the results once that sample size is reached. Distractor A is incorrect; peeking doesn't reduce power. Distractor B describes a potential separate issue (sample representativeness), but the core statistical sin of peeking is the inflation of the false positive rate. Distractor C is incorrect; the data points themselves are likely still independent, but the stopping rule is what is flawed.
