Marketing Quiz: A B Testing

20 questions · exam conditions

0:00

A B TestingQuestion 1 of 20

A company A/B tests a new feature on its app. The overall results show no significant difference in user retention between the control and the variant (p = 0.35). However, upon segmenting the data, the analyst discovers that for users under 30, the feature led to a 10% increase in retention (p = 0.01). For users 30 and over, it caused a 7% decrease in retention (p = 0.04).

What is the most strategically sound action to take based on this analysis?

Discard the feature entirely, as the overall impact on retention was not statistically significant.

Roll out the feature to all users, because the positive impact on the younger segment outweighs the negative impact on the older one.

Rerun the A/B test with a larger sample to verify if the conflicting segment results are real.

Implement the feature but configure it to be active only for users under 30, while users 30 and over continue to see the control experience.

← Back to quizzes

Marketing Quiz

Marketing Quiz: A B Testing

Practice A B Testing in Marketing with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on A B Testing, giving you a quick way to practice the rules, question types, and explanations that matter most for Marketing.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

What is the most strategically sound action to take based on this analysis?

Discard the feature entirely, as the overall impact on retention was not statistically significant.
Roll out the feature to all users, because the positive impact on the younger segment outweighs the negative impact on the older one.
Rerun the A/B test with a larger sample to verify if the conflicting segment results are real.
Implement the feature but configure it to be active only for users under 30, while users 30 and over continue to see the control experience. (correct answer)

Explanation: This scenario demonstrates the power of segmentation. The overall non-significant result was masking two opposite, significant effects in different user segments (a Simpson's Paradox-like situation). The most logical business decision is to capitalize on the positive effect and avoid the negative one by deploying the feature to the segment it helps and shielding the segment it harms. A ignores the valuable segment insights. B ignores the significant negative impact on a large part of the user base. C is unnecessary as the segment results are already statistically significant.

Question 2

Before launching a critical A/B test to redesign their homepage, an analytics team decides to first run an A/A test for one week.

What is the primary goal of running this A/A test?

To determine the natural conversion rate of the homepage before introducing any changes.
To ensure the A/B testing software and traffic splitting mechanism are working correctly and not introducing bias. (correct answer)
To prime a segment of the user base for the upcoming changes in the A/B test.
To shorten the duration of the subsequent A/B test by pre-calculating population variance.

Explanation: An A/A test involves serving the exact same version (the control) to both groups in the test. In a properly functioning system, there should be no statistically significant difference in the conversion rates between the two groups. The purpose of this is to validate the testing infrastructure itself. If the A/A test shows a significant difference, it indicates a problem with the tool, traffic allocation, or data collection that must be fixed before a real A/B test can be trusted.

Question 3

An analyst is planning an A/B test on a corporate blog that gets about 2,000 visitors per month. The goal is to achieve a 10% relative lift in newsletter sign-ups. The current sign-up rate is 1%. A sample size calculator suggests they need 78,000 visitors per variation to reliably detect this effect at 95% confidence.

What is the most significant practical issue with running this test as specified?

The test would need to run for over 6 years to collect enough data, making the results irrelevant by the time they are available. (correct answer)
A 10% relative lift (from 1% to 1.1%) is too small of a change to be of any business value.
A 95% confidence level is too low for a test involving a primary business goal like newsletter sign-ups.
The baseline conversion rate of 1% is too high, which makes it difficult to show a significant improvement.

Explanation: When evaluating A/B tests, you need to consider whether the test design is practically feasible, not just statistically sound. The key issue here is sample size versus traffic availability. Let's work through the math: The test needs 78,000 visitors per variation, meaning 156,000 total visitors. With only 2,000 visitors per month, this would take

\frac{156,000}{2,000} = 78

months, or 6.5 years to complete. By the time you get results, your website, audience, market conditions, and business priorities will have changed dramatically, making the findings obsolete. Looking at the wrong answers: Option B misunderstands business impact—a 10% relative lift in newsletter sign-ups could be quite valuable depending on customer lifetime value. Option C incorrectly suggests 95% confidence is too low, when it's actually the standard threshold for most marketing tests. Option D gets the relationship backwards—lower baseline rates (not higher ones) typically require larger sample sizes to detect changes, and 1% is actually a reasonable conversion rate for many websites. The core lesson is that statistical requirements must align with practical constraints. Even if a sample size calculator gives you a number, you need to evaluate whether you can realistically collect that much data in a reasonable timeframe. Study tip: Always check whether the required sample size is achievable given your traffic volume and timeline. If a test would take longer than 3-6 months to complete, consider testing larger effect sizes, accepting lower confidence levels, or finding higher-traffic pages to test on.

Question 4

A very high-traffic website runs a test on a button's color. With over 2 million users in the test, Variant B shows a 0.2% higher conversion rate than Control A. The result is statistically significant, with a p-value of 0.005.

What is the most appropriate business interpretation of this outcome?

Since the p-value is extremely low, the change is a major success and should be implemented immediately.
The lift is so small that it must be a statistical error or false positive, and the results should be discarded.
The result is statistically significant, but the practical significance (the 0.2% lift) must be weighed against implementation costs and business impact. (correct answer)
The test should be re-run with a smaller sample size to see if the lift becomes more pronounced.

Explanation: This question tests the distinction between statistical significance and practical significance. A low p-value indicates the effect is likely real (not due to chance), but it doesn't measure the size or importance of the effect. With a very large sample size, even tiny effects can be statistically significant. The correct business practice is to evaluate if the small, but real, lift of 0.2% is meaningful enough to justify the change. A ignores practical significance. B incorrectly dismisses a statistically valid result. D is illogical; reducing sample size would make it harder, not easier, to detect an effect.

Question 5

A marketing team wants to test if a new email subject line increases open rates. They send an email with the original subject line (Control) to one segment of their list on a Tuesday morning. They send the same email with the new subject line (Variant) to a different segment on a Saturday afternoon. The variant shows a significantly higher open rate.

Why are the results of this A/B test most likely invalid?

The content of the email body was not changed between the two versions.
Open rate is a vanity metric that does not correlate with business goals.
The test introduces a confounding variable by sending the emails on different days and at different times. (correct answer)
The two segments may have been different sizes, which invalidates the comparison of open rates.

Explanation: A valid A/B test isolates a single variable. In this case, the team changed both the subject line AND the send time/day. User behavior on a Tuesday morning is systematically different from a Saturday afternoon. This difference in timing is a confounding variable, making it impossible to attribute the change in open rate solely to the new subject line. A is incorrect; keeping the body the same is proper procedure. B is debatable but not the primary methodological flaw. D is incorrect as open rates are percentages, which normalize for different list sizes, although similar sizes are best practice.

Question 6

An e-commerce site is A/B testing two versions of its product page. The system is set up to assign users to either group A or group B based on their user ID, ensuring a given user always sees the same version. However, a bug causes the system to malfunction: logged-out users are randomly assigned to a group at the start of each new session. As a result, a single anonymous user could see the control page on one visit and the variant page on a subsequent visit.

What is the most direct and damaging consequence of this technical bug on the A/B test's validity?

The variant group will receive unfairly high traffic, biasing the results in its favor.
It will cause sample pollution, as users are exposed to both treatments, which violates the assumption of independent groups and will likely dilute any true effect. (correct answer)
It constitutes a valid 'repeated measures' design, which will actually increase the statistical power of the test.
The test results will be unaffected as long as the total number of sessions in each group remains roughly equal.

Explanation: A core assumption of a standard A/B test is that the two groups are independent and a user in one group is not exposed to the treatment of the other group. This bug violates that assumption by allowing the same user to be in both groups over time. This 'cross-contamination' or 'sample pollution' muddies the data. The effect of one version may carry over to the user's experience with the other version, typically making it harder to detect a real difference and biasing the results toward showing no effect.

Question 7

A fintech company tests a simplified loan application form. Their primary metric is the completion rate of the application. They also track the average loan amount requested as a secondary, or 'guardrail,' metric. The test shows the simplified form increases the completion rate by 15% (p=0.01), but the average requested loan amount decreases by 20% (p=0.02).

What is the most critical business implication of these results?

The test is inconclusive because the primary and secondary metrics show conflicting outcomes.
The test is a clear success, as the primary goal of increasing application volume was achieved with high statistical significance.
The simplified form may be attracting a less qualified or lower-intent applicant pool, potentially harming overall profitability despite higher volume. (correct answer)
The drop in average loan amount is likely random noise and should be disregarded in favor of the significant lift in the primary metric.

Explanation: This question requires looking beyond the primary metric to understand the full business impact. While more applications were submitted (primary metric success), they were for significantly smaller amounts (secondary metric failure). This suggests a change in the type of user being converted. The most insightful conclusion is that the simplification, while increasing volume, may be reducing the overall value of conversions, which could lead to lower revenue and profitability. A is wrong; the results are conclusive, showing a trade-off. B exhibits tunnel vision by ignoring the guardrail metric. D is incorrect because the p-value of 0.02 indicates the drop is statistically significant.

Question 8

A product team wants to improve their user registration form. They believe a new headline, a different call-to-action (CTA) button color, and removing two optional fields will increase completions. They create a single variant that incorporates all three changes and run it against the original form.

What is the primary methodological weakness of this testing approach?

It is not possible to attribute a change in the completion rate to any single one of the three modifications. (correct answer)
A/B tests are not suitable for form optimizations; usability studies are required.
The test lacks a third variant (C) to act as an additional control group.
Removing form fields will inevitably lead to lower quality data from registrations.

Explanation: This is a classic example of a confounded or bundled test. By changing the headline, CTA color, and number of fields all at once in a single variant, the team has made it impossible to isolate the effect of each change. If the variant wins, they won't know if it was the headline, the color, the fewer fields, or some combination. This prevents them from learning which specific changes are effective. To test these elements and their interactions, a multivariate test would be more appropriate.

Question 9

A SaaS company runs an A/B test on its homepage headline. After collecting enough data, the results show that the new headline (Variant B) has a 5% higher sign-up rate than the original (Control A). The calculated p-value for this difference is 0.08. The team's standard threshold for statistical significance is an alpha (α) of 0.05.

Based on these results, what is the most statistically sound conclusion?

The test proves there is a 92% probability that Variant B is better than Control A, so it should be implemented.
The test results are not statistically significant, meaning the observed 5% lift is likely due to random chance, and the null hypothesis cannot be rejected. (correct answer)
The test should be allowed to run longer until the p-value drops below 0.05, which will confirm the superiority of Variant B.
Since the p-value is close to 0.05, the new headline shows a promising trend and should be implemented for a segment of users to gather more data.

Explanation: A p-value of 0.08 is greater than the significance level of 0.05. This means we fail to reject the null hypothesis. The correct interpretation is that we do not have sufficient evidence to conclude that the variant's higher conversion rate is due to anything other than random variation. A is a common misinterpretation of p-values. C describes 'peeking,' which invalidates statistical tests. D makes a business decision based on statistically insignificant results, which is poor practice.

Question 10

An analyst launches an A/B test for a new landing page design. After just three hours, the variant is showing a 35% higher conversion rate than the control. The analyst's manager suggests stopping the test and declaring the variant a winner to capitalize on the gains immediately.

Which statistical concept best explains why the manager's suggestion is premature and risky?

The Simpson's Paradox, where trends in subgroups are reversed when the groups are combined.
The Hawthorne effect, where subjects modify their behavior because they are aware of being observed.
Regression to the mean, where extreme early results are likely to become less extreme as more data is collected. (correct answer)
Statistical power, which may be too high in the early stages of a test, leading to false positives.

Explanation: Early results in a test are based on a small sample size and are highly susceptible to random fluctuations. An unusually high or low initial result is likely to 'regress' or move closer to the true average as the sample size grows. Stopping the test early based on an extreme result (an issue known as 'peeking') is a common error that leads to invalid conclusions. Simpson's Paradox (A) is a specific issue with data aggregation. The Hawthorne effect (B) is not relevant to this automated test. Statistical power (D) is the ability to detect a true effect, and it is typically low, not high, with small sample sizes.

Question 11

An A/B test is concluded, and the results for the variant's impact on conversion rate are reported as a 95% confidence interval of [+2.5%, +9.5%].

Which of the following is the most accurate interpretation of this confidence interval?

There is a 95% probability that the true conversion rate lift provided by the variant is between 2.5% and 9.5%.
The variant's conversion rate lift is statistically significant because the entire confidence interval is above zero. (correct answer)
If the experiment were repeated 100 times, 95 of the repetitions would result in a lift between 2.5% and 9.5%.
The most likely lift is the midpoint of the interval (6%), with a 95% chance of being correct.

Explanation: A confidence interval provides a range of plausible values for the true effect. The key to interpreting significance is whether the interval contains the value of no effect (zero). Since this interval [2.5%, 9.5%] is entirely positive and does not cross zero, we can conclude the result is statistically significant at the 95% confidence level. A is a common but subtle misinterpretation of frequentist confidence intervals; it implies probability about the parameter itself. C is also a common misinterpretation; it should state that 95 of the intervals would capture the true (but unknown) lift. D is incorrect as the interval provides a range, not a point estimate with a probability.

Question 12

An online flower delivery service runs an A/B test in the two weeks leading up to Valentine's Day. Variant B, featuring a prominent 'Valentine's Day Specials' banner, shows a 40% lift in sales compared to the control version without the banner. The team is thrilled with the results.

What is the most significant threat to the external validity of this test's conclusion?

The test's timing during a major holiday means the observed preference for a themed banner may not apply to non-holiday periods. (correct answer)
The results are only valid if both the control and variant groups had an equal number of male and female customers.
The lift of 40% is so high that it is likely a measurement error or a result of the novelty effect.
A two-week test duration is not long enough to account for customers' full purchase cycle and consideration phase.

Explanation: When evaluating A/B test results, you need to consider external validity—whether your findings will hold true in different contexts, times, or situations beyond your specific test conditions. The correct answer is A because this test was conducted during Valentine's Day, an exceptional period when customers actively seek holiday-themed content and romantic messaging. A banner promoting "Valentine's Day Specials" naturally resonates during this time, but this preference may completely disappear during regular periods when customers aren't in a holiday mindset. The seasonal context makes it impossible to conclude that themed banners will always outperform plain designs. Option B is wrong because A/B tests don't require equal gender distribution between groups—they need random assignment, which naturally balances demographics across large sample sizes. Gender balance affects who you can generalize to, but doesn't invalidate the test methodology. Option C misunderstands that a 40% lift, while substantial, isn't inherently suspicious during major shopping holidays when promotional messaging typically sees dramatic performance increases. High effect sizes don't automatically indicate measurement errors. Option D confuses the test duration issue. While two weeks might be short for understanding long-term customer behavior, the question asks about external validity (generalizability), not internal validity (measurement accuracy). The duration adequately captured the Valentine's Day period they were testing. Remember: External validity questions focus on generalizability across different contexts. Always ask yourself, "Would these results hold true in a different time, place, or situation?" Seasonal tests are particularly vulnerable to context-specific effects.

Question 13

A marketing manager is planning an A/B test and has several goals for the test's design. The manager wants to be able to confidently detect a very small improvement in conversion rate. They also want to be extremely certain that any detected difference is not due to chance, so they demand a very high level of statistical significance (e.g., 99.9% confidence).

Holding all other factors constant, how will these two requirements—detecting a smaller minimum effect size and requiring a higher confidence level—impact the necessary sample size for the test?

Detecting a smaller effect requires a larger sample, but requiring higher confidence requires a smaller sample.
Both requirements will decrease the necessary sample size, allowing the test to be run more quickly.
Detecting a smaller effect requires a smaller sample, and requiring higher confidence also requires a smaller sample.
Both requirements will substantially increase the necessary sample size. (correct answer)

Explanation: The relationship between sample size and these factors is fundamental to test design (power analysis). To reliably detect a smaller difference (a smaller effect size), you need more data to distinguish the 'signal' from the 'noise'. Similarly, to achieve a higher level of confidence or significance (e.g., reducing the alpha from 0.05 to 0.001), you need more evidence to reject the null hypothesis, which also requires a larger sample size. Therefore, both conditions work in the same direction, substantially increasing the required sample.

Question 14

A digital marketer hypothesizes that adding trust symbols (e.g., security badges, partner logos) to the sign-up page will increase the number of new account registrations. An A/B test is designed where Control A is the existing page and Variant B includes the trust symbols.

What is the correct null hypothesis (H₀) for this experiment?

The addition of trust symbols to the sign-up page will have no effect on the account registration rate. (correct answer)
The addition of trust symbols to the sign-up page will increase the account registration rate.
The account registration rate is positively correlated with the number of trust symbols displayed.
The existing sign-up page (Control A) has a lower registration rate than the page with trust symbols (Variant B).

Explanation: The null hypothesis (H₀) is a statement of no effect or no difference. It is the default assumption that the experiment seeks to disprove. In this case, it posits that the variable being introduced (trust symbols) will not change the outcome metric (registration rate). B and D describe the alternative hypothesis (Hₐ), which is what the marketer hopes to prove. C is a related but different type of hypothesis about correlation, not a statement of no effect for a specific test.

Question 15

A B2B software company's homepage currently has a primary call-to-action (CTA) button that says 'Discover Our Features'. Their business goal is to increase the number of qualified leads, which are defined as users who complete a demo request form. They hypothesize that a more action-oriented CTA will better achieve this goal.

Which of the following text options for a variant CTA button would provide the most direct and effective test of their hypothesis?

Learn More
Request a Demo (correct answer)
See Pricing
How It Works

Explanation: The hypothesis is that a more direct, action-oriented CTA will increase demo requests. 'Request a Demo' is the most direct and unambiguous text that aligns with the specific conversion action the company wants to drive. 'Learn More' and 'How It Works' are similar in intent to the original 'Discover Our Features' and are not more action-oriented toward the goal. 'See Pricing' tests a different user motivation and directs them to a different consideration path, not directly testing the hypothesis about demo requests.

Question 16

An e-commerce company wants to increase the number of users who add products to their shopping cart from the product detail page. The current 'Add to Cart' button is blue. The marketing team prepares to run an A/B test.

Which of the following represents the most effective, testable hypothesis for this A/B test?

A redesigned product page will lead to a better user experience and more engagement.
Changing the 'Add to Cart' button color from blue to orange and making the product images 30% larger will increase the number of items added to the cart.
Changing the 'Add to Cart' button color from blue to a more prominent color, like orange, will increase the button's click-through rate because it will stand out more against the page's white background. (correct answer)
Customers will find a brighter 'Add to Cart' button more appealing, which will improve their perception of our brand.

Explanation: A strong hypothesis should be specific, measurable, and articulate a clear cause-and-effect relationship. Choice C identifies a single, specific change (button color), a precise metric (click-through rate), and a rationale (prominence). A is too vague ('redesigned', 'better experience'). B tests two variables at once (color and image size), making it impossible to know which change caused the effect. D proposes measuring an untestable subjective feeling ('more appealing', 'perception of our brand') as the primary outcome.

Question 17

A news media website wants to increase the time readers spend on their site and encourage them to view more content. They hypothesize that adding a 'Recommended for You' article suggestion widget at the end of each article will improve reader engagement. They decide to A/B test the presence of this widget.

To evaluate the success of this test against its specific goal, which of the following should be the primary success metric?

Total number of unique visitors to the website.
Average session duration. (correct answer)
The click-through rate on advertisements within the articles.
The conversion rate for the website's premium subscription pop-up.

Explanation: The stated goal is to increase the time readers spend on the site and the amount of content they view. Average session duration directly measures this form of engagement. If the widget is successful, users will click to another article instead of leaving, increasing their session duration. Total unique visitors (A) is an acquisition metric, not an engagement metric affected by this change. Ad click-through rate (C) and subscription rate (D) are secondary metrics that might be influenced, but they are not the most direct measure of the test's primary goal.

Question 18

What is the most appropriate business interpretation of this outcome?

Since the p-value is extremely low, the change is a major success and should be implemented immediately.
The lift is so small that it must be a statistical error or false positive, and the results should be discarded.
The result is statistically significant, but the practical significance (the 0.2% lift) must be weighed against implementation costs and business impact. (correct answer)
The test should be re-run with a smaller sample size to see if the lift becomes more pronounced.

Question 19

What is the most strategically sound action to take based on this analysis?

Discard the feature entirely, as the overall impact on retention was not statistically significant.
Roll out the feature to all users, because the positive impact on the younger segment outweighs the negative impact on the older one.
Rerun the A/B test with a larger sample to verify if the conflicting segment results are real.
Implement the feature but configure it to be active only for users under 30, while users 30 and over continue to see the control experience. (correct answer)

Question 20

What is the most significant threat to the external validity of this test's conclusion?

The test's timing during a major holiday means the observed preference for a themed banner may not apply to non-holiday periods. (correct answer)
The results are only valid if both the control and variant groups had an equal number of male and female customers.
The lift of 40% is so high that it is likely a measurement error or a result of the novelty effect.
A two-week test duration is not long enough to account for customers' full purchase cycle and consideration phase.