Business Analytics Quiz: Cis And P Values For Experiments

10 questions · exam conditions

0:00

Cis And P Values For ExperimentsQuestion 1 of 10

A bank randomly assigns eligible customers to receive either its usual credit-card offer or a personalized offer. For the null hypothesis of no difference in application rates, the experiment produces a two-sided p-value of $0.03$ .

Which interpretation of the p-value is most accurate?

There is a

3\%

probability that the personalized offer has no causal effect on applications.

If the offers truly have equal effects, results at least this extreme would occur about

3\%

of the time.

There is a

97\%

probability that repeating the experiment will produce the same application-rate difference.

About

3\%

of customers changed their application decision because they received the personalized offer.

← Back to quizzes

Business Analytics Quiz

Business Analytics Quiz: Cis And P Values For Experiments

Practice Cis And P Values For Experiments in Business Analytics with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Cis And P Values For Experiments, giving you a quick way to practice the rules, question types, and explanations that matter most for Business Analytics.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

Which interpretation of the p-value is most accurate?

There is a $3\%$ probability that the personalized offer has no causal effect on applications.
If the offers truly have equal effects, results at least this extreme would occur about $3\%$ of the time. (correct answer)
There is a $97\%$ probability that repeating the experiment will produce the same application-rate difference.
About $3\%$ of customers changed their application decision because they received the personalized offer.

Explanation: Whenever you encounter a p-value interpretation question, anchor yourself to the formal definition: a p-value measures how often you'd see results at least as extreme as yours assuming the null hypothesis is true. It is not a probability about whether the null is true, nor a prediction about future experiments. B captures this precisely. A p-value of

0.03

means that if the two offers truly had equal effects (null hypothesis), random chance alone would produce an application-rate difference this large or larger only about

3\%

of the time. Because this is below a typical significance threshold (e.g.,

\alpha = 0.05

), you'd reject the null — but the p-value itself is just that conditional frequency, nothing more. A is wrong because it commits the classic "inverse probability" fallacy — treating the p-value as the probability that the null hypothesis is true. That would require Bayesian reasoning with a prior; a p-value doesn't give you that. C is wrong because p-values say nothing about replication probability. Whether a future experiment yields the same result depends on sample size, variability, and effect size — not the original p-value. D is wrong because p-values don't quantify the size or share of a causal effect. That figure (

3\%

of customers) would come from an effect-size estimate, not significance testing. Study tip: On exam questions about p-values, eliminate any answer that assigns a probability to a hypothesis being true or false (that's a Bayesian posterior, not a p-value) — the correct answer almost always frames the p-value as a conditional frequency under the null.

Question 2

A subscription company uses a randomization test to compare retention under two onboarding processes. Of $1{,}000$ simulated assignments generated under the null hypothesis, $18$ produce an absolute difference in retention at least as large as the observed absolute difference. The analyst uses the plus-one correction, calculating the p-value as $(b+1)/(m+1)$ .

What should the analyst report about statistical significance?

The p-value is about $0.009$ ; the result is significant at both stated levels.
The p-value is about $0.018$ ; the result is significant at both $0.05$ and $0.01$ .
The p-value is about $0.982$ ; the result is not significant at either stated level.
The p-value is about $0.019$ ; the result is significant at $0.05$ but not at $0.01$ . (correct answer)

Explanation: When working with randomization (permutation) tests, the p-value estimates the probability of observing a result as extreme as yours under the null hypothesis. The plus-one correction formula

(b+1)/(m+1)

is used to avoid a p-value of exactly zero and to account for the observed statistic itself as one possible outcome. Here,

b = 18

(simulated statistics at least as extreme as observed) and

m = 1{,}000

(total simulations), giving:

p = \frac{18 + 1}{1{,}000 + 1} = \frac{19}{1{,}001} \approx 0.01898 \approx 0.019

This value falls below

\alpha = 0.05

, so the result is significant at that level. However,

0.019 > 0.01

, so it fails to reach significance at

\alpha = 0.01

. That makes D the correct answer. A gets the denominator wrong — dividing

19/2{,}000

or similar, arriving at

0.009

. It misapplies the correction formula, and claiming significance at both levels requires clearing

\alpha = 0.01

, which this result does not. B uses

b/m = 18/1{,}000 = 0.018

— omitting the plus-one correction entirely. While close numerically, the formula explicitly requires adding 1 to both numerator and denominator, and

0.018 < 0.01

is false anyway. C appears to compute

1 - (b/m)

, essentially reading the p-value backwards as a retention probability rather than a tail probability. This reflects a fundamental misunderstanding of what a p-value measures. Study tip: Always apply the full

(b+1)/(m+1)

formula when the plus-one correction is specified, then carefully compare your result against each significance threshold separately — don't assume that significance at

0.05

implies significance at

0.01

Question 3

A retailer tests a scheduling tool using $25$ matched store pairs. Within each pair, one store uses the tool and the other does not. The mean paired difference in weekly profit, defined as tool minus control, is $120$ dollars, and the sample standard deviation of the paired differences is $250$ dollars. Use a critical value of $2.064$ for a $95\%$ confidence interval.

Which is the appropriate confidence interval for the mean treatment effect?

Approximately $[-26,266]$ dollars, multiplying the paired standard error by $\sqrt{2}$
Approximately $[22,218]$ dollars, using a normal critical value with paired data
Approximately $[-396,636]$ dollars, using the standard deviation as the standard error
Approximately $[17,223]$ dollars, using the standard error of the paired differences (correct answer)

Explanation: When you see paired data in a confidence interval problem, the key insight is that pairing already accounts for store-to-store variability — so you treat the differences as a single sample and compute one standard error from them. The correct formula is

\bar{d} \pm t^* \cdot \dfrac{s_d}{\sqrt{n}}

. Plugging in:

SE = \dfrac{250}{\sqrt{25}} = \dfrac{250}{5} = 50

. The margin of error is

2.064 \times 50 = 103.2

, giving you

120 \pm 103.2

, or approximately

[17, 223]

dollars. That's answer D, the correct choice. Answer A multiplies the standard error by

\sqrt{2}

, which would only apply if you were comparing two independent samples and pooling variances — a method that ignores the paired structure entirely and artificially inflates the margin of error. Answer B uses a normal (z) critical value of roughly

1.96

instead of the correct t-value of

2.064

. With only

n = 25

pairs, you must use the t-distribution with

24

degrees of freedom; using z underestimates uncertainty and produces a falsely narrow interval. Answer C skips the division by

\sqrt{n}

altogether, treating the raw standard deviation (

250

) as the standard error. This is a fundamental mistake — standard deviation measures spread across individual observations, while standard error measures precision of the mean, and they differ by a factor of

\sqrt{n} = 5

. Your study tip: always ask whether data is paired or independent before choosing a method. Paired data collapses to a one-sample t-problem on the differences — never apply two-sample formulas to it.

Question 4

In a randomized pricing experiment, the control conversion rate is $20\%$ and the treatment conversion rate is $23\%$ . A $95\%$ confidence interval for treatment minus control is from $0.5$ to $5.5$ percentage points. For reporting purposes, the analyst expresses each difference as a percentage of the observed control rate, treating that rate as the fixed baseline.

Which statement correctly reports the estimated relative lift and the corresponding transformed confidence interval?

The relative lift is $3\%$ , with an interval from $0.5\%$ to $5.5\%$ , because the percentage-point values are reported without further scaling.
The relative lift is $15\%$ , with an interval from $2.5\%$ to $27.5\%$ , because each value is divided by the control rate of $20\%$ . (correct answer)
The relative lift is approximately $13\%$ , with an interval from about $2.2\%$ to $23.9\%$ , because each value is divided by the treatment rate of $23\%$ .
The relative lift is $115\%$ , with an interval from $102.5\%$ to $127.5\%$ , because the treatment rate itself is expressed as a percentage of the control rate.

Explanation: Whenever you see a question about relative lift, remember the core idea: you're rescaling an absolute difference to express it as a proportion of the baseline. The baseline is always the control rate, because that's what existed before any intervention — it's your reference point for "how much did things change?" Here, the absolute difference in conversion rates is

23\% - 20\% = 3

percentage points. To express this as relative lift, divide by the control rate:

\frac{3\%}{20\%} = 0.15 = 15\%

. Apply the same division to each endpoint of the confidence interval:

\frac{0.5\%}{20\%} = 2.5\%

and

\frac{5.5\%}{20\%} = 27.5\%

. That gives you B — a relative lift of

15\%

with an interval from

2.5\%

27.5\%

. Choice A confuses absolute percentage-point differences with relative lift. Reporting

3\%

without dividing by the baseline is just restating the raw gap — no scaling has occurred. Choice C divides by the treatment rate (

23\%

) instead of the control rate, which is the wrong denominator; relative lift is always anchored to what came before the treatment. Choice D expresses the treatment rate itself (

23\%

) as a percentage of the control rate (

20\%

), yielding

115\%

— a ratio of levels, not a ratio of the change, which conflates two different concepts entirely. Study tip: Always anchor relative lift to the control rate. A quick check: if the absolute difference is zero, relative lift should also be zero — only the control-rate denominator guarantees that property.

Question 5

Before launching an email experiment, a marketing team specifies the directional alternative that a new subject line will increase the open rate. The estimated effect is positive, and the resulting two-sided p-value is $0.08$ . The test statistic has a symmetric null distribution.

What conclusion follows for the prespecified one-sided test at significance level $0.05$ ?

The one-sided p-value is about $0.04$ , so the team rejects the null hypothesis. (correct answer)
The one-sided p-value remains $0.08$ , so the team does not reject the null hypothesis.
The one-sided p-value is about $0.16$ , so the team does not reject the null hypothesis.
The one-sided p-value is about $0.04$ , but rejection is invalid for any directional hypothesis.

Explanation: Whenever you see a question mixing one-sided and two-sided p-values, the key relationship to remember is how they connect under a symmetric null distribution (like the normal or t-distribution): the one-sided p-value is exactly half the two-sided p-value — but only when the observed effect is in the same direction as your alternative hypothesis. Here, the team prespecified that the new subject line would increase open rates, and the estimated effect is indeed positive. Because the null distribution is symmetric, the two-sided p-value of

0.08

splits equally into two tails of

0.04

each. Since the result falls in the predicted direction, the relevant one-sided p-value is

0.08 / 2 = 0.04

. Comparing to

\alpha = 0.05

, we have

0.04 < 0.05

, so the team rejects the null hypothesis. That makes A correct. B is wrong because it incorrectly keeps the p-value at

0.08

. That would apply if you were still running a two-sided test — switching to one-sided always changes the p-value when the distribution is symmetric. C is wrong in the opposite direction — doubling the p-value to

0.16

would be appropriate only if the observed effect were opposite to the predicted direction, meaning you'd look at the far tail rather than the near one. D is wrong because there is nothing statistically invalid about directional (one-sided) hypotheses when they are prespecified before data collection. Pre-registration is precisely what makes a one-sided test legitimate. Study tip: Always check two things before halving a two-sided p-value — (1) is the null distribution symmetric, and (2) does the effect match your predicted direction? Both must be true.

Question 6

After an A/B test, an analyst examines four secondary customer-engagement metrics and obtains p-values of $0.009$ , $0.014$ , $0.031$ , and $0.044$ . The company requires control of the familywise Type I error rate at $0.05$ and directs the analyst to use the Bonferroni method.

Which conclusion is appropriate under the required multiple-testing adjustment?

All four metrics are significant because every unadjusted p-value is below $0.05$ .
The first two metrics are significant because their p-values are both below the adjusted cutoff of $0.025$ .
Only the metric with p-value $0.009$ is significant because the adjusted cutoff is $0.0125$ . (correct answer)
No metric is significant because each p-value must be below the adjusted cutoff of $0.0025$ .

Explanation: Whenever you see a question involving multiple simultaneous hypothesis tests, your first instinct should be: how does the testing method change my significance threshold? The Bonferroni correction is the most conservative and straightforward adjustment — it divides the familywise error rate by the number of tests to set a new per-test cutoff. Here, the company requires a familywise Type I error rate of

\alpha = 0.05

across

m = 4

tests. The Bonferroni-adjusted threshold is:

\alpha^* = \frac{0.05}{4} = 0.0125

Now compare each p-value against

0.0125

: only

0.009 < 0.0125

. The remaining values —

0.014

0.031

, and

0.044

— all exceed the cutoff. So C is correct: only the first metric clears the Bonferroni threshold. A is the classic multiple-testing trap. Comparing every p-value to the unadjusted

0.05

ignores the inflation of Type I error that occurs when you run multiple tests simultaneously — exactly the problem Bonferroni is designed to fix. B uses an adjusted cutoff of

0.025

, which would be correct if there were only

m = 2

tests (

0.05/2

). The student who picks B is applying the right method but miscounting the number of comparisons. D uses

0.0025

, which would require dividing by

20

tests — a factor-of-five overcount. This reflects a misapplication of the formula with an incorrect denominator. Study tip: On any multiple-testing question, always identify

m

(number of tests) first, then compute

\alpha/m

. Mistakes almost always come from using the wrong

m

Question 7

A logistics company tests routing software intended to reduce average delivery cost. The estimated saving is $1.8$ dollars per delivery, with a $95\%$ confidence interval from $0.4$ to $3.2$ dollars. Management will adopt the software only if the true average saving is at least $2.5$ dollars.

Which conclusion best distinguishes statistical significance from the management requirement?

The software is statistically beneficial, but the interval does not establish that savings meet the $2.5$ -dollar requirement. (correct answer)
The software is not statistically beneficial because the point estimate is below the $2.5$ -dollar requirement.
The software meets the management requirement because the interval's upper endpoint exceeds $2.5$ dollars.
The software is statistically harmful because most values in the confidence interval are below the requirement.

Explanation: Whenever you see a question mixing confidence intervals with a business threshold, you need to keep two separate questions in mind: "Is there a statistically significant effect?" and "Does that effect meet the practical requirement?" These are genuinely different questions, and confusing them is the classic trap here. The confidence interval

[0.4, 3.2]

tells you that the true average saving is likely positive — since the entire interval sits above

0

, you have statistical evidence that the software does save money. That's statistical significance. However, management's threshold is

\$2.5

per delivery, and the interval ranges from

\$0.40

all the way up to

\$3.20

. Because

2.5

falls inside that interval, the data are consistent with savings below

\$2.5

just as much as above it. You cannot confidently conclude the true saving clears the hurdle. Answer A correctly captures both truths simultaneously: the software is statistically beneficial (savings exist), but the evidence doesn't confirm savings reach

\$2.5

. Answer B is wrong because statistical significance is determined by whether zero is excluded from the interval — not by comparing the point estimate to the management threshold. Answer C misreads the interval: the upper endpoint exceeding

\$2.5

only means that value is possible, not that it's guaranteed or even likely. Answer D is wrong on two counts — the software isn't "statistically harmful" (the interval is entirely positive), and using the spread of values to infer harm conflates uncertainty with negative effect. Study tip: Always ask two separate questions — "Does the interval exclude zero?" (significance) and "Does the interval exclude the threshold?" (practical requirement). A result can be statistically significant yet still fail a business benchmark.

Question 8

A pilot A/B test reports a $95\%$ confidence interval of $[-0.01,0.07]$ for the difference in purchase rates, defined as treatment minus control. A planned follow-up will assign four times as many customers to each group. Assume the observed rates and allocation ratio remain the same.

What is the best approximation to the follow-up experiment's $95\%$ confidence interval?

Approximately $[-0.01,0.07]$ because the confidence level is unchanged
Approximately $[0.01,0.05]$ because the standard error is roughly halved (correct answer)
Approximately $[0.02,0.04]$ because the interval width is roughly quartered
Approximately $[-0.05,0.11]$ because more observations add sampling variability

Explanation: Whenever you see a question about sample size and confidence intervals, your instinct should be: how does sample size affect the standard error? The standard error of a difference in proportions scales as

\frac{1}{\sqrt{n}}

, so quadrupling the sample size cuts the standard error in half — not by a factor of four. Here's the math: the original interval

[-0.01, 0.07]

has a center (point estimate) of

0.03

and a half-width of

0.04

, meaning the margin of error is

\pm 0.04

. If

n

increases by a factor of 4, the new standard error is

\frac{1}{\sqrt{4}} = \frac{1}{2}

of the original, so the new margin of error is approximately

0.04 / 2 = 0.02

. The new interval is

0.03 \pm 0.02 = [0.01, 0.05]

, confirming B is correct. A is wrong because keeping the same confidence level (95%) does not keep the same interval width — width depends on standard error, which depends on sample size. C reflects the mistake of thinking the width shrinks by the same factor as the sample size increase (factor of 4), but it actually shrinks by the square root of that factor (factor of 2). D inverts the logic entirely — more data reduces uncertainty, it does not increase it. A quick rule to memorize: to cut your margin of error in half, you must quadruple your sample size, because the relationship goes through a square root. Expect this pattern to appear whenever a question mentions changing sample size.

Question 9

An online retailer randomly assigns equal numbers of visitors to its current checkout and a redesigned checkout. The current checkout converts $480$ of $4{,}000$ visitors, while the redesign converts $612$ of $4{,}000$ visitors. The estimated standard error of the difference in conversion rates, defined as redesign minus current, is $0.009$ .

Which is the best approximate $95\%$ confidence interval for the treatment effect?

An increase from about $1.5$ to $5.1$ percentage points (correct answer)
A decrease from about $1.5$ to $5.1$ percentage points
An increase from about $2.4$ to $4.2$ percentage points
An increase from about $-0.3$ to $6.9$ percentage points

Explanation: When you see a question about A/B testing with a given standard error, your goal is to build a confidence interval around the observed difference in proportions using the formula:

\text{point estimate} \pm z^* \times SE

. Start by calculating the two conversion rates. The current checkout converts

480/4{,}000 = 0.12

(12%), and the redesign converts

612/4{,}000 = 0.153

(15.3%). The point estimate of the treatment effect (redesign minus current) is

0.153 - 0.12 = 0.033

, or 3.3 percentage points. For a 95% confidence interval, you use

z^* \approx 2

, giving a margin of error of

2 \times 0.009 = 0.018

(1.8 percentage points). The interval is therefore

3.3\% \pm 1.8\%

, which runs from approximately 1.5% to 5.1%. Since the entire interval is positive (redesign minus current), this represents an increase in conversion — confirming A is correct. B is wrong because it describes a decrease, but the redesign outperforms the current checkout, so the difference is positive throughout. C is wrong because it uses

z^* \approx 1

(roughly a 68% interval) instead of the correct

z^* \approx 2

for 95% confidence, producing too narrow a range of 2.4% to 4.2%. D would result from using

z^* \approx 3

(a 99.7% interval), which overshoots and unnecessarily widens the bounds. Your study tip: always keep the three standard z-values straight — 1 for 68%, 2 for 95%, 3 for 99.7%. On exam questions, choosing the wrong z-value is the most common trap in confidence interval problems.

Question 10

For the same experimental treatment effect, a company's $90\%$ confidence interval places zero strictly outside the interval, while its $95\%$ confidence interval places zero strictly inside the interval. Both intervals are based on the same two-sided procedure.

What can be inferred about the corresponding two-sided p-value?

It must be less than $0.05$ because one reported interval excludes zero.
It must be greater than $0.10$ because the wider interval includes zero.
It must lie between $0.05$ and $0.10$ because the test decisions differ. (correct answer)
It must equal $0.075$ because that value is midway between the two levels.

Explanation: Whenever you see a question connecting confidence intervals to p-values, the key relationship to remember is this: a two-sided confidence interval at confidence level

(1 - \alpha)

corresponds exactly to a two-sided hypothesis test at significance level

\alpha

. Zero falls outside the interval if and only if the p-value is less than

\alpha

. Here, the

90\%

interval excludes zero, which means the p-value is less than

\alpha = 0.10

. The

95\%

interval includes zero, which means the p-value is greater than

\alpha = 0.05

. Combining both constraints:

0.05 < p\text{-value} < 0.10

. That's exactly what answer C states — the p-value is pinned between the two critical thresholds because the test decisions flip between those two levels. A is wrong because it only uses one piece of information (the 90% interval excluding zero) and draws an overly strong conclusion. That exclusion tells you

p < 0.10

, not

p < 0.05

— you'd need the 95% interval to also exclude zero to conclude

p < 0.05

. B is wrong for the mirror-image reason. The 95% interval including zero tells you

p > 0.05

, not

p > 0.10

. Calling the interval "wider" and jumping to the larger threshold misapplies the logic. D is a trap answer that invents a false precision. There is no rule that averages the two alpha levels to produce the p-value; the actual p-value could be anywhere in

(0.05, 0.10)

. Study tip: Memorize the duality rule — a

(1-\alpha)

confidence interval excludes zero if and only if

p < \alpha

. Questions that give you two intervals are simply asking you to apply this rule twice and intersect the results.