Sample Size & Power - Business Analytics

1. What is the smallest whole number of stores that should be included using this approximation?
   Answer: $72$ stores, representing $3{,}600$ shoppers When a study randomizes clusters (like stores) instead of individuals, the effective sample size must be inflated to account for the fact that people within the same cluster tend to behave similarly. The design effect (DEFF) captures this inflation: $\text{DEFF} = 1 + (m-1)\rho$, where $m$ is the cluster size and $\rho$ is the intraclass correlation. You multiply your original individual-level sample size by this factor to get the required number of individuals under cluster randomization, then divide by the cluster size to find the number of clusters needed. Here, $m = 50$ shoppers per store and $\rho = 0.04$, so: $\text{DEFF} = 1 + (50-1)(0.04) = 1 + 49 \times 0.04 = 1 + 1.96 = 2.96$ The adjusted total number of shoppers needed is: $1{,}200 \times 2.96 = 3{,}552$ Dividing by 50 shoppers per store gives $3{,}552 / 50 = 71.04$ stores. Since you must round up to the next whole number (you can't have a partial store), you need 72 stores, representing 3,600 shoppers — confirming answer D. Choice A ignores the design effect entirely, using the original individual-randomization figure as if clustering didn't matter. Choice B applies an incorrect design effect of 2.5, which would correspond to a different $\rho$ or $m$. Choice C makes the critical rounding error — 71.04 rounds up to 72, not down to 71, because you must meet the minimum threshold. Remember: always round cluster counts up, never down. Rounding down would leave you underpowered, which defeats the purpose of the calculation.
2. Which allocation is approximately optimal?
   Answer: $600$ treatment customers and $300$ control customers When you see an experiment with unequal variances across groups, the key insight is that equal sample sizes are not automatically optimal. The goal here is to minimize the variance of the estimated difference in means, which equals $\frac{\sigma_T^2}{n_T} + \frac{\sigma_C^2}{n_C}$, subject to the constraint $n_T + n_C = 900$. The optimal allocation rule (Neyman allocation) says you should assign participants proportionally to each group's standard deviation: $n_T \propto \sigma_T$ and $n_C \propto \sigma_C$. With $\sigma_T = 18$ and $\sigma_C = 9$, the ratio is $18:9 = 2:1$. Splitting 900 in a 2:1 ratio gives $n_T = 600$ and $n_C = 300$, confirming that C is correct. You can verify: the resulting variance is $\frac{18^2}{600} + \frac{9^2}{300} = 0.54 + 0.27 = 0.81$, which is lower than any other split. A (300 treatment, 600 control) reverses the logic — it allocates more participants to the lower-variance group, which inflates the variance of the high-variance group unnecessarily, yielding $\frac{324}{300} + \frac{81}{600} = 1.08 + 0.135 = 1.215$, far worse. B (450/450) is the intuitive "fair split" trap — equal allocation ignores the variance difference entirely, producing $0.72 + 0.18 = 0.90$, worse than C. D (720 treatment, 180 control) over-allocates to treatment beyond the optimal ratio, giving $0.45 + 0.45 = 0.90$, also worse. Your study tip: whenever standard deviations differ across groups, remember Neyman allocation — assign proportionally to $\sigma$, not equally. Equal splits only minimize variance when $\sigma_T = \sigma_C$.
3. Which statement about the proposed change is correct?
   Answer: If specified in advance, it can increase power for an increase but will not support the same claim for a decrease. When a question asks about switching from a two-sided to a one-sided test, focus on two things: when the decision is made and what the test can and cannot conclude. A one-sided test concentrates all of your significance level (α) into one tail — the direction you care about. When you pre-specify a one-sided test before seeing any data, you gain statistical power in that direction compared to a two-sided test at the same α, because you're no longer "wasting" half the rejection region on the opposite tail. However, this comes with a strict trade-off: you formally surrender the ability to make a statistically supported claim in the other direction. In the retailer's scenario, a one-sided test increases power to detect a revenue increase, but if the engine actually decreased revenue, the test provides no statistical basis to claim a significant decrease. That's exactly what A describes — and it's correct precisely because the decision is made in advance. B is wrong because the timing matters enormously. Choosing a one-sided test after observing a positive estimate is p-hacking. It artificially inflates Type I error (false-positive risk) rather than controlling it, making the result invalid. C has the logic backwards. A one-sided test increases power for the specified direction by consolidating α into one tail, not decreasing it. D is a fabrication. No such guarantee exists — sample size requirements depend on effect size, variance, and desired power, not a fixed halving rule. Study tip: On hypothesis-testing questions, always ask: was the directionality decided before or after seeing the data? Pre-specification is what separates a legitimate one-sided test from a biased one.
4. How many employees should be randomized in total, ignoring any additional safety margin?
   Answer: Approximately $2{,}200$ employees When a study expects dropout, you must inflate the randomized sample to ensure the analyzed sample meets your target. The key insight is that each arm may have a different retention rate, so you must calculate the required randomized count separately for each arm, then sum them. Start with the treatment arm: you need $900$ analyzed employees, and you expect to retain $90\%$. So you randomize $\frac{900}{0.90} = 1{,}000$ employees. For the control arm: you need $900$ analyzed employees, but retention is only $75\%$, so you randomize $\frac{900}{0.75} = 1{,}200$ employees. The total randomized is $1{,}000 + 1{,}200 = 2{,}200$, confirming D is correct. Choice A ($1{,}800$) is the trap of simply doubling the analyzed target without any dropout adjustment at all — it assumes perfect retention in both arms, which the passage explicitly contradicts. Choice B ($2{,}000$) likely comes from applying a single average retention rate (roughly $82.5\%$) to the total, but averaging retention rates across arms is mathematically incorrect when dropout rates differ. Choice C ($2{,}115$) may result from applying the treatment arm's $90\%$ retention rate to both arms or from another partial adjustment, but it fails to correctly account for the control arm's lower $75\%$ retention. A reliable study tip: whenever arms have different dropout rates, always inflate each arm independently before summing. Never average retention rates across groups — the arm with the worst retention drives a disproportionately large inflation and must be handled separately.
5. Which statement best describes the consequences of this change?
   Answer: The false-positive risk decreases, power decreases, and the minimum detectable effect increases. Whenever you see a hypothesis testing question involving a change to the significance level, think about the three-way relationship between Type I error risk, statistical power, and minimum detectable effect (MDE) — they're all connected through the same underlying probability thresholds. Lowering $\alpha$ from $0.05$ to $0.01$ means you're demanding stronger evidence before rejecting the null hypothesis. This directly reduces your false-positive rate (Type I error), which is exactly what the compliance team wanted. But this strictness comes at a cost: with a fixed sample size, you're moving the critical value further into the tail of the distribution. That makes it harder for a true effect to clear the new, higher bar — so power (the probability of detecting a real effect) decreases. Because power falls, the smallest effect the test can reliably detect must be larger — meaning the MDE increases. Answer A captures all three of these consequences correctly. Answer B is wrong because it claims power increases, which is the opposite of what happens. Lower $\alpha$ reduces power when nothing else changes. Answer C incorrectly states that false-positive risk increases — lowering $\alpha$ does the opposite, it tightens the threshold. Answer D claims false-positive risk is unchanged, which contradicts the definition of $\alpha$ itself; changing $\alpha$ is literally changing the false-positive rate. A useful memory anchor: think of the significance level as a filter. A finer filter (smaller $\alpha$) catches fewer false positives, but it also lets real signals slip through more often — so you need bigger effects to be seen.
6. Approximately how many customers per group should the revised test include?
   Answer: About $2{,}844$ customers per group When a question asks how sample size changes after adjusting the minimum detectable effect, you need to remember the fundamental relationship: sample size scales inversely with the square of the effect size. Specifically, $n \propto \frac{1}{(\Delta)^2}$, where $\Delta$ is the lift you want to detect. Here's the logic. The original test was designed to detect a $4$-percentage-point lift with $1{,}600$ customers per group. Now management wants to detect a $3$-percentage-point lift — a smaller effect — which always requires a larger sample. The scaling factor is: $n_{\text{new}} = n_{\text{old}} \times \left(\frac{\Delta_{\text{old}}}{\Delta_{\text{new}}}\right)^2 = 1{,}600 \times \left(\frac{4}{3}\right)^2 = 1{,}600 \times \frac{16}{9} \approx 2{,}844$ That confirms C is correct. Looking at why the other choices fail: A (900 customers) implies the sample decreases, which is backwards — detecting a smaller effect always demands more data, not less. B (2,133) corresponds to a linear scaling $1{,}600 \times \frac{4}{3}$, forgetting that the relationship is squared, not linear. D (3,200) doubles the original sample, which would correspond to halving the detectable effect, not reducing it by one percentage point. A useful rule of thumb: whenever the detectable effect shrinks by a factor of $k$, your required sample grows by $k^2$. Here, the effect shrank by $\frac{4}{3}$, so the sample grew by $\left(\frac{4}{3}\right)^2 \approx 1.78\times$. Memorize this squared inverse relationship — it appears frequently in A/B testing and experimental design questions.
7. Which interpretation of the power specification is correct?
   Answer: If the true lift is exactly $6$ percent, repeated tests will reject the null about $80$ percent of the time. Whenever you see a question about statistical power, anchor yourself to this definition: power is the probability of correctly rejecting a false null hypothesis, given a specific assumed effect size. It is a statement about what happens if the true effect exists — nothing more. With that foundation, B is the only valid interpretation. If the true sales lift is exactly $6\%$, then across many repeated experiments of this design, the null hypothesis will be rejected approximately $80\%$ of the time. Power lives entirely in the world of "assuming the effect is real — how often do we catch it?" That is precisely what B states. A gets the conditional backwards. Power conditions on the true effect existing; A conditions on the result being significant and then makes a probability claim about the true lift. That's a positive predictive value calculation, which depends on the base rate of true effects and the false-positive rate — not on power alone. This is the exact confusion the manager in the passage makes. C confuses power with the false-negative rate ($1 - \text{power} = 20\%$). A nonsignificant result means you failed to detect an effect — it says nothing about the probability that the null is actually true. Confusing "failing to reject" with "the null is true" is a classic logical error. D is simply a misreading of the word "power" in plain English, importing a consumer-behavior meaning that has nothing to do with hypothesis testing. Study tip: Always ask, "What is being conditioned on?" Power conditions on the true effect — not on the observed result. Questions that flip that conditional are testing whether you know the difference.
8. Which comparison of required sample sizes is most accurate?
   Answer: Test L needs about one-quarter as many observations because its binomial outcome variance is lower. Whenever you see a sample-size comparison across A/B tests, your instinct should be to reach for the two-proportion z-test formula. Required sample size per group scales with $\frac{p_1(1-p_1) + p_2(1-p_2)}{(p_2 - p_1)^2}$. The denominator is identical for both tests — a two-point absolute lift means $(0.07-0.05)^2 = (0.52-0.50)^2 = 0.0004$ in both cases. So the difference comes entirely from the numerator, the sum of the two binomial variances $p(1-p)$. For Test L: $0.05(0.95) + 0.07(0.93) \approx 0.0475 + 0.0651 = 0.1126$. For Test H: $0.50(0.50) + 0.52(0.48) \approx 0.2500 + 0.2496 = 0.4996$. The ratio is roughly $0.4996 / 0.1126 \approx 4.4$, meaning Test H requires about four times more observations — or equivalently, Test L needs roughly one-quarter the sample of Test H. That confirms answer A. Answer B gets the direction exactly backwards: Test H does not need fewer observations. Its higher baseline inflates variance, which increases required sample size. Answer C is the most seductive trap — students often think "same absolute lift = same sample size," but this ignores the variance term entirely. Answer D introduces an irrelevant consideration; events-per-unit-time affects how quickly you accumulate data in calendar time, but it has no bearing on the statistical sample size requirement. Your takeaway: never equate "same absolute lift" with "same required sample." Always check the variance $p(1-p)$, which peaks at $p = 0.50$ — high-baseline tests are the most expensive to run.
9. Under the usual sample-size approximation, what total sample should now provide approximately the original power?
   Answer: About $1{,}280$ deliveries When you reduce variability in an experiment by controlling for a strong covariate, you directly reduce the noise in your outcome measure. Sample size under the standard approximation scales with the square of the standard deviation — specifically, required $n \propto \sigma^2$. This means if you can shrink $\sigma$, you need proportionally fewer observations to achieve the same power. Here, the residual standard deviation drops from $20$ to $16$ minutes. The ratio of the new required sample to the original is: $\frac{n_{\text{new}}}{n_{\text{original}}} = \frac{\sigma_{\text{new}}^2}{\sigma_{\text{original}}^2} = \frac{16^2}{20^2} = \frac{256}{400} = 0.64$ Applying this to the original sample of $2{,}000$: $n_{\text{new}} = 2{,}000 \times 0.64 = 1{,}280$ That confirms B as the correct answer. Looking at the distractors: A (800) applies a ratio of $0.40$, which might come from mistakenly using the ratio of standard deviations ($16/20 = 0.8$) and then squaring incorrectly or halving — a common algebraic slip. C (1,600) uses the linear ratio $16/20 = 0.8$ directly without squaring, a classic error of forgetting that sample size scales with variance, not standard deviation. D (2,500) goes in the wrong direction entirely, implying you'd need more observations after reducing variability — the opposite of the truth. Study tip: Always remember that reducing $\sigma$ by a factor of $k$ reduces required sample size by a factor of $k^2$. When you see covariate adjustment or blocking in a question, immediately think "variance reduction → squared savings in sample size."
10. Which recommendation is most defensible?
    Answer: Run an adequately powered follow-up because the pilot did not rule out worthwhile effects. Whenever you see a question combining a nonsignificant p-value with a low-powered study, your first instinct should be to think carefully about what "nonsignificant" actually means — it does not mean "no effect exists." Statistical power of only $35\%$ means this pilot had a $65\%$ chance of missing a real, worthwhile effect even if one existed. With such a weak test, a p-value of $0.18$ is essentially uninformative — you cannot distinguish "no effect" from "real effect that the study was too small to detect." Crucially, the confidence interval spans $-1$ to $+5$ percentage points, which includes the commercially meaningful $+3$-point threshold. This means a worthwhile lift remains entirely plausible. The only honest conclusion is that the pilot was inconclusive, and a properly powered study is needed — making C the most defensible recommendation. A commits the classic low-power fallacy: interpreting a nonsignificant result as proof of no effect. Absence of evidence is not evidence of absence, especially when power is this low. B is tempting but dangerously selective — the interval's upper bound exceeding the worthwhile threshold doesn't justify launch, because the interval equally includes zero and even a small decline. Cherry-picking the optimistic endpoint ignores real uncertainty. D describes p-hacking: repeatedly running small tests until chance produces a significant result inflates your false-positive rate and is both statistically invalid and ethically indefensible. As a study tip, remember this pattern: low power + nonsignificant result = inconclusive, not negative. On business analytics exams, questions pairing underpowered pilots with "no effect" conclusions are almost always traps.

Start this quiz