Historical Context & Motivation
Randomized controlled experiments have been the gold standard for establishing causal relationships since R.A. Fisher formalized their design in the 1920s and 1930s. When internet companies began scaling rapidly in the early 2000s, they adapted Fisher's framework into what the technology industry now calls A/B testing — splitting live traffic between a control (A) and a variant (B) to measure whether a change in design, copy, pricing, or algorithm moves a business metric. Companies like Google, Amazon, and Netflix soon reported running hundreds of concurrent experiments at a time, and the practice quickly spread to startups, e-commerce brands, and even political campaigns.
Yet the democratization of experimentation brought a predictable consequence: many teams adopted the tool without fully understanding the statistical assumptions that make it valid. By the early 2010s, industry practitioners and academic statisticians alike began documenting systematic errors — false discoveries, inflated effect sizes, and irreproducible results — that traced back to a small set of recurring mistakes. Three pitfalls emerged as the most damaging: peeking at results before the sample size is reached, running multiple comparisons without adjusting significance thresholds, and selection bias in how users are assigned to groups. Understanding these pitfalls is essential for any business analyst who wants experiment-driven decisions to be trustworthy rather than merely data-flavored guesswork.
The central question this lesson addresses is deceptively simple: if A/B testing is just 'randomly split users and measure outcomes,' why do so many experiments produce conclusions that fail to replicate? The answer lies in the gap between the assumptions of classical hypothesis testing and the realities of how businesses actually run experiments — a gap that peeking, multiple comparisons, and selection bias exploit relentlessly.
Core Principles & Definitions
Before diagnosing what goes wrong, it helps to recall how a correctly executed A/B test is supposed to work. You begin by stating a null hypothesis (H₀: the variant has no effect on the metric) and an alternative hypothesis (H₁: the variant does have an effect). You choose a significance level (α, often 0.05) and compute the required sample size based on the minimum detectable effect, baseline conversion rate, and desired statistical power. Users are randomly assigned, the experiment runs until the pre-determined sample size is reached, and only then is the test statistic calculated and compared to the critical value. Each of the three pitfalls violates one or more of these steps.
Peeking (Optional Stopping)
Multiple Comparisons
Selection Bias
Type I vs. Type II Errors
Visual Explanation — The Peeking Problem
The diagram below illustrates how peeking inflates the false positive rate. In a correctly run test (left path), the analyst waits until the pre-determined sample size is reached before evaluating the p-value. In the peeking scenario (right path), the analyst checks the p-value at multiple interim points — and stops as soon as any check crosses the significance threshold. Because random noise is largest in small samples, early p-values fluctuate wildly, and the cumulative probability of at least one spurious 'significant' result grows with every peek.
The diagram makes clear that the issue is not merely carelessness — it is a structural consequence of the test's design. A p-value of 0.05 guarantees a 5% false positive rate only if the test is evaluated once at the pre-specified sample size. Every additional peek is like buying an extra lottery ticket: individually each has low odds of a false alarm, but collectively the chances compound. This is why modern experimentation platforms increasingly implement sequential testing methods that control the overall error rate even when analysts check results continuously.
Mathematical Framework
Each of the three pitfalls can be understood quantitatively. The mathematics are not complex, but they reveal how quickly error rates escalate when assumptions are violated. Below we formalize the key relationships.
Peeking — Cumulative Type I Error
Multiple Comparisons — Family-Wise Error Rate
Selection Bias — Confounding in Estimation
Detailed Breakdown of Each Pitfall
Each pitfall has distinct causes, symptoms, and remedies. The visual below maps the full landscape, showing how each error enters the experimentation workflow and what safeguards exist.
Deep Dive: Multiple Comparisons in Practice
Multiple comparisons occur more often than analysts realize. Every time you slice your experiment by a user segment — mobile vs. desktop, new vs. returning, US vs. international — you are implicitly running additional hypothesis tests. A test with one variant and ten metrics already has ten comparisons; add three user segments and the count jumps to thirty. At α = 0.05, the probability of at least one false positive among thirty independent tests is 1 − 0.95³⁰ ≈ 0.785, or nearly 79%. The Bonferroni correction addresses this by dividing the per-test significance level by the number of comparisons (0.05 / 30 ≈ 0.00167), though its conservatism reduces power. In practice, many organizations adopt the Benjamini-Hochberg procedure, which controls the false discovery rate rather than the family-wise error rate, offering a better balance between rigor and sensitivity.
Deep Dive: Selection Bias in Digital Experiments
Selection bias can enter an A/B test through several doors. The most obvious is non-random assignment — for example, routing all high-value customers to the variant because the product team wants them to see the 'better' experience. Subtler forms include survivorship bias (analyzing only users who completed the funnel, ignoring those who dropped out) and novelty or primacy effects (early adopters of a new feature behave differently from the general population). A practical safeguard is the AA test — running both groups through identical experiences to verify that the randomization mechanism produces statistically indistinguishable groups before introducing the treatment.
Worked Example — Diagnosing Pitfalls in a Pricing Test
Imagine you are a data analyst at an e-commerce company. The marketing team ran a pricing experiment: half of visitors saw the original price ($49.99) and half saw a discounted price ($39.99). The team reports the following: they checked the dashboard after three days (about 40% of the planned sample), found a p-value of 0.03 on conversion rate, and declared the discount a winner. They also note that revenue per visitor, average order value, and return rate were analyzed but were not significant — only conversion rate was. Finally, they mention that the experiment was shown only to users who had previously created an account, not to anonymous visitors. Let us diagnose the pitfalls.
Remedies — Strengths and Limitations
Each pitfall has well-established statistical remedies, but no remedy is without trade-offs. The table below compares the most common approaches across practical dimensions that matter to business teams: ease of implementation, impact on statistical power, and organizational adoption barriers.
| Remedy | Pitfall Addressed | Strengths | Limitations |
|---|---|---|---|
| Fixed-horizon testing | Peeking | Simple to implement; exact Type I error control at α. No special software required. | Requires analysts to resist checking early; may run longer than necessary if true effect is large. |
| Group sequential testing | Peeking | Allows pre-specified interim looks while controlling overall α via spending functions (e.g., O'Brien-Fleming). | Requires upfront planning of interim analysis times; slightly reduced power at each look. |
| Bonferroni correction | Multiple comparisons | Guarantees FWER ≤ α. Easy to compute and explain. Works regardless of dependence among tests. | Very conservative — may miss real effects when many comparisons are made. |
| Benjamini-Hochberg | Multiple comparisons | Controls false discovery rate (FDR), offering greater power than Bonferroni when many hypotheses are tested. | Assumes independence or positive dependence among tests; less intuitive to explain to non-technical stakeholders. |
| Proper randomization + AA test | Selection bias | Eliminates systematic differences at baseline; AA test provides empirical evidence that assignment is balanced. | Requires engineering infrastructure for true randomization; AA test adds time; doesn't address attrition bias. |
Connection to Advanced Experimentation Methods
The three pitfalls discussed here are entry points into deeper statistical territory. Understanding them positions you to engage with advanced methods that modern experimentation teams use to scale testing without sacrificing rigor. The table below maps each pitfall to the advanced methodology that addresses it most directly.
| Pitfall | Basic Remedy | Advanced Method | Key Idea |
|---|---|---|---|
| Peeking | Fixed-horizon design | Always-valid inference (confidence sequences) | Construct confidence intervals that are valid at every sample size, allowing continuous monitoring without inflating α. Based on martingale theory. |
| Multiple comparisons | Bonferroni / BH correction | Bayesian hierarchical models | Model all metrics jointly, letting partial pooling (shrinkage) reduce extreme estimates. Naturally handles multiplicity through the prior. |
| Selection bias | Randomization + AA test | Regression adjustment / CUPED | Use pre-experiment covariates to reduce variance and correct for residual imbalance. CUPED (Controlled-experiment Using Pre-Experiment Data) is widely used at Netflix, Microsoft, and others. |
| All three | Pre-registration | Experimentation platforms with guardrails | Automated platforms (e.g., Statsig, Eppo, Optimizely) enforce sample size commitments, apply corrections, and flag imbalanced assignments by default. |
The trajectory from basic to advanced is clear: modern experimentation seeks to make rigor automatic rather than optional. Always-valid inference eliminates the peeking problem entirely by redefining what a valid confidence interval means. Bayesian hierarchical models replace the ad-hoc Bonferroni adjustment with a principled framework for borrowing information across metrics. CUPED leverages historical data to both reduce variance and guard against residual selection bias. As you progress in business analytics, these methods will become essential tools for designing experiments that stakeholders can trust.
Practice Problems
Lesson Summary
A/B testing is one of the most powerful tools in business analytics, but its value depends entirely on the rigor with which experiments are designed and analyzed. This lesson examined three pervasive pitfalls: peeking (optional stopping), which inflates the false positive rate by evaluating results before the planned sample size; multiple comparisons, which compounds error when many metrics, variants, or segments are tested simultaneously; and selection bias, which occurs when non-random assignment or differential attrition makes the treatment and control groups fundamentally incomparable. Each pitfall has a mathematical signature — peeking raises the cumulative Type I error rate, multiple comparisons increase the family-wise error rate, and selection bias introduces a confounding term into the estimated treatment effect.
The remedies are well established: fixed-horizon or sequential testing controls error across interim looks; the Bonferroni correction and Benjamini-Hochberg procedure adjust significance thresholds for multiplicity; and proper randomization combined with AA tests ensures balanced groups at baseline. The overarching principle is pre-registration — committing to a primary metric, sample size, analysis plan, and correction method before data collection begins. This single practice neutralizes most of the pathways through which these pitfalls operate and is the hallmark of a mature experimentation culture.