Random Sampling & Bias — Random Sampling and Bias - College Statistics

SECTION 1

Historical Context & Motivation

Long before statistics emerged as a formal discipline, governments and institutions needed to learn about large populations without examining every individual. Ancient census-taking—recorded in Roman, Chinese, and Egyptian archives—attempted complete enumeration, but the sheer cost and logistical difficulty of reaching every person made total counts impractical even for powerful empires. The recognition that a carefully chosen subset could stand in for an entire population set the stage for modern sampling theory. However, the critical question of how to choose that subset remained unresolved for centuries, and the consequences of choosing poorly—what we now call sampling bias—produced some of the most spectacular failures in the history of data collection.

1895 • Kiaer's Representative Method

Norwegian statistician Anders Kiaer proposed using a carefully selected subset—a representative sample—instead of a full census. His presentation at the International Statistical Institute sparked decades of debate about whether samples could ever substitute for complete enumeration.

1936 • The Literary Digest Debacle

The Literary Digest mailed ten million questionnaires predicting Alf Landon would defeat Franklin Roosevelt in the presidential election. The magazine sampled from telephone directories and automobile registrations—sources that over-represented wealthier voters—producing one of the most famous examples of selection bias in history.

1948 • Neyman's Formal Framework

Jerzy Neyman published foundational work establishing the mathematical justification for probability sampling, proving that random selection provides unbiased estimates of population parameters and enables the construction of confidence intervals.

1960s–1980s • Survey Sampling Matures

Cochran, Kish, and others refined stratified, cluster, and multistage designs. Government agencies such as the U.S. Census Bureau and the Bureau of Labor Statistics adopted sophisticated probability sampling frameworks for national surveys.

2010s–Present • Big Data and New Bias Challenges

The explosion of convenience data from social media, electronic health records, and web analytics has reignited concerns about bias. Large sample size does not eliminate bias when the data-generating mechanism is non-random, echoing the Literary Digest lesson in a digital age.

The central thread across this history is a deceptively simple insight: the way you select your sample determines whether your conclusions about the population are trustworthy. A sample of ten million biased observations is less reliable than a well-designed random sample of one thousand. Understanding why this is so—and how to design sampling procedures that avoid systematic distortion—is the objective of this lesson.

SECTION 2

Core Principles & Definitions

Before examining specific sampling methods, we must establish the vocabulary that makes precise discussion possible. The logic of statistical inference flows from a clear distinction between the population (the complete set of units about which we wish to draw conclusions) and the sample (the subset we actually observe). A sampling frame is the operational list from which sample members are drawn, and discrepancies between the frame and the true population constitute one of the most common sources of bias.

Random Sampling

A sampling procedure in which every member of the population has a known, non-zero probability of being selected. Randomization is what permits valid probability statements about the distance between sample statistics and population parameters.

Sampling Bias

A systematic tendency for the sampling procedure to favor certain members of the population over others. Unlike random error, bias does not diminish with larger sample sizes; it is a structural defect in the selection mechanism itself.

Simple Random Sample (SRS)

A sample of size n drawn so that every possible combination of n units from the population is equally likely. SRS serves as the theoretical benchmark against which other designs are evaluated.

Sampling Frame

The actual list or mechanism used to identify potential sample members. Frame under-coverage (missing population members) and over-coverage (including non-members) are primary sources of selection bias.

Non-Response Bias

Occurs when selected individuals decline or are unable to participate, and their characteristics differ systematically from those who do respond. Even a perfectly random initial selection can yield biased estimates if non-response is differential.

✦

KEY TAKEAWAY

Think of a random sample like drawing names from a thoroughly mixed bowl. If every slip of paper has the same size, shape, and texture, each name is equally likely to be drawn. Bias is what happens when some slips are larger, stickier, or closer to the top—the draw is no longer fair. Crucially, reaching into the bowl a thousand times instead of ten does not fix the problem if the slips are unequal; you simply get a very large, very biased sample.

SECTION 3

Visual Explanation — Population, Frame, and Sample

The nested Venn-style diagram shows three concentric regions. The outermost dashed ellipse represents the population (N). Inside it, the solid cyan ellipse is the sampling frame—the list from which we actually draw. The innermost green ellipse is the observed sample (n). Red dots outside the frame illustrate under-coverage bias, and amber dots near the sample boundary represent non-response bias.

This diagram captures the fundamental geometry of the sampling process. The gap between the population ellipse and the frame ellipse represents population members who have no chance of being selected—they are simply absent from the list. The Literary Digest poll's reliance on telephone directories created precisely this gap: Americans without telephones were effectively invisible to the survey, and those Americans disproportionately supported Roosevelt. Within the frame, the transition from the full frame to the realized sample introduces additional potential for distortion through non-response. A truly unbiased estimate requires that (1) the frame matches the population, (2) selection is random, and (3) response rates are uniform or appropriately weighted.

SECTION 4

Mathematical Framework

The mathematical power of random sampling lies in the fact that randomization allows us to derive the expected value and variance of sample statistics without knowing every value in the population. When selection is truly random, the sample mean is an unbiased estimator of the population mean—a property that fails under biased sampling designs regardless of sample size.

UNBIASEDNESS OF THE SAMPLE MEAN

E(X̄) = μ

Under a simple random sample, the expected value of the sample mean X̄ equals the population mean μ. This holds for any sample size n ≥ 1. Bias occurs when E(X̄) ≠ μ.

BIAS DEFINITION

Bias(X̄) = E(X̄) − μ

Bias is the systematic difference between the expected value of the estimator and the true parameter. A sampling procedure is unbiased when this quantity equals zero. Note that bias is a property of the procedure, not of any single sample.

VARIANCE OF THE SAMPLE MEAN (SRS WITHOUT REPLACEMENT)

Var(X̄) = (σ² / n) × (1 − n/N)

Here σ² is the population variance, n is the sample size, and N is the population size. The factor (1 − n/N) is the finite population correction (fpc), which reduces variance when sampling a non-negligible fraction of the population.

MEAN SQUARED ERROR DECOMPOSITION

MSE(X̄) = Var(X̄) + [Bias(X̄)]²

The mean squared error captures total estimation error by combining variance (random scatter) and squared bias (systematic shift). Increasing n reduces Var(X̄) but does nothing to the bias term—making this decomposition the formal reason that large biased samples remain unreliable.

⚠

Why Increasing n Doesn't Fix Bias

Consider the MSE decomposition. As n → ∞, the variance component σ²/n → 0, but the [Bias(X̄)]² term remains unchanged. A biased sampling procedure converges not to the true population mean μ, but to some other value μ + Bias(X̄). This is the mathematical formalization of the Literary Digest lesson: ten million observations converged on the wrong answer because the bias term was large and fixed.

SECTION 5

Classification of Sampling Methods & Bias Types

Not all sampling procedures are created equal, and not all deviations from ideal randomization produce the same kind of error. Understanding the taxonomy of sampling methods—and the specific biases each is susceptible to—is essential for evaluating the credibility of any empirical study. We distinguish broadly between probability sampling methods (in which every unit has a known, non-zero selection probability) and non-probability sampling methods (in which selection probabilities are unknown or zero for some units).

A hierarchical taxonomy of sampling methods (top) and common bias types (bottom). Probability methods (left branch, green border) provide theoretical guarantees of unbiasedness. Non-probability methods (right branch, red border) lack these guarantees and are susceptible to the bias types shown below.

Sampling Method	Key Feature	Primary Bias Risk
Simple Random Sample	Every subset of size n equally likely	Minimal if frame matches population; non-response still possible
Stratified Random	Population divided into homogeneous strata; SRS within each	Misclassification of strata; disproportionate allocation without proper weighting
Cluster	Randomly select clusters (groups); census within selected clusters	Within-cluster homogeneity inflates variance; unequal cluster sizes
Systematic	Select every k-th unit from a randomized list	Periodicity in the list can create hidden patterns
Convenience	Select whoever is easiest to access	Severe selection bias; no probability model for inference
Voluntary Response	Individuals self-select into the sample	Over-representation of strong opinions; extreme bias

Comparison of common sampling methods and their associated bias risks

SECTION 6

Worked Example — Identifying and Quantifying Bias

A university wants to estimate the average number of hours per week its N = 12,000 students spend studying. The administration emails a survey to all students; 800 respond. The sample mean is X̄ = 18.5 hours. A subsequent mandatory census (conducted during registration) reveals the true population mean μ = 14.2 hours. Let us analyze the bias in the voluntary-response sample and compare it to what a properly conducted SRS would have yielded.

Analyzing Bias in a Voluntary-Response Survey vs. SRS

1
Step 1 — Identify the Sampling Method and Its Limitations

The administration emailed all 12,000 students, but only those who chose to respond (800 out of 12,000, a response rate of 6.7%) contributed data. This is a voluntary response sample. Students who study more may be more motivated to respond to a survey about studying—creating systematic over-representation of high-study-hour students.
2
Step 2 — Calculate the Observed Bias

Using the bias formula: Bias(X̄) = E(X̄) − μ. Since the census reveals μ = 14.2 and the voluntary sample produced X̄ = 18.5, the estimated bias is:

Bias ≈ 18.5 − 14.2 = +4.3 hours (overestimate)
3
Step 3 — Compute MSE for the Biased Sample

Suppose the sample variance is s² = 25.0 (standard deviation s = 5.0). The MSE decomposes as MSE = Var(X̄) + [Bias]². The variance of the mean is Var(X̄) ≈ s²/n = 25/800 = 0.03125, so MSE = 0.03125 + (4.3)² = 0.03125 + 18.49.

MSE ≈ 18.52. Note that 99.8% of the MSE comes from the bias term, not the variance.
4
Step 4 — Compare with a Hypothetical SRS of n = 400

An SRS of only n = 400 (half the biased sample size) with population variance σ² = 30 and N = 12,000 yields Var(X̄) = (30/400) × (1 − 400/12000) = 0.075 × 0.9667 ≈ 0.0725. Because SRS is unbiased, Bias = 0, so MSE = 0.0725 + 0 = 0.0725.

MSE(SRS, n=400) ≈ 0.0725 — roughly 255 times smaller than the biased sample's MSE, despite being half the size.
5
Step 5 — Draw the Conclusion

This example vividly demonstrates the MSE decomposition in practice. The voluntary response sample of 800 has a tiny variance component but an enormous bias component. The SRS of 400 has a slightly larger variance component but zero bias, producing dramatically lower total error. A smaller random sample beats a larger biased one.

SECTION 7

Strengths & Limitations of Random Sampling

Random sampling is the gold standard for statistical inference, but it is not without practical challenges. The following table contrasts the key advantages with the real-world difficulties that researchers face when implementing probability-based designs.

Strengths	Limitations
Provides unbiased estimates with quantifiable precision (confidence intervals, margins of error)	Requires a complete, accurate sampling frame—often expensive or impossible to obtain
Results generalize to the population with known confidence levels	Non-response can undermine randomization even when initial selection is perfect
Mathematical framework enables sample size planning before data collection	Logistically complex for geographically dispersed or hard-to-reach populations
Eliminates systematic selection bias by design	Does not prevent response bias (e.g., social desirability, leading questions)
Stratification and clustering can improve efficiency and reduce costs	Complex designs require careful weighting in analysis; ignoring weights reintroduces bias

Strengths and limitations of probability-based random sampling

✦

KEY TAKEAWAY

Random sampling is analogous to a well-calibrated measuring instrument: it gives you readings that are centered on the true value (unbiased), even though individual readings fluctuate (variance). A biased sampling procedure is like a scale that consistently reads 5 kg too high—taking more measurements gives you a very precise wrong answer. The practical lesson is that effort spent eliminating bias (improving the frame, boosting response rates) has far greater payoff than effort spent simply increasing sample size.

SECTION 8

Connection to Advanced Theory

The principles of random sampling and bias extend into more advanced statistical territory. Understanding these connections helps situate the current lesson within the broader landscape of inferential statistics and causal reasoning. Two particularly important bridges are the relationship between random sampling (which supports generalizability) and random assignment (which supports causal inference), and the formal theory of design-based inference versus model-based inference.

Concept	This Lesson (Introductory)	Advanced Extension
Sampling Design	SRS with equal selection probabilities	Unequal-probability designs (PPS sampling), multistage designs with complex weighting
Variance Estimation	Var(X̄) = σ²/n with finite population correction	Linearization (Taylor series), jackknife, and bootstrap variance estimators for complex surveys
Bias Correction	Prevent bias through proper random selection	Post-stratification, raking, inverse propensity weighting to adjust for non-response and selection on observables
Randomization	Random sampling → external validity (generalizability)	Random assignment → internal validity (causal inference); experiments combine both
Inference Paradigm	Design-based: randomness comes from the sampling mechanism	Model-based (superpopulation): assumes a probability model for the population, enabling inference from non-random samples under model assumptions

From introductory sampling concepts to advanced statistical methodology

As you advance in statistics, you will encounter situations where perfectly random samples are infeasible—observational health studies, economics data from administrative records, or social media analytics. In these contexts, researchers rely on model-based adjustments to approximate what random sampling would have achieved. Techniques such as propensity score matching and inverse probability weighting attempt to reconstruct the balance that randomization provides naturally. However, these methods rest on untestable assumptions, reinforcing the lesson that prevention of bias through design is always preferable to post-hoc correction.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A television news program asks viewers to call a toll-free number to vote on whether the minimum wage should be raised. Over 50,000 people call in, and 78% say yes. The program reports this as evidence that 'Americans overwhelmingly support a minimum wage increase.' Identify the sampling method used and explain at least two reasons why this conclusion is not supported by the data.

Show answer

This is a voluntary response sample: individuals self-selected into the survey by choosing to call. First, voluntary response bias means that people with strong opinions (particularly those who favor the increase) are more likely to take the time to call, systematically over-representing one viewpoint. Second, the sampling frame implicitly excludes people without access to the broadcast or a telephone, introducing undercoverage. Despite the large sample size (50,000), the bias term in the MSE decomposition remains large and positive, so the 78% figure cannot be generalized to all Americans. A properly conducted random sample of 1,000 adults would yield a far more reliable estimate.

Question 2: BASIC CALCULATION

A population of N = 5,000 college students has a true mean GPA of μ = 3.05 and a population variance of σ² = 0.36. A simple random sample of n = 100 students is drawn without replacement. Calculate the variance of the sample mean, the standard error, and verify that the sample mean is an unbiased estimator of μ.

Show answer

Using the SRS variance formula with the finite population correction: Var(X̄) = (σ²/n) × (1 − n/N) = (0.36/100) × (1 − 100/5000) = 0.0036 × 0.98 = 0.003528. The standard error is SE(X̄) = √0.003528 ≈ 0.0594. Because the sample is a simple random sample, E(X̄) = μ = 3.05, confirming unbiasedness. The bias is zero, so MSE = Var(X̄) = 0.003528. Note that the fpc reduces the variance by 2% compared to sampling with replacement, a modest correction because n/N = 0.02 is small.

Question 3: INTERMEDIATE

A researcher collects data from a convenience sample of n = 500 patients at a single urban hospital to estimate the national prevalence of Type 2 diabetes. The sample proportion is p̂ = 0.18. National registry data indicate the true prevalence is p = 0.11. (a) Calculate the bias and MSE of this estimate, assuming the sample variance of p̂ is approximately p̂(1 − p̂)/n. (b) Determine the sample size needed for an unbiased SRS to achieve the same MSE.

Show answer

(a) Bias = p̂ − p = 0.18 − 0.11 = +0.07. The variance component is Var(p̂) ≈ (0.18 × 0.82)/500 = 0.1476/500 = 0.000295. MSE = 0.000295 + (0.07)² = 0.000295 + 0.0049 = 0.005195. The bias² term accounts for about 94% of the total error. (b) For an unbiased SRS, Bias = 0 so MSE = Var(p̂) = p(1−p)/n. Setting p(1−p)/n = 0.005195 with p = 0.11: (0.11 × 0.89)/n = 0.005195, so n = 0.0979/0.005195 ≈ 19 individuals. An unbiased SRS of just 19 patients matches the total MSE of the biased convenience sample of 500—a striking illustration of bias's cost.

Question 4: APPLIED

A market research firm wants to estimate the proportion of smartphone users aged 18–34 who use a particular app. The firm has two options: (A) Recruit 2,000 respondents through an online advertisement on a tech blog (cost: $1,500), or (B) Commission a stratified random sample of 600 from a probability-based consumer panel, stratified by age group and region (cost: $12,000). The client demands an estimate within ±3 percentage points of the true value at 95% confidence. Advise the firm on which option to choose and justify your recommendation using concepts from this lesson.

Show answer

Option A is a convenience sample drawn from readers of a tech blog, who are likely more tech-savvy than the general population of 18–34-year-olds, creating substantial selection bias. The MSE decomposition tells us that even with n = 2,000, the bias² term dominates, and the ±3pp confidence interval is meaningless because confidence intervals assume unbiased estimation—they quantify random error, not systematic error. Option B, despite being smaller and more expensive, uses a stratified random sample from a probability panel. Stratification by age and region ensures representation of key subgroups, and the probability design supports valid confidence intervals. With n = 600 and p ≈ 0.5 (worst case), the margin of error is approximately 1.96 × √(0.25/600) ≈ ±4.0pp before stratification gains, which can bring it below ±3pp. The firm should choose Option B: paying more for a valid inference framework is far preferable to a cheap estimate of unknown accuracy.

Question 5: CRITICAL THINKING

A data scientist argues: 'In the age of big data, random sampling is obsolete. We can analyze entire populations through administrative databases, web scraping, and sensor data, making sampling unnecessary.' Critically evaluate this claim. Under what conditions might administrative or organic data introduce bias? Can a complete dataset (census) still produce biased estimates of a target quantity? Construct a specific scenario to support your argument.

Show answer

The claim confuses dataset size with dataset quality and conflates the observed dataset with the target population. First, administrative and organic data are generated by non-random processes: hospital records capture only those who seek care (missing undiagnosed individuals), social media data reflect only platform users (skewed by age, socioeconomics, and geography), and sensor data cover only instrumented locations. These are effectively massive convenience samples with unknown selection probabilities. Second, even a nominally complete dataset can yield biased estimates if the quantity of interest differs from what the data literally measure. For example, a company analyzing 100% of its customer service calls to estimate 'customer satisfaction' only captures customers who called—the silent majority who neither complain nor praise are invisible, and those who call are disproportionately dissatisfied. The 'census' of calls is a biased sample of all customer experiences. The MSE framework shows that no amount of additional call data reduces this bias. Random sampling remains essential precisely because it provides the probabilistic foundation that makes formal inference valid; big data supplements but does not replace this foundation.

SUMMARY

Lesson Summary

Summary

This lesson established that random sampling—in which every member of the population has a known, non-zero probability of selection—is the foundation of trustworthy statistical inference. We traced the historical evolution from Kiaer's representative method through the Literary Digest failure to Neyman's formal probability framework, showing that sampling bias is a systematic error that does not diminish with larger samples. The MSE decomposition (MSE = Variance + Bias²) formalizes why: increasing n reduces the variance component but leaves the bias term unchanged, so a small unbiased sample routinely outperforms a large biased one.

We classified sampling methods into probability designs (simple random, stratified, cluster, systematic) and non-probability designs (convenience, voluntary response, quota), and identified key bias types including selection bias, non-response bias, response bias, and undercoverage. The worked example demonstrated that a voluntary-response sample of 800 with a bias of +4.3 hours had an MSE 255 times larger than an SRS of just 400. The core principle to carry forward: invest in the quality of the sampling design, not merely the quantity of observations.