Sampling Bias — I can identify sampling bias and propose...

SECTION 1

Historical Context & Motivation

Every time a poll predicts an election outcome or a medical study announces a breakthrough, the conclusions depend on who was included in the sample. If the sample doesn't reflect the full population, the results can be wildly misleading. This problem — sampling bias — has haunted researchers, journalists, and governments for over a century. Understanding where bias creeps in is the first step toward collecting data you can actually trust.

1936

The Literary Digest Disaster

The magazine mailed 10 million questionnaires to predict the U.S. presidential election, but respondents were drawn from telephone directories and car registrations — skewing the sample toward wealthier voters. The poll predicted Alf Landon would win; Franklin Roosevelt won in a landslide.

1948

"Dewey Defeats Truman"

Pollsters stopped sampling too early and used non-random quota sampling, leading newspapers to print a premature headline that became one of the most famous errors in polling history.

1970s

Rise of Random Sampling Standards

Statisticians formalized probability-based sampling methods, establishing best practices that made surveys far more reliable across social science and public health research.

2016

Online Polling Challenges

Several major polls underestimated voter turnout among certain demographics in the U.S. presidential election, reigniting debates about nonresponse bias and the limits of modern survey methods.

These historical blunders all share a common thread: the people studied were systematically different from the broader population. The central question this lesson addresses is: How do we spot that mismatch, and what can we do to fix it?

SECTION 2

Core Principles & Definitions

Before you can fix sampling bias, you need precise language for what's going wrong. A population is the entire group you want to learn about — every student at your school, every voter in a state, or every smartphone on the market. A sample is the smaller subset you actually collect data from. Sampling bias occurs when certain members of the population are systematically more or less likely to be included in the sample than others, causing the sample to misrepresent the population.

Selection Bias

Occurs when the method used to choose participants favors some groups over others. Example: surveying only students who eat in the cafeteria ignores those who leave campus for lunch.

Nonresponse Bias

Arises when people who don't respond to a survey differ meaningfully from those who do. If busy students skip a questionnaire, their opinions are missing from the results.

Voluntary Response Bias

Happens when individuals self-select into a study. People with strong opinions — especially negative ones — are more likely to volunteer, skewing results away from moderate views.

Undercoverage Bias

Results when parts of the population are left out of the sampling frame entirely. A phone survey that calls only landlines systematically excludes younger adults who rely on cell phones.

Convenience Sampling Bias

Occurs when a researcher samples whoever is easiest to reach — friends, nearby classmates, or online followers — rather than a cross-section of the full population.

✦ KEY TAKEAWAY

Think of sampling bias like taste-testing a pot of soup by scooping only from the top. If the spices settled to the bottom, your taste test tells you nothing about the whole pot. A good sample is like stirring the pot first — it mixes every ingredient so that each spoonful truly represents the batch.

SECTION 3

Visual Explanation

The population on the left contains five distinct groups (shown by color). The biased sample (top right, dashed border) over-represents the purple group, while the representative sample (bottom right) includes members from every group in roughly the same proportions as the population.

The diagram above illustrates the core problem. When a sampling method systematically favors one segment of the population — whether by location, availability, or willingness to participate — the resulting data paints a distorted picture. A representative sample doesn't need to be enormous; it just needs to give every member of the population a known, nonzero chance of being selected. That principle is the foundation of probability sampling.

SECTION 4

Mathematical Framework

While sampling bias is often discussed qualitatively, we can quantify it. The bias of a sample statistic is the difference between the expected value of that statistic and the true population parameter. If we repeatedly drew samples using the same method, bias measures how far off-center our estimates would land on average.

BIAS OF AN ESTIMATOR

Bias(θ̂) = E(θ̂) − θ

Where θ̂ (theta-hat) is the sample statistic (e.g., sample mean), E(θ̂) is the expected value of that statistic across many samples, and θ (theta) is the true population parameter. When bias = 0, the estimator is called unbiased.

MARGIN OF ERROR (95% CONFIDENCE)

ME = 1.96 × √(p̂(1 − p̂) / n)

Where p̂ is the sample proportion, n is the sample size, and 1.96 corresponds to a 95% confidence level. This formula assumes a random sample. If bias exists, the true error exceeds this margin because systematic error is added on top of random error.

TOTAL ERROR

Total Error = Bias + Random Sampling Error

Increasing sample size reduces random sampling error, but it does not reduce bias. Even a sample of one million people can be heavily biased if the selection process is flawed — as the 1936 Literary Digest poll demonstrated.

⚠️ Why Size Alone Doesn't Fix Bias

A common misconception is that a bigger sample always means better data. In reality, a well-designed random sample of 1,000 people will outperform a biased sample of 1,000,000. The Literary Digest polled 2.4 million people and got the election wrong, while George Gallup polled only 50,000 using better methods and got it right. Method matters more than size.

SECTION 5

Types of Sampling Bias — A Detailed Breakdown

Recognizing the specific type of bias in a study is crucial because each type requires a different fix. The diagram below maps common bias types to their causes and the sampling stage where they occur.

This flowchart traces the four stages of a sampling process — defining the population, creating a sampling frame, selecting participants, and collecting responses. Each stage introduces a different type of bias (shown by the colored warning boxes). Bias at any stage compromises the final conclusions.

Summary of common sampling bias types, examples, and remedies

Bias Type	What Happens	Real-World Example	How to Fix It
Undercoverage	The sampling frame misses part of the population.	Surveying only landline users leaves out cell-only households.	Expand the frame (e.g., include cell phones); use multiple contact methods.
Selection Bias	A non-random process systematically includes/excludes certain groups.	Choosing every 5th person in a list sorted by GPA.	Use simple random sampling (SRS) or stratified random sampling.
Nonresponse	Selected individuals refuse or fail to respond.	Mail surveys often have < 30% response rates; non-respondents may differ.	Follow up with non-respondents; keep surveys short; offer incentives.
Voluntary Response	People choose to participate, usually those with extreme views.	Online product reviews — angry customers post more often.	Recruit participants randomly rather than waiting for volunteers.
Convenience	Researcher samples whoever is nearby or easy to access.	Asking only your friend group about school policy changes.	Use random selection from a complete list of the population.

SECTION 6

Worked Example

Let's walk through a realistic scenario step by step to identify bias and propose a better design.

Is the School Cafeteria Popular?

Step 1 — Read the ScenarioA student council member wants to know what percentage of the school's 1,200 students are satisfied with the cafeteria food. She posts a survey link on the school's Instagram story and collects 95 responses. Of those, 78 say they are dissatisfied. She concludes: "82% of students are unhappy with the cafeteria."

Step 2 — Identify the Population and SampleThe population is all 1,200 students. The sample is the 95 students who chose to respond via Instagram.

Population: 1,200 students. Sample: 95 self-selected respondents.

Step 3 — Identify the Type(s) of BiasVoluntary response bias: Only students who felt strongly enough clicked the link — those who are indifferent probably scrolled past. Undercoverage bias: Students who don't follow the school's Instagram account were excluded entirely. Convenience bias: The survey was posted wherever it was easiest for the council member to reach people, not from a complete student roster.

Three bias types identified: voluntary response, undercoverage, and convenience.

Step 4 — Explain Why the Conclusion Is UnreliableBecause dissatisfied students are more likely to respond to a voluntary survey, the 82% figure almost certainly overestimates actual dissatisfaction. Meanwhile, students without Instagram had zero chance of being selected. The sample of 95 is not representative of the 1,200-student population, so the conclusion cannot be generalized.

Step 5 — Propose an Improved DesignObtain a complete list of all 1,200 students from the administration. Use a simple random sample — assign each student a number and use a random number generator to select, say, 150 students. Distribute the survey directly (e.g., in homeroom) and follow up with non-respondents to minimize nonresponse bias. This gives every student an equal probability of being selected.

Improved design: SRS of 150 students from the full roster, distributed in homeroom with follow-ups.

SECTION 7

Comparing Sampling Methods — Strengths & Limitations

Not every study can use a textbook random sample. Budget, time, and access all play a role. The table below compares the most common methods so you can weigh tradeoffs and choose the best approach for a given situation.

Comparison of common sampling methods

Sampling Method	How It Works	Strengths	Limitations
Simple Random (SRS)	Every member of the population has an equal chance of being selected (e.g., drawing names from a hat).	Eliminates selection bias; results are generalizable; easy to analyze.	Requires a complete list of the population; may be impractical for large or dispersed groups.
Stratified Random	Divide the population into subgroups (strata) and randomly sample within each stratum.	Ensures representation of key subgroups; can increase precision over SRS.	Requires prior knowledge of population characteristics; more complex to set up.
Cluster	Divide the population into clusters (e.g., classrooms), randomly select some clusters, and survey everyone in the chosen clusters.	Cost-effective when the population is geographically spread out.	Higher sampling error if clusters are not internally diverse.
Systematic	Select every k-th individual from a list after a random starting point.	Simple to implement; works well when lists are in random order.	Can introduce bias if the list has a hidden periodic pattern.
Convenience	Sample whoever is easiest to reach — no randomization.	Fast, cheap, and easy; useful for pilot studies.	High risk of bias; results cannot be generalized to the population.

✦ KEY TAKEAWAY

Choosing a sampling method is like choosing how to pick players for a pickup basketball game. Letting people volunteer (voluntary response) means you get only the most eager players. Picking your friends (convenience) doesn't test the school's talent pool. Randomly drawing names from the entire student body — even if it's slower — gives you the most fair and representative picture of everyone's skill level.

SECTION 8

Connection to Inferential Statistics

Everything you learn about confidence intervals and hypothesis testing later in statistics rests on one assumption: the sample was collected properly. Formulas for margin of error, p-values, and standard error all assume random sampling. When that assumption is violated by bias, the math still runs — but the outputs are meaningless. This is why understanding sampling bias now sets you up for success in more advanced coursework.

How sampling concepts connect to advanced statistics

Concept	What You Know Now	Where It Leads
Random Sampling	Every member has a known, nonzero chance of selection.	Foundation for the Central Limit Theorem, which allows us to estimate population parameters from sample statistics.
Bias	Systematic error that doesn't shrink with larger samples.	In AP Statistics, you will learn to account for potential bias sources when designing experiments and interpreting published studies.
Stratified Sampling	Dividing the population into subgroups before sampling.	Leads to post-stratification weighting in professional polling, where raw data is adjusted to match known population demographics.
Margin of Error	A measure of random sampling error at a given confidence level.	Extends to confidence intervals for means, proportions, and differences — but only if the sample is unbiased.

In college-level statistics and data science, you will encounter techniques like weighting adjustments and propensity score matching that attempt to correct for bias after data collection. However, no statistical trick fully replaces a well-designed sample. Building strong habits around sampling design now will serve you in any career that involves data — from medicine and engineering to marketing and public policy.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

A school principal wants to know whether students support a later start time. She asks for opinions at the next PTA meeting (attended mostly by parents). Identify the type(s) of sampling bias in this approach and explain why the results may not reflect student opinions.

PROBLEM 2 — BASIC CALCULATION

A researcher surveys 400 randomly selected adults and finds that 55% support a new park. Calculate the margin of error at a 95% confidence level using ME = 1.96 × √(p̂(1 − p̂) / n). Then explain what would happen to this margin of error if the sample were biased instead of random.

PROBLEM 3 — INTERMEDIATE

A tech company emails a customer satisfaction survey to all 10,000 customers. Only 600 respond, and 85% of respondents say they are satisfied. The company claims "85% customer satisfaction." Identify the bias, explain its likely direction (does it overestimate or underestimate satisfaction?), and propose two specific improvements.

PROBLEM 4 — APPLIED

You are designing a study to determine how many hours per week students at your 800-student school spend on social media. Describe a complete sampling plan: specify the population, sampling method, sample size, and how you would distribute the survey. Explain how your plan addresses at least two types of bias.

PROBLEM 5 — CRITICAL THINKING

A national news outlet reports: "90% of Americans support Policy X," based on a poll of 50,000 people who called a toll-free number shown on TV. A rival outlet publishes a random phone survey of 1,200 adults showing only 52% support. Which result is more trustworthy, and why? In your answer, discuss how sample size, sampling method, and bias interact.

SUMMARY

Lesson Summary

Sampling bias is systematic error that arises when certain members of a population are more or less likely to be included in a sample. The five major types — selection bias, undercoverage bias, nonresponse bias, voluntary response bias, and convenience bias — each enter at a different stage of the sampling process and require targeted fixes.

The most important takeaway is that increasing sample size does not eliminate bias; only improving the sampling method can do that. Probability-based methods such as simple random sampling and stratified random sampling give every member of the population a known chance of being selected, forming the foundation for reliable statistical inference. When you evaluate any study — in class, in the news, or in your own projects — always ask: Who was sampled? Who was left out? And could the results look different if the sample were collected differently?