Potential Problems with Sampling - AP Statistics

SECTION 1

Historical Context & Motivation

Statistical sampling has a storied history of spectacular failures that illuminate why careful methodology matters. One of the most famous disasters in survey history occurred in 1936, when The Literary Digest magazine mailed out over ten million questionnaires to predict the outcome of the presidential election between Franklin D. Roosevelt and Alf Landon. The magazine predicted a Landon landslide, yet Roosevelt won by one of the largest margins in American electoral history. The root cause was not a small sample—indeed, over two million responses were collected—but rather a deeply flawed sampling frame that overrepresented affluent Americans who were more likely to favor the Republican candidate. This episode powerfully demonstrates that sample size alone does not guarantee accuracy; the method by which the sample is drawn is equally, if not more, important.

1936 • The Literary Digest Polling Disaster

A massive voluntary-response survey of over 2 million people produces a spectacularly wrong election prediction, demonstrating the devastating effects of undercoverage bias and nonresponse bias.

1948 • Dewey Defeats Truman

Pollsters using quota sampling predicted Dewey's victory, but the non-random nature of the quotas introduced systematic bias, prompting the profession to adopt probability sampling methods.

1970s • Rise of Telephone Surveys

Random digit dialing becomes standard practice, but researchers soon discover that households without telephones—often lower-income—are systematically excluded, creating undercoverage.

2016 • Modern Polling Challenges

Widespread election polling errors highlight persistent problems with nonresponse bias and the increasing difficulty of reaching representative samples in the digital age.

These historical episodes raise a fundamental question in statistics: even when we intend to collect data that faithfully represents a population, what can go wrong between our sampling plan and the conclusions we draw? Understanding the sources of sampling problems is essential because flawed data collection undermines every subsequent analysis, no matter how sophisticated the statistical techniques applied. In this lesson, we systematically examine the types of bias and error that can compromise a sample and learn to identify, diagnose, and—where possible—mitigate these threats.

SECTION 2

Core Principles & Definitions

Before diagnosing specific problems, it is critical to distinguish between two broad categories of error that affect any survey or study. Sampling error refers to the natural variability that arises because we observe only a subset of the population rather than conducting a census; it is expected, quantifiable, and decreases as sample size increases. Non-sampling error, by contrast, encompasses all other mistakes—faulty sampling frames, biased wording, nonresponse, measurement problems—and these errors do not shrink with larger samples. In fact, enlarging a biased sample merely produces a more precise wrong answer. The AP Statistics exam places heavy emphasis on identifying and explaining non-sampling errors, and these form the core of what we mean by "potential problems with sampling."

Bias

A systematic tendency to overestimate or underestimate a population parameter. Unlike random error, bias pushes results consistently in one direction and cannot be reduced by increasing sample size.

Undercoverage

Occurs when some members of the population are left out of the sampling frame and therefore have no chance of being selected. If the excluded group differs systematically from the included group, the results are biased.

Nonresponse Bias

Arises when selected individuals fail to participate and their characteristics differ from those who do respond. High nonresponse rates are problematic only when nonrespondents are systematically different.

Voluntary Response Bias

When individuals self-select into the sample, those with strong opinions (often negative) are overrepresented. Online reviews and call-in polls are classic examples.

Response Bias

Results from factors that cause respondents to give inaccurate answers—leading questions, social desirability, poor recall, or interviewer influence all fall under this umbrella.

✦

KEY TAKEAWAY

Think of sampling like drawing a map of an unfamiliar territory from a helicopter. Sampling error is like the natural fuzziness of the photograph—you can improve it by zooming in (larger sample). Bias is like a tilted camera that consistently crops out the left side of the landscape—no amount of better resolution fixes the missing region. The only way to correct bias is to straighten the camera itself, which means redesigning the sampling method.

SECTION 3

Visualizing Sampling Problems

The following diagram illustrates how different types of bias distort the relationship between a population and the sample that is actually analyzed. In an ideal scenario, the sampling frame perfectly matches the population, every selected individual responds, and every response accurately reflects reality. Each type of bias represents a breakdown at a different stage of this pipeline, and recognizing where the breakdown occurs is the key to diagnosing and naming the bias correctly on the AP exam.

The diagram traces the path from the full population to the data actually collected. Undercoverage occurs between the population and the sampling frame. Voluntary response bias distorts who enters the sample. Nonresponse bias reduces who actually participates, and response bias corrupts the answers themselves.

Notice that the pipeline has multiple stages, and bias can enter at any one of them—or at several simultaneously. A survey might suffer from both undercoverage (the sampling frame excludes certain demographics) and nonresponse bias (those who are contacted disproportionately refuse to participate). When answering AP exam questions, you should identify the specific stage at which bias enters and name it precisely; saying a sample is simply "biased" without specifying the type and direction will not earn full credit.

SECTION 4

How Bias Distorts Estimation

Although many sampling problems are conceptual rather than computational, it is useful to formalize how bias and variability jointly determine the accuracy of a sample statistic. The total survey error framework decomposes the difference between a sample statistic and the true population parameter into systematic and random components. Understanding this decomposition clarifies why increasing sample size does not remedy bias.

MEAN SQUARED ERROR DECOMPOSITION

MSE(θ̂) = [Bias(θ̂)]² + Var(θ̂)

Where θ̂ is the sample statistic, Bias(θ̂) = E(θ̂) − θ is the systematic deviation from the true parameter θ, and Var(θ̂) is the sampling variability. Increasing n reduces Var(θ̂) but does not affect Bias(θ̂).

MARGIN OF ERROR (FOR PROPORTIONS)

ME = z* × √(p̂(1 − p̂) / n)

The margin of error quantifies only random sampling variability. It does NOT account for bias. A survey with a ±3% margin of error can still be off by 15% or more if systematic bias is present.

This distinction is crucial for interpreting confidence intervals on the AP exam. When a problem states that a 95% confidence interval for a proportion is (0.42, 0.48), the interval accounts for sampling variability but implicitly assumes that the sample was collected without systematic bias. If the sampling method was flawed—say, a voluntary response internet poll—then the true parameter might lie well outside the reported interval, regardless of its width. The confidence level of 95% applies to the repeated-sampling probability of capturing θ only when the sampling procedure is unbiased.

💡

AP EXAM TIP

Free-response questions frequently ask you to "describe a potential problem with the sampling method" and "explain how it could affect the results." A complete answer must (1) name the specific type of bias, (2) identify the direction of the bias (overestimate or underestimate), and (3) explain the mechanism. For example: "This voluntary response survey is likely to suffer from voluntary response bias because people with strong negative opinions are more likely to respond, which would overestimate the proportion of dissatisfied customers."

SECTION 5

Detailed Classification of Sampling Problems

The AP Statistics curriculum identifies several distinct categories of problems that can compromise a sample. While they share the common feature of introducing systematic error, each has a unique mechanism and requires a different remedy. The diagram below provides a classification tree that can help you quickly identify which type of bias is present in an exam scenario.

The classification tree separates total survey error into sampling error (random, reducible by increasing n) and non-sampling error (systematic, requiring methodological fixes). The three major branches of non-sampling error—selection bias, nonresponse bias, and response bias—each have distinct causes and remedies.

Type of Bias	When It Occurs	Classic Example	Likely Direction
Undercoverage	The sampling frame omits part of the population	A telephone survey that excludes cell-phone-only households, underrepresenting younger adults	Depends on how the excluded group differs from those included
Voluntary Response	Individuals self-select into the sample	An online restaurant review site where only customers with extreme experiences bother to post	Overrepresents strong (often negative) opinions
Nonresponse	Selected individuals refuse or fail to respond	A mailed health survey where sicker individuals are too ill to complete and return it	Depends on how nonrespondents differ from respondents
Response (Wording)	Question wording pushes answers in a particular direction	"Don't you agree that taxes are too high?" leads respondents toward agreement	Toward the implied "correct" answer
Response (Social Desirability)	Respondents give socially acceptable rather than truthful answers	A face-to-face survey about drug use where respondents underreport illegal behavior	Overestimates "good" behaviors, underestimates "bad" ones

Summary of the five major types of bias tested on the AP Statistics exam

A neighborhood survey shows homes, invitation routes and returned responses. A survey can lose people before selection or after invitation. Keeping the target population, sampling frame, selected sample and respondents separate reveals where each problem arises.

A 24-ID population narrows to a 20-ID frame, eight selected invitations and six responses. Suppose the target population has 24 IDs. The frame omits IDs 21–24; eight listed IDs are selected, and six respond. Frame omissions are undercoverage; selected people who do not respond are nonresponse. Bias direction depends on how missing responses differ.

Two versions of a survey question contrast neutral phrasing with wording that encourages a particular response. A respondent can still be influenced by the wording of a question. Compare the neutral and leading versions by identifying the assumption each asks the respondent to accept.

SECTION 6

Worked Example: Identifying and Explaining Bias

Consider the following scenario, which is typical of what you would encounter on an AP Statistics free-response question: A school principal wants to estimate the proportion of students who have been bullied in the past year. She places a box in the main office with slips of paper for students to voluntarily report whether they have been bullied. After one month, she collects the slips and finds that 72% of respondents report being bullied. She concludes that bullying is a severe problem at her school. Identify all potential problems with this sampling method and explain how each could affect the conclusion.

Diagnosing Bias in a School Bullying Survey

1
Step 1 — Identify the Sampling Method

The principal placed a box in the office and allowed students to voluntarily deposit slips. No random selection mechanism was used. This is a voluntary response sample, not a probability sample.

Sampling method: Voluntary response
2
Step 2 — Identify Voluntary Response Bias

Students who have been bullied are more likely to feel strongly about the issue and are therefore more motivated to take the time to fill out a slip and drop it in the box. Students who have not been bullied have little reason to participate. As a result, the sample likely overrepresents bullied students.

Direction: Overestimates the true proportion of bullied students
3
Step 3 — Identify Potential Undercoverage

The box is located only in the main office. Students who rarely visit the office—perhaps those in certain grades, extracurricular programs, or those who avoid the office for personal reasons—have effectively zero probability of being included. This is undercoverage, and if the excluded students have different bullying experiences, the result is biased.

Additional bias: Undercoverage of students who do not visit the main office
4
Step 4 — Identify Potential Response Bias

The slips are not anonymous in a guaranteed way—students might worry that office staff could identify them. If students fear retaliation for reporting bullying or feel embarrassed, some may underreport, introducing response bias due to social desirability. This could work against the voluntary response bias, but the net effect is uncertain.

Possible response bias: Underreporting due to fear of identification
5
Step 5 — State the Conclusion

The principal's estimate of 72% is almost certainly an overestimate of the true proportion of bullied students because voluntary response bias is the dominant problem here. The result cannot be generalized to the entire student body because the sample is not representative. A better approach would be to use a simple random sample of students selected from the school enrollment list, with an anonymous questionnaire administered during class to maximize the response rate.

72% likely overestimates the true bullying rate; no valid generalization is possible

SECTION 7

Remedies and Mitigation Strategies

Knowing how to identify bias is essential, but a strong understanding of sampling problems also includes knowledge of how to prevent or mitigate them. The table below summarizes the primary remedies for each type of bias. On the AP exam, you may be asked not only to identify what went wrong but to propose an improved sampling design.

Problem	Primary Remedy	Why It Works
Undercoverage	Use a comprehensive, up-to-date sampling frame; consider stratification to ensure all subgroups are represented	A complete frame ensures every member of the population has a known, nonzero probability of selection
Voluntary Response	Replace with a probability sampling method (SRS, stratified, cluster, or systematic)	Probability sampling removes self-selection, giving the researcher control over who is included
Nonresponse	Follow up persistently with nonrespondents; offer incentives; keep surveys short and convenient	Higher response rates reduce the gap between respondents and nonrespondents, minimizing potential bias
Response Bias (Wording)	Use neutral, balanced question wording; pilot-test the instrument; avoid leading or loaded language	Neutral wording reduces the likelihood that the question itself pushes respondents toward a particular answer
Response Bias (Social Desirability)	Guarantee anonymity; use self-administered questionnaires rather than face-to-face interviews for sensitive topics	Anonymity removes the social pressure that causes respondents to shade their answers toward what is deemed acceptable

Remedies for common sources of bias in sampling

✦

KEY TAKEAWAY

Think of bias remedies like calibrating scientific instruments in a research laboratory. A spectrometer that is systematically reading wavelengths 5 nm too high will produce consistently wrong results no matter how many measurements you take—the only fix is to recalibrate the instrument. Similarly, no amount of data collection fixes a biased sampling method; you must fix the method itself. Probability sampling is the gold standard for calibration in the world of surveys—it is the single most important tool for ensuring that sample results can be trusted to generalize to the population.

SECTION 8

Connection to Inference and Advanced Topics

Sampling problems do not exist in isolation; they directly affect the validity of every inferential procedure you will learn in AP Statistics. When you construct a confidence interval or perform a hypothesis test, the formulas assume that the data were collected using a method that gives every individual (or at least identifiable groups) a known probability of selection. The conditions for inference—randomness, independence, normality—begin with the randomness condition, which is impossible to satisfy if the sample was collected via a flawed method. The table below connects the sampling problems covered in this lesson to the inferential concepts you will encounter later in the course.

Sampling Problem	Effect on Inference
Undercoverage	The confidence interval or test result applies only to the covered population, not the target population. Generalization is invalid.
Voluntary Response	The randomness condition is violated. No confidence interval or p-value is meaningful because the sample does not represent any well-defined population.
Nonresponse Bias	The effective sample is no longer random even if the original selection was. The margin of error understates the true uncertainty.
Response Bias	The data values themselves are distorted. Even a perfectly random sample produces biased estimates if the measurements are systematically off.

How sampling problems propagate into later inferential procedures

In more advanced statistics courses and in professional survey methodology, researchers use techniques such as post-stratification weighting, propensity score adjustment, and multiple imputation to partially correct for known biases after data collection. However, these methods require strong assumptions and are imperfect substitutes for good sampling design. The fundamental lesson remains: the best defense against bias is a well-designed probability sampling plan executed with rigorous follow-up procedures to minimize nonresponse. Prevention is always preferable to correction.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A researcher mails a survey to 1,000 randomly selected households to estimate the average number of hours residents spend exercising per week. Only 230 surveys are returned. The researcher reports a 95% confidence interval based on these 230 responses. Which of the following is the most serious concern about this study?

A. The sample size of 230 is too small to construct a valid confidence interval.
B. The margin of error will be larger than if all 1,000 had responded.
C. People who exercise more may be more likely to respond, making the results systematically too high.
D. A census of all households in the area should have been conducted instead.
E. The original 1,000 households were not enough to represent the population.

Show answer (C)

C is correct. With a 77% nonresponse rate, the most serious concern is nonresponse bias. People who are health-conscious and exercise regularly may be more motivated to respond to an exercise survey, systematically inflating the estimate. Choice A is incorrect because 230 is generally a sufficient sample size for a confidence interval. Choice B is true but describes increased variability (a less serious issue than systematic bias). Choice D is impractical and not a statistical criticism of the method. Choice E is incorrect because 1,000 randomly selected households is typically adequate for population-level inference.

Question 2: BASIC CALCULATION

A school district wants to survey parents about satisfaction with the new cafeteria menu. They post the survey on the school's social media page and ask interested parents to complete it. Of the 500 parents who respond, 78% express dissatisfaction. Which type of bias is MOST clearly present?

A. Response bias due to question wording
B. Undercoverage bias because not all parents use social media
C. Sampling error due to the sample being too small
D. Voluntary response bias because parents self-selected into the survey
E. Nonresponse bias because many parents chose not to participate

Show answer (D)

D is correct. The defining feature of this survey is that parents chose to respond—no one was randomly selected. This is a voluntary response sample, and people with strong negative opinions about the new menu are disproportionately likely to take the time to complete it, inflating the dissatisfaction rate. Choice A is incorrect because there is no information about biased question wording. Choice B identifies a real secondary concern (parents without social media are excluded), but undercoverage is not the primary problem since even among social media users, self-selection is the dominant bias mechanism. Choice C is wrong because 500 is not a small sample and the issue is bias, not sample size. Choice E confuses the mechanism: nonresponse bias occurs when people selected for a sample fail to respond, but here nobody was selected—people volunteered.

Question 3: INTERMEDIATE

A political polling firm conducts a phone survey by randomly dialing landline numbers. In a region where 40% of adults have only cell phones and no landline, the poll estimates that 55% of adults support a particular candidate. If cell-phone-only adults support the candidate at a rate of 65%, in which direction is the poll likely biased?

A. The poll likely overestimates support for the candidate.
B. The poll likely underestimates support for the candidate.
C. The poll is unbiased because the dialing was random.
D. The direction of the bias cannot be determined from the information given.
E. The bias is due to nonresponse rather than undercoverage.

Show answer (B)

B is correct. This is a case of undercoverage: 40% of adults are excluded because they only have cell phones. Since the excluded group supports the candidate at 65% (higher than the 55% measured among landline users), the poll systematically underestimates true support. We can verify: the true population support would be approximately 0.60 × 0.55 + 0.40 × 0.65 = 0.33 + 0.26 = 0.59, or 59%, which is higher than the poll's 55%. Choice A has the direction reversed. Choice C is wrong because random dialing of landlines does not make the sample representative of all adults—only of the landline-owning population. Choice D is incorrect because we are given enough information to determine the direction. Choice E misidentifies the type of bias; the problem is not that selected people refused to respond, but that cell-phone-only adults were never in the sampling frame.

Question 4: APPLIED

4 pts

A university wants to estimate the proportion of its 20,000 students who support extending library hours to midnight. The administration sends an email survey to all students, and 1,200 students respond. Of those, 84% favor the extension. (a) Identify a type of bias that is likely present in this survey and explain how it arises. (1 point) (b) Explain the likely direction of the bias—that is, whether the 84% is likely an overestimate or underestimate of the true proportion. Justify your answer. (1 point) (c) If the administration had instead selected 400 students using a simple random sample and obtained an 80% response rate, would this method eliminate all potential for bias? Explain. (1 point) (d) Describe one additional modification to the survey design in part (c) that could further reduce bias. Explain why it would help. (1 point)

Show answer

Rubric (4 points): Part (a) — 1 point: Voluntary response bias is likely present. The email was sent to all 20,000 students, but only 1,200 (6%) chose to respond. Students who feel strongly about library hours—particularly those who frequently use the library late at night—are more likely to take the time to complete the survey. Students who rarely use the library or who are indifferent have less motivation to respond. Award 1 point for correctly identifying voluntary response bias AND explaining the mechanism by which it arises (self-selection of those with strong opinions). Part (b) — 1 point: The 84% is likely an overestimate of the true proportion of all students who support extending hours. Students who would personally benefit from longer library hours have a stronger incentive to respond to the survey, while students who are indifferent to or would not use extended hours are underrepresented among respondents. Because supporters are overrepresented in the sample, the sample proportion is pushed above the true population proportion. Award 1 point for stating the correct direction (overestimate) with a plausible justification that connects the bias mechanism to the direction of the effect. Part (c) — 1 point: No, this method would not eliminate all potential for bias. While using a simple random sample removes the voluntary response (self-selection) problem from the original design, the 20% who do not respond (80 of the 400 selected students) could differ systematically from those who do respond. For example, very busy students who do not check email regularly might be less likely to respond, and these students might hold different opinions about library hours than those who did respond. Therefore, nonresponse bias could still be present even with a properly drawn SRS. Award 1 point for correctly stating that bias is not fully eliminated AND identifying nonresponse bias as the remaining concern with a brief explanation of why nonrespondents might differ from respondents. Part (d) — 1 point: Acceptable answers include any one of the following: (1) Following up with nonrespondents through multiple contact attempts using different methods (phone call, text message, or in-person visit) to increase the response rate, thereby reducing the potential for nonresponse bias because fewer selected students are left unaccounted for. (2) Offering a small incentive (such as entry into a gift card drawing) to encourage all selected students to participate, which reduces nonresponse bias by motivating even indifferent students to complete the survey. (3) Administering the survey in person during randomly selected class sections to ensure higher participation rates, which reduces nonresponse bias because students are present and can complete the survey immediately rather than needing to act on an email. Award 1 point for describing a specific, plausible modification to the design in part (c) AND explaining why it would help reduce bias.

Question 5: CRITICAL THINKING

4 pts

A national news network wants to estimate public opinion on a proposed environmental regulation. They use two methods simultaneously: Method A: A random digit dialing telephone survey of 1,500 adults, achieving a 35% response rate (525 respondents). Method B: An online poll posted on the network's website, generating 12,000 responses. Method A finds 52% support. Method B finds 71% support. (a) For each method, identify the most serious type of bias and explain the mechanism by which it arises. (2 points) (b) A commentator argues that Method B is more reliable because it has a much larger sample. Explain why this reasoning is flawed, referencing the distinction between sampling error and bias. (1 point) (c) Suppose the network wants to produce the most accurate estimate possible using one of these methods. Which method would you recommend, and what specific steps could be taken to improve it? Justify your recommendation. (1 point)

Show answer

Rubric (4 points): Part (a) — 2 points (1 point per method): Method A: Nonresponse bias. Although the sampling method (random digit dialing) is sound, only 35% of selected adults responded. People who care more about environmental issues—or who are more politically engaged—may be more likely to answer the phone and complete the survey. If respondents differ systematically from nonrespondents in their views on the regulation, the result is biased. Award 1 point for identifying nonresponse bias and explaining the mechanism. Method B: Voluntary response bias. The online poll is open to anyone who visits the website and chooses to respond. Viewers of this particular network may already lean in a certain political direction, and among those viewers, the ones with the strongest opinions about environmental regulation are most motivated to click through and vote. This self-selection produces a sample that overrepresents extreme views and the network's audience, not the general public. Award 1 point for identifying voluntary response bias and explaining the mechanism. Part (b) — 1 point: The commentator confuses sample size with sample quality. Larger samples reduce sampling error (random variability), making the estimate more precise, but they do not reduce bias (systematic error). Method B's 12,000 responses produce a very precise estimate—but it is a precise estimate of the wrong quantity because the sample is not representative of the general public. A biased sampling method with n = 12,000 is less trustworthy than a properly randomized method with n = 525. This is exactly the lesson of the 1936 Literary Digest poll. Award 1 point for clearly distinguishing between sampling error and bias and explaining why large n does not fix bias. Part (c) — 1 point: Method A is preferable because it begins with a probability sampling method (random digit dialing), which provides a principled basis for generalization. To improve Method A, the network should: (1) make multiple callback attempts at different times of day and different days of the week to reach people who were initially unavailable; (2) offer modest incentives for participation; (3) keep the survey brief to reduce respondent burden. These steps would increase the response rate and reduce the potential for nonresponse bias, making the results more representative of the adult population. Method B cannot be meaningfully improved because its fundamental flaw—self-selection—is inherent to the design and cannot be corrected after the fact. Award 1 point for recommending Method A with a clear justification and at least one specific improvement strategy.

SUMMARY

Lesson Summary

Summary

Sampling problems fall into two broad categories: sampling error, which is the natural random variability inherent in any sample and decreases with larger sample sizes, and non-sampling error (bias), which is systematic and cannot be reduced by increasing the sample size. The major types of bias include undercoverage (parts of the population are excluded from the sampling frame), voluntary response bias (individuals self-select into the sample, overrepresenting those with strong opinions), nonresponse bias (selected individuals fail to participate and differ from those who do), and response bias (respondents give inaccurate answers due to leading questions, social desirability, or interviewer effects).

For the AP Statistics exam, always remember the three-part framework for addressing bias: (1) name the type of bias, (2) explain the mechanism by which it arises, and (3) state the direction (overestimate or underestimate). The gold standard remedy is probability sampling—simple random samples, stratified random samples, and cluster samples—combined with vigorous follow-up to minimize nonresponse. Remember that the margin of error reported with a confidence interval quantifies only sampling variability, not bias; a biased sample can produce a confidence interval that entirely misses the true parameter.