Potential Problems
with Sampling
Understanding bias, variability, and other pitfalls that can make your sample misleading about the population.
Why Sampling Matters
In statistics, we almost never study an entire population. Instead, we collect data from a sample: a subset chosen to represent the whole. The conclusions we draw are only as good as the sample itself.
Population vs Sample
Bias vs Variability
The Target Analogy
Major Types of Bias
Undercoverage in Detail
Nonresponse and Response Bias
A neighborhood survey with visible routes for missing voices
A survey can lose people before selection or after invitation. Keeping the target population, sampling frame, selected sample and respondents separate reveals where each problem arises.Track who is missing at each stage
Suppose the target population has 24 IDs. The frame omits IDs 21–24; eight listed IDs are selected, and six respond. Frame omissions are undercoverage; selected people who do not respond are nonresponse. Bias direction depends on how missing responses differ.Participation does not remove wording effects
A respondent can still be influenced by the wording of a question. Compare the neutral and leading versions by identifying the assumption each asks the respondent to accept.Voluntary Response and Convenience Samples
Sampling Error vs Bias
| Sampling Error (Variability) | Bias | |
|---|---|---|
| Definition | Natural random variation from sample to sample | Systematic tendency to miss the true value |
| Cause | Random chance in who gets selected | Flawed sampling design or method |
| Fix | Increase sample size \( n \) | Improve the sampling method |
| Direction | Equally likely above or below truth | Consistently in one direction |
| Reducible? | Yes, with larger \( n \) | No, not by increasing \( n \) |
How to Identify Bias in a Study
Example 1: School Lunch Survey
A school principal wants to know how students feel about the cafeteria food. She places a suggestion box in the cafeteria and asks students to fill out a comment card if they wish.
Example 2: Mailed Health Survey
A researcher mails a 10-page health questionnaire to 5,000 randomly selected adults. Only 800 return the completed survey.
Example 3: Wording Effect
Two versions of a question are asked to separate random samples from the same population:
Result: 46% said yes
Result: 54% said yes
Common Mistakes
Summary of Sampling Problems
| Problem | What Happens | Example |
|---|---|---|
| Undercoverage | Part of population has no chance of selection | Landline-only phone survey |
| Nonresponse | Selected people don't participate | Long mailed questionnaire |
| Response Bias | Answers are systematically inaccurate | Leading or sensitive questions |
| Voluntary Response | Self-selected sample with strong opinions | Online review site ratings |
| Convenience | Easiest-to-reach people are chosen | Surveying friends at lunch |
Question 1 of 5
A radio station asks listeners to call in and vote on whether a new highway should be built. What is the primary sampling concern?
Question 2 of 5
A researcher surveys a random sample of 2,000 adults by mail. Only 300 respond. The researcher reports results based on the 300 responses. What is the main problem?
Question 3 of 5
Which of the following is true about increasing sample size?
Question 4 of 5
A survey asks: "Don't you agree that the school should invest more in student wellness programs?" What type of bias is most likely present?
Question 5 of 5
A political polling firm wants to estimate support for a new tax policy among all registered voters in a state. They use the state voter registration database as their sampling frame and select 1,500 voters at random. They conduct the survey by calling voters on weekday afternoons between 1 PM and 4 PM. Of the 1,500 selected, 900 answer and complete the survey.
Which sources of bias are most relevant in this scenario?