College Political Science Quiz: Survey Design

20 questions · exam conditions

0:00

Survey DesignQuestion 1 of 20

A polling firm aims to predict the winner of a Republican primary election. The firm obtains a list of all registered Republicans from the local board of elections and draws a simple random sample from it. Which potential source of bias is most directly related to this specific choice of sampling frame?

Non-response bias, as older voters are more likely to answer the phone than younger voters.

Response bias, as voters may be hesitant to reveal their true preference for a controversial candidate.

Coverage bias, as the frame of all registered Republicans is not identical to the population of actual primary voters.

Selection bias, because a simple random sample is not the most efficient method for this type of poll.

← Back to quizzes

College Political Science Quiz

College Political Science Quiz: Survey Design

Practice Survey Design in College Political Science with focused quiz questions that help you check what you know, review explanations, and build confidence with test-style prompts.

What this quiz covers

This quiz focuses on Survey Design, giving you a quick way to practice the rules, question types, and explanations that matter most for College Political Science.

How to use this quiz

Try each quiz question before looking at the correct answer. Use the explanations to review missed ideas, then come back to similar questions until the pattern feels familiar.

All questions

Question 1

Non-response bias, as older voters are more likely to answer the phone than younger voters.
Response bias, as voters may be hesitant to reveal their true preference for a controversial candidate.
Coverage bias, as the frame of all registered Republicans is not identical to the population of actual primary voters. (correct answer)
Selection bias, because a simple random sample is not the most efficient method for this type of poll.

Explanation: Coverage bias occurs when the sampling frame (the list from which the sample is drawn) does not perfectly match the target population. In this case, the target population is 'Republican primary voters.' The sampling frame is 'registered Republicans.' These are not the same; many registered Republicans will not vote in the primary, and in some states, independents or new registrants may be eligible to vote. This mismatch is a form of coverage bias.

Question 2

During the final month of a presidential campaign, a polling organization interviews 300 likely voters each night. To report a daily trend in candidate support, they use a 4-day rolling average, combining the most recent 1,200 interviews for each day's report. What is the primary analytical trade-off involved in using this rolling average method?

It reduces the poll's margin of error for the daily estimate but is less sensitive to very sudden shifts in opinion. (correct answer)
It reduces non-response bias by including more people over time but increases the risk of coverage bias.
It is less expensive to conduct than a traditional poll but cannot be used to make election predictions.
It ensures the sample reflects the most up-to-date public opinion while maintaining a perfectly random sample.

Explanation: The main benefit of a rolling average is that it uses a larger sample size (1,200 instead of 300), which reduces the random sampling error (margin of error) and creates a more stable, less volatile trend line. The trade-off is that because it includes data from previous days, it will react more slowly to a sudden, major event that changes public opinion overnight. It sacrifices some responsiveness for greater statistical stability.

Question 3

An exit poll's initial sample contains 55% women and 45% men. After the election, official data from the Secretary of State reveals that the actual electorate was composed of 52% women and 48% men. To correct this discrepancy, the pollster performs a statistical adjustment on the dataset before releasing the final results.

What is the primary purpose of this statistical adjustment?

To increase the reported margin of error to reflect the uncertainty introduced by the unrepresentative sample.
To impute data for the male voters who were missed in order to correct the non-response bias.
To apply a weight to responses so that the sample's gender composition matches the known population parameter. (correct answer)
To stratify the sample after data collection, allowing for more accurate subgroup analysis between men and women.

Explanation: This process is called weighting. The responses of the underrepresented group (men) are given slightly more weight, and the responses of the overrepresented group (women) are given slightly less weight. The goal is to make the final, adjusted sample accurately reflect the demographic composition of the population it is intended to represent (in this case, the actual electorate).

Question 4

In post-election surveys, the proportion of respondents who claim to have voted is consistently higher than the actual official turnout rate. Analysis shows this gap persists even when polls are perfectly weighted to match population demographics. This persistent discrepancy is a classic illustration of:

Coverage bias, where the survey's sampling frame fails to include a representative proportion of non-voters.
Non-response bias, where citizens who vote are more likely to agree to take surveys than citizens who do not vote.
Social desirability bias, where respondents provide the answer they believe is more socially respectable. (correct answer)
Sampling error, which accounts for the natural, random variation between a sample and the true population value.

Explanation: Social desirability bias occurs when respondents answer questions in a way that will be viewed favorably by others. Since voting is widely seen as a civic duty, some people who did not vote will report that they did to avoid embarrassment or to appear as good citizens. While non-response bias (B) also contributes to turnout overestimation, the act of misreporting one's own behavior is specifically social desirability bias.

Question 5

A major news website features a daily online poll on its homepage asking visitors to rate the president's performance. The poll consistently receives over 100,000 responses each day. Despite the very large sample size, these results are not considered scientifically valid primarily because:

The margin of error for such a large sample is practically zero, which is statistically unrealistic.
The sample is self-selected, making it unrepresentative of the general population. (correct answer)
The question wording is too simplistic and does not allow for nuanced opinions about presidential performance.
The poll lacks a proper weighting scheme to adjust for demographic imbalances in the sample.

Explanation: This is an opt-in or self-selected poll. The participants are not chosen through a random process; they choose to participate themselves. This introduces massive selection bias. The sample will be composed of people who visit that specific website and who feel strongly enough to participate, rather than a representative cross-section of the population. No sample size, however large, can overcome this fundamental flaw. Weighting (D) cannot fix a sample this biased.

Question 6

A university surveys students about their experiences with illegal substance use on campus. It considers two methodologies: a live, in-person interview conducted by a trained research assistant, or a self-administered, anonymous online questionnaire. The anonymous online questionnaire is likely to yield more accurate data on this topic primarily because it reduces:

Coverage bias, as nearly all students have internet access for an online survey.
Sampling error, because online surveys can achieve a much larger sample size for the same cost.
Non-response bias, because the convenience of an online survey guarantees a higher participation rate.
Response bias stemming from social desirability, as respondents are more willing to admit to sensitive behaviors anonymously. (correct answer)

Explanation: The key factor here is the sensitive nature of the topic (illegal substance use). In a face-to-face or live interview setting, respondents are more likely to feel judged or worried about consequences, leading them to underreport sensitive behaviors. This is a form of response bias driven by social desirability. An anonymous, self-administered format removes the interviewer from the equation, making respondents feel safer and more likely to provide truthful answers.

Question 7

A researcher is studying the political attitudes of a minority group that constitutes 4% of a state's population. To ensure the group's views are captured for meaningful subgroup analysis, the researcher designs a sample of 1,200 people that includes 300 members of the minority group (25% of the sample) and 900 from the rest of the population. To produce an accurate estimate of opinion for the entire state's population from this sample, what is the most critical next step?

The data must be statistically weighted to down-weight the oversampled minority group to its true 4% proportion in the population. (correct answer)
The sample must be considered a non-probability sample, and only qualitative conclusions can be drawn from the data.
The results for the two groups must be reported separately, as they cannot be combined into a valid statewide estimate.
The margin of error must be increased to account for the disproportionate sample design before reporting any statewide results.

Explanation: When you encounter a question about survey sampling with oversampling, you're dealing with the fundamental challenge of making valid population inferences from non-representative samples. Oversampling minorities is a common technique to ensure adequate sample sizes for subgroup analysis, but it creates a critical problem for population estimates. The correct approach is A - the data must be weighted to reflect the true population proportions. Since the minority group represents 25% of your sample but only 4% of the actual population, you need to statistically down-weight their responses when calculating statewide estimates. This weighting process restores representativeness and allows valid population inference. B is incorrect because oversampling doesn't make this a non-probability sample - it's still probability-based sampling with known selection probabilities that can be corrected through weighting. C is wrong because the groups can absolutely be combined once proper weights are applied; reporting separately would actually prevent you from achieving your goal of statewide estimates. D misunderstands the solution - while disproportionate sampling does affect precision, simply increasing the margin of error doesn't address the fundamental bias problem of overrepresentation. Remember this pattern: oversampling creates bias (systematic error), not just increased variance (random error). Weighting fixes bias by mathematically restoring proper population proportions. When you see oversampling questions, immediately think "weighting" rather than "larger margins of error" - the latter only addresses random sampling variation, not systematic overrepresentation.

Question 8

To estimate farmer support for a new agricultural subsidy, a government agency employs a multi-stage sampling design. First, they randomly select 200 counties from all U.S. counties with significant agricultural output. Second, from the property tax rolls in those counties, they randomly select 50 farm-owning households. They then attempt to interview the primary operator of each selected farm.

Which of the following describes a potential source of coverage bias that is inherent in this research design?

The initial selection of counties may, by chance, exclude certain types of agricultural regions, such as those specializing in fruit orchards.
Farmers who are financially struggling may be less likely to respond to a government survey, skewing the results.
The attitudes of farmers within a given county may be more similar to each other than to farmers in other counties, increasing total survey error.
Using property tax rolls as the sampling frame in the second stage may systematically exclude tenant farmers who do not own their land. (correct answer)

Explanation: When evaluating survey methodology, you need to distinguish between different types of bias. Coverage bias occurs when your sampling frame systematically excludes certain members of your target population, creating a gap between who you want to study and who you can actually reach. In this agricultural survey, the researchers want to estimate farmer support broadly, but their second-stage sampling frame—property tax rolls—creates a fundamental coverage problem. Property tax rolls only list landowners, which means tenant farmers who rent or lease their land will never appear in the sampling frame and thus can never be selected for the survey. This systematically excludes an entire category of farmers from potential participation, regardless of their attitudes toward the subsidy. Since tenant farmers may have different economic circumstances and policy preferences than landowners, this exclusion could significantly bias the results. Looking at the other options: (A) describes sampling variability, not coverage bias—random selection could theoretically include any agricultural region, even if some are missed by chance. (B) identifies nonresponse bias, where selected participants choose not to respond, rather than coverage bias where people can't be selected at all. (C) describes clustering effects that increase sampling error but don't prevent any farmers from being included in the sampling frame. Remember that coverage bias is about who can't even be reached by your sampling design, while other biases involve problems with people who could theoretically be included. Always check whether the sampling frame matches the target population.

Question 9

In a pre-election poll for a mayoral race with a prominent Latino candidate and a white candidate, the Latino candidate leads by 10 points. On election day, however, the Latino candidate wins by only 2 points. Polls for other races in the same election were highly accurate.

This discrepancy, where a minority candidate's support is inflated in pre-election polls, is most consistent with which polling phenomenon?

The bandwagon effect, where undecided voters support the candidate who appears to be winning.
A late swing among undecided voters who disproportionately broke for the white candidate.
The Bradley Effect, a form of social desirability bias affecting polls with minority candidates. (correct answer)
Non-response bias, where supporters of the white candidate were systematically less likely to participate in the poll.

Explanation: The Bradley Effect (or Wilder Effect) is the theory describing a phenomenon where polls overstate the support for a minority candidate. It is attributed to a form of social desirability bias, where some white voters tell pollsters they are undecided or will vote for the minority candidate because they feel it is the socially acceptable answer, but then vote for the white candidate in the privacy of the voting booth.

Question 10

A student researcher studying political engagement on campus stands in front of the main library on a weekday and surveys students who pass by. The researcher's goal is to survey 150 students, ensuring that the sample includes 50 freshmen, 40 sophomores, 30 juniors, and 30 seniors. This sampling method is best described as:

Stratified random sampling, because the student population was divided into strata based on class year.
Systematic random sampling, because students were approached in a sequential but non-random pattern.
Simple random sampling, because every student had an equal chance of being near the library.
Quota sampling, because participants were selected non-randomly until pre-defined categories were filled. (correct answer)

Explanation: This is a non-probability sampling method. While it involves creating categories (strata) like stratified sampling, the selection of individuals within those categories is not random. The researcher surveys conveniently available students until the quota for each class year is met. This combination of convenience selection with demographic targets is the definition of quota sampling.

Question 11

A national poll on attitudes toward tech regulation is conducted using random-digit dialing of landline telephone numbers. The final unweighted sample consists of a higher proportion of older, retired individuals and a lower proportion of younger, urban-dwelling individuals compared to census data.

A critic argues the poll is biased. Which of the following statements most accurately diagnoses the primary sources of error?

The poll primarily suffers from sampling error, which could be rectified by substantially increasing the sample size.
The main issue is response bias, as respondents may not feel comfortable sharing their true opinions on a controversial topic with a live interviewer.
The methodology is flawed by both coverage bias from excluding cell-phone-only households and non-response bias from certain groups being less likely to answer. (correct answer)
This is a classic example of selection bias resulting from the use of a non-probability convenience sample.

Explanation: The methodology has two main flaws. First, using only landlines creates coverage bias because it excludes the significant portion of the population that only uses cell phones (who are often younger and more urban). Second, even among those called, older individuals being overrepresented suggests that younger people were less likely to answer, which is non-response bias.

Question 12

A survey on government priorities asks respondents the following question: 'Do you agree or disagree that the federal government should reduce taxes on small businesses and decrease regulations on the energy sector?' What is the most significant flaw in this survey question's design?

It contains loaded language that is likely to bias the responses in a particular direction.
It is a double-barreled question, compelling a single response to two separate policy issues. (correct answer)
It presents a false dichotomy by limiting the respondent's options to simply 'agree or disagree'.
It is likely to induce significant acquiescence bias due to its agree/disagree format.

Explanation: This is a classic double-barreled question. It asks about two distinct issues: 1) reducing taxes on small businesses, and 2) decreasing regulations on the energy sector. A respondent might agree with one part but disagree with the other, but the question forces them into a single 'agree' or 'disagree' answer, making the results impossible to interpret accurately.

Question 13

A researcher studying political trust in a large, geographically diverse state divides the state into its 58 counties. The researcher then randomly selects 10 of these counties and attempts to survey every adult within those 10 selected counties. Which of the following is the most significant methodological weakness of this approach for generating a precise statewide estimate?

It is a non-probability sampling method, meaning the results cannot be generalized to the entire state's population.
It is susceptible to high sampling error because individuals within counties may be more politically homogeneous than the state as a whole. (correct answer)
It fails to stratify by key demographic groups like age and race, which is a necessary step for any valid political survey.
It will suffer from severe non-response bias because the chosen counties may not be representative of the state's urban-rural divide.

Explanation: The method described is one-stage cluster sampling. Its primary weakness is that clusters (counties) are often internally homogeneous but different from other clusters. By selecting only a few clusters, the sample may not capture the full diversity of the state, leading to higher sampling error compared to a simple random sample of the same size. The estimate's precision is therefore reduced.

Question 14

A political advocacy group commissions a survey that includes the following question: 'In light of the constant threat of foreign cyberattacks and the need to protect national security, do you support or oppose government programs that monitor internet communications?' This question is most likely to produce biased results due to its use of:

A double-barreled structure that asks about two separate issues at once.
A leading premise and loaded terminology that encourages an affirmative response. (correct answer)
Social desirability pressure, as respondents will want to appear patriotic to the interviewer.
Acquiescence bias, as respondents have a general tendency to agree with propositions.

Explanation: The question's primary flaw is the introductory clause, 'In light of the constant threat...and the need to protect national security'. This is a leading premise that frames the issue in a way that makes support seem like the only reasonable option. Terms like 'constant threat' are loaded and designed to elicit an emotional response, biasing the results toward agreement.

Question 15

A voter receives an automated phone call that says: 'Hello, we are conducting a political survey. Would you be more or less likely to vote for Councilwoman Miller if you knew she voted to double her own salary during a city-wide budget crisis?' After the voter responds, the call immediately ends. This activity is best characterized as:

A tracking poll with a poorly worded, leading question.
A survey experiment designed to test the effect of negative information.
A push poll intended to disseminate negative information under the guise of research. (correct answer)
An exit poll designed to understand the motivations behind voter choice.

Explanation: A push poll is not a legitimate survey. It is a political persuasion tactic disguised as a poll. Its purpose is to 'push' voters away from a candidate by asking a question that contains negative, often false or misleading, information. The lack of demographic questions or further survey items is a key indicator.

Question 16

Non-response bias, as older voters are more likely to answer the phone than younger voters.
Response bias, as voters may be hesitant to reveal their true preference for a controversial candidate.
Coverage bias, as the frame of all registered Republicans is not identical to the population of actual primary voters. (correct answer)
Selection bias, because a simple random sample is not the most efficient method for this type of poll.

Question 17

A researcher conducts a split-ballot experiment. Half the respondents are asked Question A ('Overall, do you approve or disapprove of the job Congress is doing?') and then Question B ('Do you approve or disapprove of the job your own representative in Congress is doing?'). The other half are asked the questions in reverse order (B then A). The researcher finds that approval for 'your own representative' is significantly higher when asked before the general question about Congress. This research design is explicitly measuring the impact of:

Question order effects, where the context of prior questions influences subsequent answers. (correct answer)
The reliability of the survey instrument using a split-half correlation method.
Response bias caused by confusing or ambiguous terminology in the questions.
Social desirability bias related to expressing anti-government sentiment.

Explanation: This is a classic experimental design to test for question order effects. By randomly assigning respondents to different question orders and observing a systematic difference in responses, the researcher can isolate the causal impact of the question sequence. In this case, asking about the unpopular Congress first 'primes' respondents to be more critical, and some of that negativity carries over to their evaluation of their own representative.

Question 18

Coverage bias, where the survey's sampling frame fails to include a representative proportion of non-voters.
Non-response bias, where citizens who vote are more likely to agree to take surveys than citizens who do not vote.
Social desirability bias, where respondents provide the answer they believe is more socially respectable. (correct answer)
Sampling error, which accounts for the natural, random variation between a sample and the true population value.

Question 19

It contains loaded language that is likely to bias the responses in a particular direction.
It is a double-barreled question, compelling a single response to two separate policy issues. (correct answer)
It presents a false dichotomy by limiting the respondent's options to simply 'agree or disagree'.
It is likely to induce significant acquiescence bias due to its agree/disagree format.

Question 20

The margin of error for such a large sample is practically zero, which is statistically unrealistic.
The sample is self-selected, making it unrepresentative of the general population. (correct answer)
The question wording is too simplistic and does not allow for nuanced opinions about presidential performance.
The poll lacks a proper weighting scheme to adjust for demographic imbalances in the sample.