- A pharmaceutical company wants to test a new blood pressure medication. They randomly select 500 participants from a database of patients with hypertension across multiple medical centers. These participants are then randomly divided into two groups: 250 receive the new medication and 250 receive a placebo. Which statement best describes the roles of the two types of randomization used?
Answer: Random assignment enables causal conclusions about drug effectiveness, while random sampling allows generalization to the hypertensive patient population
Random assignment (randomly dividing participants into treatment and placebo groups) enables causal conclusions by ensuring groups are equivalent except for the treatment. Random sampling (selecting participants from the patient database) enables generalization to the broader population of hypertensive patients. These serve distinct and complementary purposes.
- A psychology professor wants to study whether taking notes by hand versus laptop affects exam performance. She teaches two sections of the same course. In Section A (morning class), all students are required to take notes by hand. In Section B (afternoon class), all students must use laptops for note-taking. She compares average exam scores between the two sections. What is the primary threat to drawing valid causal conclusions from this study design?
Answer: Confounding variables associated with class timing and student characteristics may influence the observed differences in exam scores
When evaluating research designs for causal conclusions, you need to identify what could create alternative explanations for any observed differences between groups. The gold standard is random assignment, which helps ensure groups are equivalent except for the treatment.
In this study, the professor assigned note-taking methods based on class sections rather than randomly assigning students to conditions. This creates a major problem: students who choose morning versus afternoon classes may differ systematically in ways that affect exam performance. Morning students might be more organized, have fewer work conflicts, or differ in motivation levels. Additionally, factors like alertness, hunger, or competing afternoon activities could influence performance regardless of note-taking method. These confounding variables make it impossible to determine whether any score differences result from the note-taking method or from pre-existing differences between the groups.
Option B incorrectly focuses on external validity (generalizability) rather than internal validity (causal conclusions). While random sampling does affect generalizability, the question asks specifically about threats to causal conclusions. Option C assumes sample size is the issue, but even large samples can't fix confounding variables. Option D is completely wrong—random assignment is crucial for causal inference, and its absence is precisely the problem here.
Study tip: When analyzing research designs, always ask "What else could explain the results?" If groups weren't randomly assigned, look for systematic differences between them that could serve as alternative explanations. Confounding variables are the primary threat to causal conclusions in quasi-experimental designs like this one.
- A researcher wants to determine if a new teaching method improves student performance compared to traditional methods. She randomly selects 200 students from all students in the district, then randomly assigns 100 to receive the new method and 100 to receive traditional instruction. After the study, she finds that students using the new method scored significantly higher. What conclusion is most appropriate?
Answer: The new teaching method likely causes improved performance, and results can be generalized to all students in the district from which the sample was drawn.
This study uses both random sampling (from the district population) and random assignment (to treatment groups). Random sampling allows generalization to the district population, while random assignment allows causal conclusions about the treatment effect. Choice A overgeneralizes beyond the sampled population. Choice C incorrectly states that random assignment wasn't used. Choice D fails to recognize that random sampling enables generalization.
- A health researcher wants to determine if a new exercise program prevents heart disease. She identifies 1,000 healthy adults and randomly assigns 500 to follow the exercise program for 5 years while 500 continue their usual activities. After 5 years, she finds that 3% of the exercise group developed heart disease compared to 7% of the control group. What additional information would most strengthen the causal interpretation of these results?
Answer: Evidence that participants in both groups had similar baseline characteristics after randomization, confirming the absence of confounding variables.
Random assignment should create balanced groups, but verifying that it actually worked strengthens causal inference by confirming no confounding variables differ between groups. This is especially important with only 1,000 participants. Choice B addresses compliance but the results already show a difference suggesting the program had some effect. Choice C addresses generalizability, not causation within this study. Choice D addresses a potential confounder but baseline balance is more fundamental to the randomization's success.
- A university researcher wants to study the effect of class size on student learning outcomes. She identifies all statistics courses offered across campus and randomly selects 20 courses for her study. However, she cannot randomly assign students to different class sizes due to administrative constraints, so she compares learning outcomes between naturally occurring small classes (fewer than 20 students) and large classes (more than 40 students). What type of conclusions can be most confidently drawn from this study?
Answer: Results can be generalized to statistics courses at similar universities, but causal claims about class size are not well-supported
When evaluating research studies, you need to distinguish between two key components: the sampling method (which affects generalizability) and the assignment method (which affects causal inference). These determine what conclusions you can confidently draw.
The researcher used random sampling by randomly selecting 20 statistics courses from all available courses on campus. This strong sampling approach means the results can reasonably be generalized to similar statistics courses at comparable universities. However, students weren't randomly assigned to different class sizes—they were already enrolled in naturally occurring small or large classes. This lack of random assignment weakens causal claims because other factors might explain differences in learning outcomes (perhaps more motivated students self-select into smaller classes, or different professors teach different-sized classes).
Answer D correctly identifies that generalization is supported by the random sampling, while causal conclusions are limited by the lack of random assignment.
Answer A is wrong because causal conclusions aren't valid—the absence of random assignment means confounding variables could explain the results. Answer B incorrectly assumes that natural variation equals random assignment; it doesn't, because systematic differences might exist between students who end up in small versus large classes. Answer C is wrong because random sampling was properly implemented when courses were randomly selected, supporting generalization even though causal inference is limited.
Remember: Random sampling supports generalization to broader populations, while random assignment supports causal inference. A study can have one without the other, and your conclusions must reflect which elements are present.
- Based on the sampling procedure described above, which statement about the validity of conclusions is most accurate?
Answer: The systematic selection process eliminates sampling bias, but the low response rate creates nonresponse bias affecting representativeness.
When evaluating survey validity, you need to assess both sampling design quality and potential bias sources. This question tests your ability to identify how different aspects of data collection affect the reliability of conclusions.
The sampling procedure described uses systematic random selection at multiple stages: regions are stratified, counties are randomly selected within regions, and voters are randomly selected within counties. This methodical approach effectively eliminates sampling bias because every registered voter has a known, non-zero probability of selection. However, the 36% response rate creates a significant concern—nonresponse bias occurs when people who respond differ systematically from those who don't, potentially skewing results away from the true population values.
Looking at the wrong answers: Choice A incorrectly suggests the design ensures representativeness despite nonresponse—but low response rates can severely compromise representativeness regardless of good initial design. Choice B overstates the problem by claiming the sampling design itself compromises validity, when random selection actually strengthens it. Choice C correctly identifies the limitation to registered voters but misses the more critical issue of nonresponse bias affecting even that population.
Choice D correctly identifies both strengths and weaknesses: the systematic random selection does eliminate sampling bias, but the low response rate creates nonresponse bias that threatens representativeness.
Study tip: For survey methodology questions, evaluate sampling design and response rates separately. Good random sampling eliminates selection bias, but poor response rates can still undermine validity through nonresponse bias—both factors matter for drawing reliable conclusions.
- A researcher studying the effectiveness of two different math curricula has access to 10 elementary schools. She randomly assigns 5 schools to use Curriculum A and 5 schools to use Curriculum B for one academic year. At the end of the year, she compares average test scores between the two groups. What potential confounding factor is most likely to threaten the validity of her causal conclusions?
Answer: Students were not randomly assigned to schools, so pre-existing differences in student ability between school populations could account for observed differences in test scores.
While schools were randomly assigned to curricula, students were not randomly assigned to schools. If schools serve different populations (different socioeconomic areas, academic backgrounds, etc.), pre-existing student differences could confound the curriculum comparison. Choice A identifies a real concern but teacher effects are somewhat controlled by random assignment of schools. Choice C addresses statistical power, not confounding. Choice D is less plausible since both curricula operate under the same seasonal conditions.
- Dr. Martinez wants to test whether meditation reduces anxiety levels. She recruits volunteers from a wellness center, measures their baseline anxiety, then has all participants complete a 6-week meditation program. She finds that anxiety levels decreased significantly from baseline. Her colleague suggests the study design has limitations. What is the most significant limitation?
Answer: The study lacks random assignment to treatment and control groups, preventing determination of whether meditation specifically caused the anxiety reduction.
This is a single-group pre-post design without a control group or random assignment. All participants received meditation, so there's no way to determine if the improvement was due to meditation specifically versus other factors like attention, expectation, or natural fluctuation in anxiety. Choice B identifies a real limitation but generalizability is secondary to establishing whether the treatment works at all. Choices C and D identify minor design issues but miss the fundamental lack of experimental control.
- A social media company conducts a study by posting an online survey asking users to rate their satisfaction with a new feature. The survey receives 50,000 responses. Based on these responses, the company concludes that users generally love the new feature. What is the primary limitation of this approach for drawing valid conclusions?
Answer: The lack of random sampling creates selection bias, as only users motivated to respond participated, potentially overrepresenting satisfied users.
This is a voluntary response sample, not a random sample. Users who choose to respond to surveys about new features are likely those with strong opinions (often positive for new features they chose to use). This creates selection bias that limits generalizability. Choice A is incorrect because 50,000 is a large sample. Choice C describes an experimental design issue but this is observational research about existing satisfaction. Choice D is irrelevant since this isn't testing causation but measuring current satisfaction levels.
- What is the most important limitation of this study design for drawing broader conclusions about exercise and stress?
Answer: Random assignment was properly implemented, but the volunteer recruitment method limits generalization to people already interested in exercise
The researcher used random assignment (participants randomly assigned to exercise groups), enabling causal conclusions about exercise effects on stress. However, recruitment from gyms and fitness centers creates a convenience sample of people already interested in exercise, limiting generalization to the broader population. Random assignment supports causation; representative sampling supports generalization.
- A market research firm wants to estimate smartphone usage patterns among teenagers in a large metropolitan area. They obtain permission to survey students at 15 randomly selected high schools from the 200 schools in the area. At each selected school, they survey all students in 5 randomly chosen classes. This yields 2,250 student responses. What is the most significant limitation for generalizing these results?
Answer: Students attending high school may not represent all teenagers, excluding those who are homeschooled, dropped out, or attend private schools.
When evaluating sampling methods, you need to distinguish between technical sampling concerns and fundamental coverage issues that affect who can even be included in your sample.
The correct answer is D because this study has a serious coverage problem. The sampling frame (high school students) systematically excludes entire groups of teenagers: those who are homeschooled, have dropped out, attend private schools, or aren't enrolled in traditional schools for other reasons. No matter how well the sampling is executed within high schools, these excluded teenagers might have very different smartphone usage patterns. This creates a fundamental bias that can't be fixed by better sampling techniques.
Let's examine why the other options miss the mark:
A is incorrect because while cluster sampling does increase sampling variability compared to simple random sampling, this is a manageable technical issue that affects precision, not the fundamental ability to generalize to all teenagers.
B is wrong because 2,250 is actually a substantial sample size for most research purposes. Sample size adequacy depends on the research goals, but this size would typically provide sufficient power for detecting meaningful differences.
C misunderstands multistage sampling. While it does introduce multiple sources of sampling error, this is a standard, well-understood statistical technique. The errors don't simply "compound" in a problematic way—they can be properly accounted for in the analysis.
Study tip: Always distinguish between sampling precision issues (which affect how accurate your estimates are for your target population) and coverage issues (which affect whether you're even studying the right population).
- What is the most significant methodological concern that affects the validity of using this 65% figure to represent city-wide opinion?
Answer: The 40% response rate introduces potential nonresponse bias that may not represent all residents' views accurately
The main concern is nonresponse bias from the 40% response rate (320/800). People who respond to surveys may systematically differ from non-respondents in their opinions. Systematic sampling (every 50th number) approximates random sampling reasonably well. Random assignment isn't relevant for opinion polling, and 320 is typically adequate sample size.
- A university researcher obtains a complete list of all registered students and uses a random number generator to select 500 students for a study about sleep habits. However, due to budget constraints, she can only interview students who live on campus. Of the 500 selected students, 180 live on campus and complete the study. What type of sampling issue does this create?
Answer: Undercoverage bias occurs because the final sample systematically excludes off-campus students, who may have different sleep patterns than on-campus students.
The initial random selection was valid, but the practical constraint of only interviewing on-campus students creates undercoverage bias. Off-campus students (who were part of the target population) are systematically excluded from the final sample, and they likely have different sleep patterns due to different living situations, commute times, etc. Choice A is wrong because the initial selection was properly random. Choice B is incorrect because students didn't self-select - they were excluded by the researcher's constraint. Choice D addresses data quality, not sampling methodology.
- A medical researcher studies whether a new rehabilitation program helps stroke patients recover motor function. She recruits 120 recent stroke patients from three hospitals. At Hospital A, patients receive the new program; at Hospital B, patients receive standard care; at Hospital C, patients receive both programs combined. After 6 months, Hospital A patients show the most improvement. What prevents a valid causal conclusion about the new program's effectiveness?
Answer: The study lacks random assignment of patients to treatment conditions, as hospital location determines treatment rather than random allocation.
Patients weren't randomly assigned to treatments - their hospital determined their treatment. Hospitals likely differ in patient populations (socioeconomic status, stroke severity, other medical conditions), staff expertise, facilities, and care quality. These systematic differences between hospitals confound the treatment comparison. Choice B addresses generalizability, not internal validity for causal inference. Choice C is wrong because multiple treatment comparisons are valid research designs. Choice D misunderstands that comparing different treatments can establish relative causal effects.
- Based on this study design, which limitation most significantly affects the types of conclusions Dr. Martinez can reasonably draw?
Answer: Random sampling was not used, so generalization beyond the volunteer participants may not be valid or reliable
Dr. Martinez used random assignment (coin flip), which allows for causal conclusions about music's effect on concentration. However, she used a convenience sample of volunteers rather than random sampling from the population, which limits generalizability to the broader college student population. Random assignment enables causal inference, but random sampling is needed for generalization.
- Which aspect of this study design most likely affects the validity of the 78% estimate for the entire adult population?
Answer: The calling schedule systematically excludes certain demographic groups, introducing coverage bias into the sample
When evaluating survey methodology, you need to assess whether the sampling process gives every member of the target population a fair chance of being included. The key issue here isn't the sampling method itself, but rather systematic barriers that prevent certain groups from participating.
The correct answer is D because calling only during 9 AM-5 PM weekdays creates a systematic bias. This schedule excludes people who work traditional daytime hours and can't answer calls during business hours. Since employment status, work schedules, and availability during weekday business hours correlate with demographic factors like age, income, and occupation, this timing systematically underrepresents entire population segments. This is coverage bias—when your sampling frame fails to adequately represent the target population.
Option A is incorrect because random digit dialing actually does provide a reasonably random sampling method for reaching adults with phone access. Option B misses the mark—1,200 is typically an adequate sample size for population estimates with acceptable margins of error (usually around ±3%). Option C confuses study design types; this is an observational study estimating a population parameter, not an experiment trying to establish causation, so random assignment isn't relevant or expected.
Remember: In survey methodology questions, always check for systematic exclusions in the data collection process. Coverage bias occurs when certain groups are systematically less likely to be included in your sample, and timing restrictions are a common source of this bias in phone surveys.
- A psychology professor wants to test whether background music affects test performance. She has two sections of the same course. Section A (morning class) takes their exam with classical music playing, while Section B (afternoon class) takes the exam in silence. Section A scores significantly higher. The professor concludes that classical music improves test performance. What is the most serious flaw in this conclusion?
Answer: The study lacks random assignment to music conditions, so differences might be due to systematic differences between morning and afternoon students.
Students self-selected into morning vs afternoon sections, creating systematic differences. Morning students might be more motivated, have different sleep patterns, be different personality types, or have different academic backgrounds than afternoon students. This confounding prevents causal attribution to music. Choice B addresses generalizability but not the internal validity problem. Choice C identifies potential confounders but they're less systematic than the time-of-day selection bias. Choice D misunderstands experimental design - one group serves as the control.
- Researchers want to test whether a new reading intervention improves comprehension scores. They randomly select 40 elementary schools from a statewide database. At each selected school, they randomly assign half of the third-grade students to receive the new intervention and half to continue with standard instruction. After 6 months, they compare reading comprehension scores. This design allows the researchers to:
Answer: Make both causal inferences about intervention effectiveness and generalize findings to third-graders throughout the state
This study uses both random sampling (schools randomly selected from statewide database) and random assignment (students within schools randomly assigned to intervention or control). Random sampling enables generalization to the state population of third-graders, while random assignment enables causal conclusions about intervention effectiveness. Both types of inference are supported by this design.
- An online retailer wants to estimate customer satisfaction across their entire customer base of 2 million people. They send email surveys to a random sample of 10,000 customers and receive 1,200 responses. The results show high satisfaction ratings. What is the most appropriate interpretation of these results?
Answer: The results may overestimate satisfaction for the entire customer base because customers who respond to surveys may be systematically different from non-respondents in their satisfaction levels.
Even though the initial sample was properly randomized, only 12% responded (1,200/10,000). This creates nonresponse bias - customers who take time to respond to satisfaction surveys may be systematically different from those who don't, potentially being more engaged or satisfied. Choice A ignores the nonresponse issue. Choice C incorrectly focuses on absolute sample size rather than response rate and bias. Choice D is wrong because random sampling was used, and stratification isn't always necessary.
- A city planning department wants to survey residents about a proposed park development. The city has 50 neighborhoods with varying income levels. Planners randomly select 10 high-income, 10 middle-income, and 10 low-income neighborhoods, then survey 20 randomly chosen residents from each selected neighborhood. What sampling design is being used and what is its main benefit?
Answer: Multistage sampling combining stratified and cluster methods, ensuring representation across income levels while maintaining cost efficiency by concentrating data collection within selected neighborhoods.
This is multistage sampling: first stage stratifies by income and randomly selects neighborhoods within each stratum, second stage randomly selects residents within chosen neighborhoods. This ensures representation across income levels (stratification benefit) while being practical since surveyors only visit 30 neighborhoods instead of potentially all 50 (clustering benefit). Choice B incorrectly describes it as systematic. Choice C is wrong because equal sample sizes from each income stratum isn't proportional to city demographics. Choice D incorrectly claims it eliminates nonresponse bias.
