- A university admissions officer claims that students who submit applications early are accepted at a 75% rate compared to 45% for regular applicants. This analysis included 320 early applications and 1,200 regular applications from last year. Which factor is most important for evaluating whether early submission actually improves acceptance chances?
Answer: Whether early applicants systematically differ from regular applicants in academic qualifications, extracurricular activities, or other admission factors.
The key question is whether early applicants are inherently different from regular applicants in ways that affect their acceptance chances. If early applicants tend to be more qualified, the higher acceptance rate might reflect their qualifications rather than the timing of their applications. Choice B addresses policy but not the fundamental confounding issue. Choice C focuses on statistical significance rather than causation. Choice D concerns generalizability rather than the causal relationship.
- The advocacy group concluded that Brand A has superior battery performance. What is the most significant methodological concern with this conclusion?
Answer: Users were not randomly assigned to phone brands, so differences in usage patterns between brand communities could explain the results.
When evaluating research conclusions, you need to distinguish between valid statistical findings and sound causal inferences. A study can show real differences between groups while still having serious flaws in its design that undermine the conclusions.
The correct answer is C because this study has a fundamental confounding variable problem. Users weren't randomly assigned to phone brands - they chose their own phones. This means Brand A and Brand B users likely differ in systematic ways beyond just their phone choice. Heavy users might gravitate toward phones marketed for long battery life, while casual users might prioritize other features. Different user communities could have vastly different usage patterns, charging habits, or even definitions of "battery duration." These pre-existing differences between the groups could entirely explain the 1.4-hour difference in battery life, making it impossible to attribute the results to the phones themselves.
Option A is wrong because while 2 weeks isn't ideal for long-term patterns, it's sufficient to detect basic performance differences if the study design were sound. Option B is incorrect - 25 users per group provides reasonable statistical power for detecting a 1.4-hour difference with these standard deviations. Option D identifies a real limitation (measurement error from self-reporting), but this would likely affect both brands equally and wouldn't systematically bias results toward one brand.
Remember this key principle: when you see comparative studies where participants weren't randomly assigned to groups, always ask whether the groups might differ in ways other than the treatment being studied. Self-selection bias is one of the most common threats to valid causal conclusions.
- Which factor most undermines the validity of concluding that new street lighting caused the crime reduction?
Answer: Neighborhoods selected for lighting upgrades may have had other crime prevention initiatives that could account for the observed reduction.
The biggest threat to causal inference is confounding variables. Neighborhoods chosen for lighting upgrades likely received them for specific reasons and may have had other concurrent crime prevention efforts. This makes it impossible to isolate the lighting effect. Choice A raises timing concerns but one year is reasonable for this analysis. Choice B wrongly suggests unequal sample sizes create bias. Choice D misses the point about causation vs. effect size.
- A pharmaceutical company reports that their new medication reduced symptoms in 85% of trial participants compared to 45% in the placebo group. However, the trial had 40 participants in the treatment group and 35 in the placebo group. A medical reviewer questions whether this difference is meaningful. Which concern is most justified?
Answer: The sample sizes are too small to reliably detect true differences between treatment and placebo effects in medical trials.
With only 40 and 35 participants, the sample sizes are indeed too small for reliable medical conclusions. Small samples have high variability and low statistical power, making chance differences appear significant. Choice B is wrong - slight size differences don't create systematic bias. Choice C incorrectly assumes large differences must be fabricated. Choice D misunderstands the design - the placebo group IS the control.
- A researcher claims that a new study method improves test scores by an average of 15 points. The study included 8 students who used the new method for one week, with scores improving from 72 to 87 points on average. The standard deviation of the improvements was 18 points. Which statement best evaluates this claim?
Answer: The claim is questionable because the high variability relative to the effect size suggests the improvement may not be reliable.
With a standard deviation of 18 points and a mean improvement of 15 points, the variability is larger than the claimed effect. Combined with the very small sample size (n=8), this high variability makes the results unreliable. Choice A ignores variability and sample size. Choice C focuses on study duration rather than statistical issues. Choice D incorrectly assumes all students improved and misinterprets the relationship between effect size and measurement error.
- Given that both groups had standard deviations of approximately 12 points, which evaluation of the researcher's claim is most appropriate?
Answer: The claim is questionable because the 2.5-point difference is small relative to the variability in scores
With standard deviations of 12 points, a 2.5-point difference in means represents only about 0.2 standard deviations - a very small effect that could easily be due to random variation rather than a real difference. This difference is not practically meaningful given the variability in the data. Choice A ignores the statistical significance relative to variability. Choice C is incorrect because means are appropriate for this type of analysis. Choice D is incorrect because 150 students is generally sufficient for this type of study.
- What is the most significant limitation in evaluating this accident reduction claim?
Answer: The study lacks a control location without bike lane changes to determine if the reduction occurred citywide or specifically here.
The fundamental flaw is the lack of a control group. Without comparing to similar intersections that didn't get bike lanes, we can't determine if the reduction was due to the bike lane or other factors affecting the entire city. Choice A mentions valid concerns but they're secondary to the control group issue. Choice B overstates the sample size problem. Choice C is incorrect - the calculation method is appropriate for the claim being made.
- A health insurance company analyzed claims data and found that members who use their wellness app have 30% lower medical costs than non-users. The analysis included 12,000 app users and 45,000 non-users over an 18-month period. The company promotes this as evidence that their app reduces healthcare costs. What is the strongest criticism of this conclusion?
Answer: The observational design cannot establish causation because app users likely differ from non-users in health consciousness and baseline health status.
This is a classic selection bias problem. People who choose to use wellness apps are typically more health-conscious, may already be healthier, and likely engage in other healthy behaviors that reduce medical costs. The cost difference probably reflects these pre-existing differences rather than app effectiveness. Choice B underestimates the adequacy of 18 months. Choice C incorrectly suggests unequal sample sizes create bias. Choice D identifies self-selection but doesn't explain why it's problematic as clearly as Choice A.
- A fitness tracker company claims their device motivates users to walk 2,000 more steps per day on average. They collected data from 150 users over 8 weeks, comparing their daily steps before and after getting the device. The mean increase was 2,100 steps with a standard deviation of 3,200 steps. What is the most concerning aspect of this claim?
Answer: The extremely high variability in step increases suggests many users had no benefit while others had unusually large increases.
A standard deviation of 3,200 steps when the mean increase is only 2,100 steps indicates enormous variability. This suggests the effect is highly inconsistent across users, with many likely showing no increase or even decreases. The high variability undermines confidence in the claimed average effect. Choice A identifies a valid design flaw but the variability issue is more fundamental. Choice C raises timing concerns but 8 weeks is reasonable for this type of study. Choice D incorrectly suggests 150 is too small.
- A supplement company surveyed 200 customers who had purchased their vitamin product and found that 78% reported feeling more energetic after 30 days of use. The company claims their vitamin increases energy levels in most users. Based on this survey design, what is the primary concern about this claim?
Answer: Response bias is probable since customers who felt no benefit may be less likely to respond to surveys from the company.
The primary concern is response bias - customers who experienced positive effects are much more likely to respond to the company's survey than those who saw no benefit. This systematically inflates the percentage of positive responses. Choice A is wrong; 200 is adequate sample size. Choice B focuses on timing rather than bias. Choice C incorrectly identifies the bias - surveying customers who bought the product is appropriate, but getting biased responses from them is the real problem.
- Which aspect of this study design most strengthens confidence in attributing the 0.5 grade level difference to the intervention?
Answer: The initial similarity in reading levels between groups, controlling for baseline differences
When evaluating experimental studies, you need to identify what makes the results credible by eliminating alternative explanations for the observed differences. The key question is: could something other than the intervention explain why one group performed better?
The correct answer is A because initial similarity in reading levels is crucial for establishing causation. If the groups started at different baseline levels, any final difference might simply reflect those pre-existing differences rather than the intervention's effect. By ensuring both groups began with similar reading abilities, researchers can confidently attribute the 0.5 grade level difference to the intervention itself, not to one group being naturally stronger readers.
Let's examine why the other options don't strengthen causal inference as much: B suggests the 6-month duration was important, but timing doesn't address whether other factors caused the difference. C mentions equal sample sizes, which helps with statistical power but doesn't eliminate confounding variables—even with balanced groups, if one group started ahead, that advantage could explain the results. D focuses on measurement units, but using grade levels versus raw scores doesn't strengthen the causal argument; it's about interpretability, not validity.
Remember this pattern: in experimental design questions, look for what eliminates alternative explanations for the results. Controlling for baseline differences through matching or random assignment is often the strongest design feature because it ensures groups are truly comparable from the start.
- Which of the following represents the most significant limitation in concluding that the new curriculum caused the improvement in test scores?
Answer: The study lacks a control group using the old curriculum during the same time period
The most critical flaw is the lack of a proper control group. Comparing this year's scores to last year's scores introduces confounding variables (different students, different teachers, different conditions, potential natural improvement over time). A proper study would need a control group using the old curriculum during the same semester. Choice A is incorrect because 5 schools can provide meaningful data if properly designed. Choice C is incorrect because the significance of a 12-point increase depends on the scale and context. Choice D is incorrect because including multiple grade levels isn't necessary for validity if the study is properly focused.
- Which factor most seriously undermines the validity of the company's effectiveness claim?
Answer: The absence of a placebo control group makes it impossible to isolate the medication's effect
When evaluating scientific studies, you need to identify what factors could prevent researchers from drawing valid conclusions about cause and effect. The fundamental question here is whether we can confidently attribute the observed improvements to the medication itself.
The correct answer is D because without a placebo control group, there's no way to determine what would have happened to these patients without the medication. Many conditions improve naturally over time, and patients often experience psychological benefits just from receiving treatment (the placebo effect). The 85% improvement rate might seem impressive, but if 80% of patients typically improve without any treatment, then the medication would actually be harmful. A control group receiving a placebo would reveal the medication's true effect by showing the baseline improvement rate.
Choice A is wrong because while long-term effects matter for comprehensive evaluation, they don't undermine claims about short-term effectiveness. Choice B incorrectly assumes the sample size is inadequate—100 patients can provide meaningful results if the study is properly designed. Choice C addresses study comprehensiveness but doesn't fundamentally challenge the validity of measuring effectiveness within the studied population.
Study tip: In research design questions, always look for the absence of proper controls first. The most sophisticated study becomes meaningless without a way to isolate the variable being tested. Remember that correlation doesn't prove causation—you need controls to establish that the treatment, not other factors, caused the observed results.
- What represents the most serious flaw in concluding that app usage frequency causes the weight loss difference?
Answer: People who frequently log workouts may have higher motivation levels that drive both app use and weight loss
When you encounter studies claiming that one factor "causes" another, always examine whether the researchers have properly established causation versus correlation. The most fundamental threat to causal conclusions is the presence of confounding variables—third factors that influence both the supposed cause and the effect.
Choice A correctly identifies the most serious flaw: motivation level is likely a confounding variable that drives both frequent app usage AND weight loss success. Highly motivated people naturally log workouts more often and also stick to diet and exercise plans more consistently. The app usage itself may not be causing the weight loss—both behaviors might stem from the same underlying motivation.
Choice B describes a real limitation, but controlling for demographics wouldn't address the core causation problem. Even with matched groups by age and weight, the motivation confound would remain. Choice C points out data quality issues that could affect the study's accuracy, but this doesn't challenge the fundamental logic of the causal claim—it's more about measurement error. Choice D questions the practical significance of the results, but this doesn't address whether the relationship is truly causal.
The key insight is that correlation studies can never prove causation when alternative explanations exist. Here, the "third variable problem"—where motivation influences both app use and weight loss—provides a more plausible explanation than direct causation.
Study tip: When evaluating causal claims, always ask "What else could explain both variables?" The most serious flaws usually involve confounding variables, not just methodological limitations.
- The group claims the emission standards successfully improved air quality. What information would be most valuable for evaluating this claim?
Answer: Pollution measurements from similar cities without new emission standards during the same time period to control for seasonal and regional factors.
A control group of similar cities without emission standards would help determine if the 22% reduction was due to the standards or other factors (seasonal changes, regional weather patterns, economic conditions). This is the strongest way to establish causation. Choice B focuses on mechanism rather than establishing causation. Choice C addresses one potential confounding factor but control cities would handle this and other factors simultaneously. Choice D identifies some confounding variables but control cities would be more comprehensive.
- A school principal claims that students who eat breakfast score 12 points higher on standardized tests than those who skip breakfast. This conclusion was based on comparing test scores of 45 breakfast-eating students (mean: 78) with 38 students who skipped breakfast (mean: 66). What additional information is most critical for evaluating this claim?
Answer: The demographic characteristics of students in each group to determine if factors other than breakfast could explain the difference.
The most critical missing information is whether the groups differ in other important ways (socioeconomic status, sleep habits, family support, etc.) that could explain the score difference. This observational study cannot establish causation without controlling for confounding variables. Choice A is important but secondary to establishing whether breakfast is actually the cause. Choices C and D assume breakfast is the cause and focus on mechanism rather than establishing causation.
- A researcher claims that students who eat breakfast score higher on standardized tests. The study included 200 students: 120 who regularly eat breakfast (mean score: 78) and 80 who skip breakfast (mean score: 72). The standard deviation for both groups was 15 points. What additional information is most crucial for evaluating this claim?
Answer: Whether other factors like socioeconomic status were controlled for in the analysis
While all factors matter, controlling for confounding variables like socioeconomic status is most crucial. Students who eat breakfast regularly may come from families with more resources, better home environments, or different cultural values around education - any of which could explain the score difference. Choice A addresses statistical significance but doesn't address causation. Choice B is important but less critical than controlling for confounders. Choice D addresses measurement validity but assumes the test is already standardized and reliable.
- A social media company reports that their new algorithm increased user engagement by 18% based on comparing average daily time spent on the platform before and after the change. The analysis used data from 2.3 million active users over a 6-week period. Despite the large sample size, what is the primary limitation in interpreting this engagement increase?
Answer: The 6-week observation period may be too short to distinguish between temporary novelty effects and sustained engagement improvements.
When evaluating study results that claim to show improvement, you need to distinguish between genuine long-term effects and temporary changes that might fade over time. This is especially critical when analyzing behavioral interventions like algorithm changes.
The correct answer is C because 6 weeks is insufficient to determine whether the 18% engagement increase represents lasting behavioral change or just a temporary "novelty effect." When platforms introduce new features or algorithms, users often initially engage more due to curiosity, changed content recommendations, or simply noticing something different. However, this heightened engagement frequently diminishes as users adapt to the changes and return to baseline behaviors. To claim genuine improvement, you'd need months of data to establish that the increase persists beyond the initial adjustment period.
Option A is incorrect because seasonal trends would affect engagement regardless of algorithm changes, and the comparison appears to be measuring before-versus-after changes during the same timeframe. Option B misunderstands statistical principles—with 2.3 million users, individual variations would actually average out effectively, making the sample size quite robust for detecting real effects. Option D suggests measurement issues, but there's no indication in the question that the company changed how they track engagement time.
Remember this pattern: when evaluating any intervention study, always consider the time dimension. Short observation periods can't distinguish between temporary novelty effects and sustainable improvements. Look for this issue especially in behavioral studies, where initial enthusiasm often masks the true long-term impact.
