- A systematic review includes 12 studies investigating a treatment effect. A funnel plot shows that smaller studies with non-significant results appear to be missing from the literature, while larger studies show a range of results. What statistical approach would be most appropriate to quantitatively assess this observation?
Answer: Egger's test for publication bias
When you encounter questions about missing studies in systematic reviews, you're dealing with publication bias detection. The key clue here is "smaller studies with non-significant results appear to be missing" — this classic pattern suggests that negative results from small studies weren't published, creating an asymmetric funnel plot.
Egger's test (B) is specifically designed to quantitatively detect this asymmetry in funnel plots. It uses linear regression to test whether smaller studies (with larger standard errors) show systematically different effect sizes than larger studies. When publication bias exists, you'll see a significant relationship between study precision and effect size, indicating missing studies.
Cochran's Q test (A) measures whether studies show more variation in results than expected by chance alone — it detects heterogeneity between studies, not missing studies. The I² statistic (D) quantifies the same heterogeneity as a percentage, telling you what proportion of variation comes from true differences rather than sampling error. Both are about existing studies disagreeing with each other, not about studies being absent. Forest plots (C) are visualization tools that display individual study results and overall effects, but they don't quantitatively assess publication bias.
Remember this pattern: when you see "missing studies," "asymmetric funnel plot," or "small studies with negative results absent," think publication bias detection. Egger's test is the gold standard quantitative approach, while Begg's test is an alternative. Don't confuse publication bias (missing studies) with heterogeneity (existing studies showing different results).
- A researcher conducts a study with 200 participants and finds a correlation between variables X and Y of r = 0.18 (p = 0.045). To increase the sample size, they collect data from 100 additional participants. In the combined dataset (n = 300), the correlation becomes r = 0.15 (p = 0.08). Which result should they report?
Answer: Both results, explaining the data collection process transparently
When you encounter questions about reporting research results, especially when data collection occurs in phases, you're dealing with scientific transparency and ethical reporting standards. The key principle is that researchers must report their complete methodology and all relevant findings.
The correct approach is to report both results with full transparency about the data collection process (C). This allows readers to understand how the study evolved and make informed judgments about the findings. The original analysis with n=200 showed r=0.18 (p=0.045), suggesting a statistically significant but weak correlation. After adding 100 participants, the correlation weakened to r=0.15 (p=0.08), losing statistical significance. Both pieces of information are scientifically valuable and should be disclosed.
Option A is problematic because reporting only the significant result while hiding the additional data constitutes selective reporting bias - a serious ethical violation that can mislead the scientific community. Option B fails because it ignores the sequential nature of data collection and doesn't explain why the sample size changed, potentially confusing readers about the study design. Option D represents effect size cherry-picking, which violates principles of honest reporting by selecting results based on strength rather than methodological completeness.
The change from significant to non-significant results with increased sample size actually provides important information about the robustness (or lack thereof) of the original finding. This scenario illustrates why effect sizes near significance thresholds require careful interpretation.
Study tip: Remember that transparency in reporting methodology and results is always preferable to selective reporting, even when results become less favorable with additional data.
- A researcher analyzes survey data and finds that income is not significantly associated with health satisfaction (p = 0.12). They then decide to examine the relationship separately for men and women, finding significant associations in women (p = 0.03) but not men (p = 0.67). They report: 'Income significantly predicts health satisfaction in women.' What practice does this exemplify?
Answer: Data dredging through post-hoc stratification
When researchers conduct statistical analyses, the timing and justification for subgroup analyses is crucial for maintaining scientific integrity. This question tests your ability to distinguish between appropriate analytical strategies and problematic data mining practices.
The scenario describes a classic case of data dredging (also called p-hacking or fishing expeditions). The researcher initially found no significant overall association between income and health satisfaction (p = 0.12). Rather than accepting this null result, they then searched through subgroups until finding a significant result in women. This post-hoc exploration without prior hypothesis or theoretical justification exemplifies data dredging, making B correct.
A is wrong because there's no mention of biological rationale driving the subgroup analysis. The decision appears purely motivated by the initial non-significant result, not by theoretical considerations about sex-based differences.
C is incorrect because effect modification (interaction) should be planned and tested formally using interaction terms in statistical models, not discovered through post-hoc subgroup hunting after finding null overall results.
D is wrong because while exploratory analysis has its place, reporting a post-hoc subgroup finding as a definitive conclusion ("Income significantly predicts health satisfaction in women") without acknowledging the exploratory nature crosses into inappropriate territory.
Study tip: Watch for scenarios where researchers find non-significant results, then slice their data various ways until something becomes significant. Legitimate subgroup analyses should be pre-specified with clear scientific rationale, not fishing expeditions triggered by disappointing overall results.
- A pharmaceutical company conducts 4 clinical trials for a new drug. Trial 1 shows p = 0.02, Trial 2 shows p = 0.15, Trial 3 shows p = 0.08, and Trial 4 shows p = 0.03. In their marketing materials, they state: 'Clinical trials demonstrate significant efficacy (p < 0.05).' What aspect of this statement is most problematic from a transparency perspective?
Answer: The statement implies consistent evidence while ignoring mixed results
When evaluating research transparency and ethical reporting, you need to consider whether the presented information gives readers a complete and honest picture of the evidence. This question tests your understanding of selective reporting and cherry-picking issues in scientific communication.
The pharmaceutical company's statement is problematic because it creates a misleading impression by emphasizing only the favorable results. While technically true that some trials showed significance (p < 0.05), the statement "clinical trials demonstrate significant efficacy" suggests consistent, strong evidence across studies. In reality, the results are mixed: two trials were significant (p = 0.02, p = 0.03) while two were not (p = 0.15, p = 0.08). This selective emphasis while ignoring contradictory findings violates transparency principles and can mislead healthcare providers and patients about the drug's true efficacy profile.
Looking at the other options: (A) is incorrect because while reporting exact p-values would be more informative, the core transparency problem isn't the level of detail but the selective presentation. (C) addresses multiple testing corrections, which is a valid statistical concern but not the primary transparency issue highlighted here. (D) is wrong because statistical information in marketing can be appropriate when presented honestly and completely.
Remember that transparency in biostatistics isn't just about statistical accuracy—it's about presenting the complete picture. When you see research claims, always ask: "What information might be missing?" Look for patterns where only favorable results are highlighted while unfavorable or mixed findings are downplayed or omitted entirely.
- A pharmaceutical company conducts 25 different analyses on their clinical trial data, including subgroup analyses, different time points, and various outcome measures. They find 2 analyses with p < 0.05 and apply the Bonferroni correction, resulting in an adjusted significance level of 0.002. Both significant results remain significant after correction (p = 0.001 and p = 0.0015). However, the company's initial protocol specified only 5 primary analyses. What is the most concerning aspect of this scenario?
Answer: Post-hoc analyses were conducted and corrected as if they were planned comparisons
The main issue is that 20 analyses (25 total minus 5 planned) were conducted post-hoc but treated as if they were pre-planned when applying corrections. This represents a form of p-hacking where researchers conduct many unplanned analyses and then apply statistical corrections to appear rigorous. Choice A incorrectly criticizes the correction method. Choice B incorrectly assumes the results are valid because they survived correction. Choice D focuses on sample size rather than the analytical approach issue.
- A meta-analysis protocol specifies inclusion of all randomized controlled trials published between 2010-2020. During the search process, researchers identify 15 relevant studies but exclude 3 studies with non-significant results, stating they had 'methodological limitations' not mentioned in their exclusion criteria. The final meta-analysis of 12 studies shows a significant treatment effect. What is the primary methodological concern?
Answer: Post-hoc exclusions based on results introduce selection bias
When you encounter meta-analysis questions, focus on whether researchers followed their pre-specified protocol and avoided bias that could distort results.
The critical issue here is that researchers deviated from their original inclusion criteria by excluding studies after seeing their results. They initially planned to include all relevant RCTs from 2010-2020, but then removed three studies with non-significant findings, claiming "methodological limitations" that weren't part of their original exclusion criteria. This is classic selection bias—systematically excluding studies based on their outcomes rather than predetermined quality standards.
Option B correctly identifies this post-hoc exclusion problem. When you exclude studies after seeing they don't support your hypothesis, you artificially inflate the treatment effect and compromise the meta-analysis's validity.
Option A is wrong because a 10-year search window is typically adequate for evidence synthesis. The timeframe itself isn't the methodological flaw. Option C misses the point—12 studies can provide reliable meta-analysis results depending on their quality and sample sizes. The number alone isn't problematic. Option D sounds reasonable but misses the core issue. While methodological assessment is important, the problem isn't when quality was assessed, but that quality concerns were invented post-hoc to justify excluding inconvenient results.
Remember this pattern: In systematic review and meta-analysis questions, always check whether researchers stuck to their original protocol. Post-hoc changes to inclusion criteria, especially ones that seem to favor positive results, are major red flags for selection bias.
- A systematic review identifies 15 published studies and 8 unpublished studies (from trial registries and conference abstracts) testing the same intervention. The published studies show a significant pooled effect (p = 0.01), while including the unpublished studies yields a non-significant result (p = 0.18). The review authors decide to include only the published studies, arguing that unpublished studies may have quality issues. What is the most appropriate interpretation of this decision?
Answer: Problematic, as it likely introduces publication bias and may lead to false positive conclusions
When you encounter questions about systematic reviews and meta-analyses, always consider how study selection decisions might introduce bias, particularly publication bias—the tendency for studies with significant results to be published while those with null or negative results remain unpublished.
The correct answer is D because excluding unpublished studies based solely on their publication status creates a classic case of publication bias. Notice the pattern here: published studies showed significance (p = 0.01) while including unpublished studies made the effect non-significant (p = 0.18). This strongly suggests the unpublished studies had negative or null results that were never published, and excluding them artificially inflates the apparent treatment effect.
Choice A is wrong because publication status doesn't guarantee quality—many unpublished studies from trial registries may actually have rigorous methodology. Choice B misunderstands best practices; high-quality systematic reviews should actively seek unpublished studies to minimize bias. Choice C is incorrect because unpublished studies can often be assessed for quality, especially those from trial registries or conference presentations where methodological details are available.
The review authors' reasoning—that unpublished studies "may have quality issues"—is a red flag. If they were truly concerned about quality, they should assess each study individually rather than making blanket exclusions based on publication status.
Study tip: In systematic review questions, be suspicious whenever researchers exclude studies based on publication status alone, especially when it changes the conclusion from non-significant to significant. This pattern screams publication bias.
- A researcher plans to conduct pairwise comparisons between 5 treatment groups using t-tests. Without any adjustment for multiple comparisons, how many individual tests will be performed, and what is the family-wise error rate if each test uses α = 0.05?
Answer: 10 tests; family-wise error rate = 0.40
When you encounter questions about multiple comparisons, you need to consider two key calculations: the number of pairwise tests and the cumulative probability of making at least one Type I error across all tests.
For 5 treatment groups, the number of unique pairwise comparisons follows the combination formula: (25)=2!(5−2)!5!=25×4=10 tests. Each group must be compared with every other group exactly once.
The family-wise error rate (FWER) represents the probability of making at least one Type I error across all tests. With independent tests, this is calculated as: FWER=1−(1−α)k where α = 0.05 and k = 10 tests. So: FWER=1−(0.95)10=1−0.599=0.401≈0.40
Answer A incorrectly calculates only 5 tests, likely confusing the number of groups with the number of comparisons needed. Answer C uses the correct number of tests but miscalculates the FWER, possibly using a simpler but incorrect formula like 10 × 0.05. Answer D shows 15 tests, which would be correct for 6 groups, not 5, though the FWER calculation appears roughly consistent with that incorrect number.
Remember this pattern: for n groups, you need (2n) comparisons, and the FWER grows exponentially with the number of tests. This is why multiple comparison corrections like Bonferroni are essential in practice to control Type I error inflation. - A clinical trial protocol specifies that the primary endpoint will be analyzed when 200 patients complete the study. After enrolling 150 patients, the investigators notice the treatment effect is not as strong as expected. They decide to change the primary endpoint to a different outcome measure that shows more promise in the current data. What type of research practice does this represent?
Answer: Outcome switching that compromises study validity
When you encounter questions about changes made to study protocols after data collection has begun, focus on the principle of scientific integrity and the potential for bias introduction.
Why C is correct: This scenario represents outcome switching, a serious threat to study validity. The investigators changed their primary endpoint mid-study specifically because the original endpoint wasn't showing favorable results. This constitutes "cherry-picking" outcomes based on observed data trends, which can dramatically inflate the apparent treatment effect and compromise the study's credibility. The timing is crucial here—they made this change after seeing disappointing results in 150 patients, not based on legitimate scientific or safety concerns.
Why the other answers are wrong: A) Adaptive trial designs involve pre-specified rules for modifications that are built into the original protocol, not post-hoc changes based on unfavorable interim results. B) Protocol amendments for patient safety would involve safety endpoints or adverse events, not switching to a "more promising" efficacy outcome. D) Standard interim analyses follow predetermined stopping rules and statistical boundaries—they don't involve changing the primary endpoint because results aren't as strong as hoped.
Key study tip: Remember that legitimate protocol modifications must be pre-specified or based on safety concerns, never on the desire to find more favorable results. When you see scenarios describing endpoint changes after interim data review, particularly when motivated by disappointing efficacy results, this almost always represents problematic outcome switching that undermines the study's validity and introduces bias.
- A journal requires authors to submit their raw data and analysis code along with manuscripts. A submitted study shows a significant result (p = 0.048), but when the journal's statistician runs the provided code, they find that the authors tested 8 different model specifications and reported only the one that achieved significance. The other 7 models had p-values ranging from 0.12 to 0.67. What should be the journal's primary concern?
Answer: The result is likely a false positive due to undisclosed multiple testing and selective reporting
The authors engaged in p-hacking by testing multiple models and reporting only the significant one, while hiding the non-significant results. This dramatically increases the probability of a false positive result. Choice B focuses on model appropriateness rather than the selective reporting issue. Choice C incorrectly suggests that p-values close to 0.05 are inherently unreliable. Choice D misinterprets the variation as indicating data problems rather than recognizing the selective reporting issue.
- A research team pre-registers a study protocol specifying that they will analyze the relationship between exercise frequency and depression scores using linear regression. After data collection, they discover the relationship is non-linear and decide to use polynomial regression instead, finding a significant result (p = 0.02). How should they report this finding?
Answer: Report both linear and polynomial results with transparency about the protocol deviation
When you encounter questions about pre-registered study protocols and post-hoc analysis changes, you're dealing with research transparency and scientific integrity principles. Pre-registration is designed to prevent p-hacking and selective reporting, but researchers must balance protocol adherence with appropriate statistical methods.
The correct approach here is Option B - reporting both the pre-registered linear regression and the post-hoc polynomial regression with full transparency about the protocol deviation. This maintains scientific integrity by showing readers both what was planned and what was actually done, allowing them to evaluate potential bias. The polynomial regression may indeed be more appropriate for the data, but the deviation from protocol must be acknowledged.
Option A fails because simply reporting the polynomial regression without mentioning the protocol deviation conceals important methodological information that could affect interpretation. Option C is problematic because rigidly adhering to an inappropriate statistical method (linear regression for non-linear data) compromises the validity of your findings and wastes valuable data. Option D represents research misconduct - retroactively changing pre-registrations defeats their entire purpose and constitutes falsification of the research record.
Study tip: Remember that pre-registration doesn't lock you into inappropriate methods forever, but any deviations must be transparently reported. On biostatistics exams, questions about research integrity often test whether you understand that transparency trumps convenience - always choose the option that maintains scientific honesty even when it's more work to report.
- A researcher analyzing survey data discovers that their primary hypothesis yields p = 0.07. They then examine the data using different variable transformations: log transformation gives p = 0.03, square root transformation gives p = 0.09, and standardization gives p = 0.04. They report the log transformation result as their primary finding. What best describes this practice?
Answer: Analytical flexibility that inflates Type I error
This question tests your understanding of p-hacking and multiple testing problems in statistical analysis. When researchers test multiple analytical approaches and selectively report the most favorable result, they're engaging in questionable research practices that compromise statistical validity.
The scenario describes a classic case of analytical flexibility inflating Type I error. The researcher started with a non-significant result (p = 0.07) and tried various transformations until finding one that achieved significance (p = 0.03 with log transformation). By testing multiple approaches and cherry-picking the significant result, they've essentially conducted multiple statistical tests without adjusting for this multiplicity. This inflates the probability of finding a false positive result well above the nominal α = 0.05 level, making answer C correct.
Answer A is wrong because appropriate model selection should be driven by theoretical considerations and data characteristics determined before analysis, not by which method yields significant results. Answer B incorrectly frames this as standard practice—while exploratory analysis is legitimate, reporting exploratory results as confirmatory findings without acknowledging the multiple testing is problematic. Answer D misses the point entirely; even if the data were non-normal and transformations were justified, the issue here is the selective reporting of results based on statistical significance rather than pre-planned analytical strategy.
Remember: whenever you see researchers trying multiple analytical approaches and reporting only the significant result, think "p-hacking." Legitimate data transformation should be justified by the data's distributional properties and planned in advance, not selected post-hoc based on p-values.
- A researcher tests the association between a biomarker and disease outcome using three different statistical approaches: logistic regression, Cox proportional hazards model, and Mann-Whitney U test. The p-values are 0.08, 0.04, and 0.12, respectively. The researcher reports only the Cox model results, stating it is the 'most appropriate' method without justification. What is the main concern with this reporting strategy?
Answer: The choice of statistical method appears to be result-driven
This question tests your understanding of research integrity and proper statistical reporting practices. When researchers apply multiple statistical methods to the same data, the method selection should be predetermined based on study design and data characteristics, not chosen after seeing which produces the most favorable results.
The correct answer is C because the researcher's behavior suggests "p-hacking" or result-driven analysis. Running three different tests and then selecting only the one that yielded a significant p-value (0.04) while dismissing the non-significant results (0.08 and 0.12) without methodological justification represents questionable research practice. The claim that Cox modeling is "most appropriate" appears to be reverse-engineered from the results rather than based on a priori statistical reasoning.
Option A is incorrect because while Cox model assumptions should indeed be verified, the primary concern here isn't about assumption checking—it's about the selective reporting pattern. Option B is wrong because multiple testing correction only applies when you're genuinely testing multiple hypotheses simultaneously by design, not when you're choosing between fundamentally different analytical approaches for the same research question. Option D misses the point entirely—sample size adequacy, while important for Cox modeling, doesn't address the core ethical issue of cherry-picking significant results.
Remember this key principle: statistical method selection should always be justified by your research question, data structure, and study design—never by which approach gives you the p-value you want. This type of selective reporting undermines scientific integrity and inflates Type I error rates.
- What is the most accurate assessment of Dr. Martinez's approach to statistical analysis and reporting?
Answer: Problematic, because she failed to apply multiple comparison corrections and engaged in selective result presentation
While Dr. Martinez mentioned testing 20 outcomes, she failed to adjust for multiple comparisons (which would reduce the chance of false positives) and selectively presented only significant results. Simply disclosing the number of tests doesn't address the statistical or transparency issues. Choice A incorrectly suggests disclosure alone is sufficient. Choice C incorrectly endorses selective reporting as standard practice. Choice D focuses on sample size rather than the core issues of multiple testing and selective reporting.
- A meta-analysis includes 8 studies that tested the same intervention. The authors note that they identified 12 relevant studies initially, but 4 studies 'did not meet inclusion criteria' - though these 4 studies all showed non-significant results and used similar methodologies to the included studies. The meta-analysis concludes the intervention is effective (p = 0.03). What is the most likely explanation for this result?
Answer: Publication bias and selective study inclusion created a false positive result
The suspicious exclusion of 4 studies with similar methodologies but non-significant results suggests selective inclusion based on results rather than legitimate methodological criteria. This creates publication bias and likely inflates the effect size, making a false positive result most probable. Choice A ignores the concerning pattern of exclusions. Choice C accepts the authors' justification despite the suspicious pattern. Choice D doesn't account for the systematic nature of the exclusions based on results.
- A pharmaceutical company conducts 8 Phase II trials for a new medication across different populations. Three trials show statistically significant benefits (p < 0.05), while five show no significant effect. In their FDA submission, they emphasize the three positive trials and briefly mention the negative ones in an appendix. From a regulatory perspective, what is the most serious issue with this approach?
Answer: The selective emphasis may misrepresent the overall evidence
When you encounter questions about research reporting and regulatory submissions, focus on the fundamental principle of scientific integrity: presenting a complete and balanced view of all available evidence.
The most serious regulatory concern here is selective emphasis that misrepresents the overall evidence (C). When a company highlights 3 positive trials while downplaying 5 negative ones, they're creating a misleading impression of the drug's effectiveness. Regulatory agencies like the FDA need to see the complete picture—both successes and failures—to make informed decisions about public safety. This selective presentation violates the principle that all conducted studies should be given appropriate weight in regulatory submissions, regardless of their outcomes.
Looking at the other options: (A) assumes inadequate statistical power without evidence—some trials may have been properly powered, and this doesn't address the core reporting issue. (B) suggests population heterogeneity is the main problem, but diverse populations are often desirable in Phase II trials to understand drug performance across different groups. (D) implies the study design was flawed, but conducting multiple smaller trials instead of one large trial is often a valid Phase II approach that allows for adaptive learning.
Remember this key principle: in biostatistics and regulatory affairs, questions about study reporting often test whether you recognize threats to scientific integrity. The most serious issues typically involve selective presentation, publication bias, or incomplete disclosure—not just statistical methodology. Always consider whether the approach gives regulators and clinicians an honest, complete view of the evidence.
- What is the most significant methodological concern with Dr. Chen's approach?
Answer: She conducted an unplanned analysis after seeing the initial results, which constitutes p-hacking
When you encounter research scenarios involving multiple statistical tests and post-hoc analyses, focus on the timing and planning of statistical decisions. The integrity of statistical inference depends heavily on whether analyses were planned before seeing the data.
Dr. Chen's critical error is conducting the pooled analysis (treatments A + B vs. placebo) after observing her initial results. This represents p-hacking or "data dredging" – the practice of performing additional analyses until finding a statistically significant result. Since she saw that Treatment A was significant (p = 0.03) but Treatment B was borderline (p = 0.06), she essentially went "fishing" for a better p-value by combining the groups. This invalidates the statistical inference because the analysis wasn't pre-specified, making answer C correct.
A is incorrect because multiple comparison adjustments, while important, aren't the primary issue here – her original three planned comparisons could be justified without adjustment depending on the research context.
B is wrong because sample size adequacy isn't the main methodological flaw; the study detected a significant difference for Treatment A, suggesting adequate power for at least some comparisons.
D is incorrect because separate t-tests for planned comparisons are methodologically acceptable, though ANOVA might be preferred. This doesn't represent the most significant concern.
Study tip: Always distinguish between pre-planned and post-hoc analyses in research scenarios. If a researcher changes their analytical approach after seeing results, suspect p-hacking – this is a major validity threat that biostatistics questions frequently test.
- A researcher conducts a study examining the relationship between 15 different dietary supplements and cognitive performance. After finding that 2 supplements showed statistically significant improvements (p < 0.05), the researcher publishes only these positive results without mentioning the other 13 supplements tested. Which combination of methodological issues is most problematic in this scenario?
Answer: Multiple testing without correction and selective reporting of results
This scenario demonstrates both multiple testing without appropriate correction (testing 15 supplements increases the chance of false positives) and selective reporting (publishing only positive results while hiding negative ones). Choice B incorrectly focuses on sample size rather than the transparency issue. Choice C mentions publication bias but misses the multiple testing problem and incorrectly identifies randomization as the issue. Choice D focuses on study design issues rather than the statistical and reporting problems.
- A research team publishes a study showing a significant treatment effect (p = 0.02). Later, another team attempts to replicate the study with a larger sample size and finds a non-significant result (p = 0.18) with a similar effect size. The original authors claim the replication 'failed' because it didn't achieve significance. What is the most accurate assessment of this situation?
Answer: The replication succeeded because it found a similar effect size
When evaluating research replications, you need to distinguish between statistical significance and effect size - two fundamentally different concepts that students often conflate.
The key insight here is that a successful replication reproduces the underlying effect, not necessarily the p-value. Both studies found similar effect sizes, meaning they detected the same magnitude of treatment benefit. The difference in statistical significance simply reflects the inherent variability in p-values across studies, even when the true effect remains constant.
Answer B is correct because finding a similar effect size indicates the replication successfully detected the same underlying phenomenon. Statistical significance is just one piece of evidence, and p-values naturally vary between studies due to sampling variation.
Answer A falls into the common trap of equating replication success with achieving significance. This misconception ignores that p-values fluctuate around the significance threshold even when effects are real. Answer C jumps to an unwarranted conclusion - while false positives are possible, the similar effect sizes actually support the original finding's validity. Answer D incorrectly suggests that larger sample sizes are problematic, when they actually provide more precise estimates and are methodologically superior.
The larger sample in the replication likely provided a more precise effect estimate, even though random variation pushed the p-value above 0.05. Remember: focus on effect sizes and confidence intervals when evaluating replications, not just whether studies cross the arbitrary p = 0.05 threshold. Successful science replicates effects, not p-values.
- In a meta-analysis of 20 studies examining a new drug's effectiveness, researchers find that studies with significant results (p < 0.05) are more likely to be published in high-impact journals, while studies with non-significant results are often unpublished or appear in lower-tier journals. This pattern most directly threatens which aspect of the meta-analysis?
Answer: The representativeness of the pooled effect estimate
When you encounter questions about systematic bias in research synthesis, focus on how different types of bias affect the validity and representativeness of meta-analytic results.
The scenario describes classic publication bias - the tendency for studies with statistically significant, positive results to be published more frequently and in higher-impact venues than studies with null or negative findings. This creates a fundamental sampling problem for meta-analyses.
Answer D is correct because publication bias directly compromises the representativeness of the pooled effect estimate. When a meta-analysis predominantly includes published studies with significant results while missing unpublished null studies, the combined effect size becomes artificially inflated. The pooled estimate no longer represents the true population effect - it represents a biased sample skewed toward positive findings.
Answer A is incorrect because publication bias doesn't affect the statistical power of individual studies that are already completed. Power is determined during study design, not publication decisions.
Answer B misses the mark because external validity refers to generalizability across populations, settings, or conditions. While publication bias affects what studies contribute to the meta-analysis, it doesn't directly threaten whether results apply to different contexts.
Answer C is wrong because internal validity concerns whether individual studies accurately measure causal relationships within their own design. Publication bias affects which studies get included in the synthesis, not the quality of the studies themselves.
Study tip: Remember that publication bias creates a "file drawer problem" - significant results get published while null results stay hidden. This sampling bias distorts meta-analytic effect estimates, making them appear larger than the true population effect.
