- In a study on decision-making, a researcher predicts that participants will be more risk-averse after viewing a sad film clip. The initial analysis shows a non-significant result (p = .12). The researcher notices high variance in the data and decides to exclude the 15% of participants with the slowest reaction times, justifying this by arguing they were inattentive. After this exclusion, the result is significant (p = .04).
Answer: The procedure is questionable because the exclusion rule was implemented after seeing its impact on the p-value.
While it can be legitimate to exclude data based on pre-defined criteria (e.g., 'we will exclude participants with RTs 3 SDs above the mean'), the problem here is that the decision was made post-hoc and was clearly motivated by the desire to achieve significance. This is a form of p-hacking. The researcher is capitalizing on chance by finding a justification to remove data points that are inconvenient for their hypothesis. The core issue is not the criterion itself (D) or the proportion (C), but the data-contingent way in which the criterion was applied.
- A social psychologist finds a non-significant correlation between social media use and depression (r = .15, p = .20). The psychologist then re-runs the analysis, this time controlling for age and socioeconomic status. This second model yields a significant partial correlation (r = .22, p = .04). The final paper only reports the significant result from the second model, arguing it is more precise.
Answer: The reported p-value is likely misleading because the analytical strategy was chosen after seeing the data.
This is an example of p-hacking through analytical flexibility. While controlling for covariates can be a valid approach, doing so only after an initial analysis fails to yield significance—and then reporting only the significant model—is a QRP. The choice of analysis was contingent on the outcome, which inflates the false positive rate. The reported p-value of .04 does not reflect the actual probability of a Type I error, which is higher due to the undisclosed analytical flexibility.
- A researcher conducts an exploratory study on personality and career choice, collecting data on hundreds of variables. They discover an unexpected, strong correlation between conscientiousness and preference for jobs with clear hierarchies. In the introduction to their manuscript, they present a detailed theory of how conscientiousness predisposes individuals to seek structured environments, framing their analysis as a direct test of this theory.
Answer: It inappropriately presents an exploratory finding as if it were a confirmatory test of a prior hypothesis.
This is a classic example of HARKing (Hypothesizing After the Results are Known). The core problem is the misrepresentation of the research process. An exploratory finding, discovered without an a priori hypothesis, is presented as if it were the result of a confirmatory test of a pre-existing theory. This practice distorts the evidence and makes the finding seem much stronger than it is. While selective reporting (D) might also have occurred, HARKing specifically refers to the retrofitting of a hypothesis.
- A researcher pre-registers her hypothesis that a new intervention will reduce anxiety. After running the study, she finds the primary outcome is not significant (p = .07). In her paper's discussion, she writes, "Although the result did not reach statistical significance, a strong trend in the predicted direction was observed. This suggests the hypothesis is likely correct and the study was merely underpowered."
Answer: This interpretation is a questionable practice that 'spins' a non-significant result to seem supportive.
Even with pre-registration, researchers can engage in 'spin' by misinterpreting results. A p-value of .07 means that if the null hypothesis were true, there is a 7% chance of observing a result at least as extreme as the one found. The established convention is that this does not provide sufficient evidence to reject the null hypothesis. Describing it as a 'strong trend' or as evidence that the hypothesis is 'likely correct' is a QRP that misrepresents the statistical evidence to fit a desired narrative. While a power issue might exist, this interpretation is a form of conclusion-distorting spin.
- A team of researchers publicly posts their study plan before collecting any data. The plan specifies their primary hypothesis, the target sample size with a power analysis, and the exact statistical models they will use to test the hypothesis. This practice is known as pre-registration.
Answer: It primarily prevents HARKing and p-hacking by constraining analytical flexibility.
Pre-registration is an open science practice designed to increase transparency and reduce bias. By committing to a hypothesis and analysis plan before seeing the data, researchers cannot engage in HARKing (Hypothesizing After the Results are Known) or p-hacking (trying multiple analyses and reporting only the significant one). It draws a clear line between confirmatory and exploratory research. While it can help with media interpretation or publication (e.g., in a Registered Report), its primary purpose is to prevent QRPs that distort statistical conclusions.
- A journal uses a publication model called a "Registered Report." Researchers submit their introduction, methods, and analysis plan for peer review before collecting data. If the proposal is accepted, the journal commits to publishing the article regardless of the study's outcome.
Answer: It eliminates publication bias by making the statistical significance of the results irrelevant to the publication decision.
When you encounter questions about research methodology and publication practices, focus on understanding how different systems address scientific bias and reliability. Registered Reports represent a major reform in scientific publishing designed to tackle specific problems in research.
The key insight here is understanding publication bias - the tendency for journals to preferentially publish studies with statistically significant, positive results while rejecting studies with null or negative findings. This creates a distorted scientific literature that doesn't reflect the true state of knowledge in a field.
Registered Reports solve this by separating the publication decision from the study's results. Since the journal commits to publishing before seeing the outcomes, statistical significance becomes irrelevant to whether the work gets published. This eliminates the bias toward positive results, making option D correct.
Let's examine why the other options miss the mark. Option A is wrong because Registered Reports don't filter for novelty or surprising hypotheses - they evaluate methodological rigor regardless of how exciting the hypothesis seems. Option B incorrectly focuses on speed; while the process is split into stages, this actually takes longer, not less time, since there are multiple review rounds. Option C makes false guarantees about statistical power and effect sizes - the format encourages good planning but doesn't guarantee these outcomes.
Remember that Registered Reports are fundamentally about improving scientific integrity, not efficiency or guaranteeing flashy results. When you see questions about publication practices, always consider how they address bias in the research pipeline.
- Which of the following actions is best classified as a questionable research practice (QRP) rather than outright scientific fraud?
Answer: Strategically dropping several 'inconvenient' data points from a dataset to achieve p < .05.
Scientific fraud typically involves fabrication (creating fake data, A) or falsification (altering data, B). QRPs are actions that distort findings but fall in a gray area short of clear fraud. Strategically removing outliers or data points is a classic QRP because the researcher is acting on real data but in a biased, undisclosed way. Submitting a manuscript to two journals (D) is a form of professional misconduct but is not a QRP related to the distortion of scientific conclusions from data.
- A researcher tests a new educational intervention with three different variations (A, B, and C) against a control group. The results show that variation A significantly outperforms the control group, but variations B and C do not. In the final manuscript, the researcher omits all mention of variations B and C and presents the study as a simple, successful comparison between intervention A and a control.
Answer: This practice misleads readers by creating a cleaner, more compelling narrative than the full data would support.
This question tests your understanding of research ethics and publication practices, specifically focusing on selective reporting and its impact on scientific integrity.
The researcher's decision to omit unsuccessful variations B and C creates a misleading narrative about the research process. By presenting only the successful comparison between variation A and control, the study appears as a straightforward success story rather than revealing the full picture: that two-thirds of the tested variations actually failed. This selective reporting gives readers an overly optimistic view of the intervention's effectiveness and suggests the research was more targeted and successful than it actually was. The correct answer is D because this practice fundamentally distorts the scientific narrative.
Let's examine why the other options are incorrect. Choice A is wrong because internal validity refers to whether the study design properly controls for confounding variables—omitting failed variations doesn't affect the quality of the A versus control comparison itself. Choice B is incorrect because effect size calculations for the A versus control comparison remain mathematically valid regardless of whether other variations are mentioned. Choice C overstates the severity—while this practice is ethically problematic, it's selective reporting rather than data falsification, since no actual data was changed or fabricated.
When you encounter research ethics questions, focus on how practices affect transparency and reader understanding. Selective reporting is a key issue in psychology research—it creates publication bias and gives an incomplete picture of what actually works, even when the reported results themselves are accurate.
- Dr. Anya Sharma publishes a surprising finding that a specific cognitive training exercise improves creativity. Dr. Ben Carter's lab conducts a high-powered, pre-registered direct replication of Dr. Sharma's study, following the original methodology precisely. Dr. Carter's study finds no significant effect.
Answer: The evidence for the original finding is substantially weakened, suggesting it could be a false positive or context-dependent.
A single failed direct replication does not prove fraud (A) or definitively falsify a broad theory (B). Given that the replication was high-powered and pre-registered, it is unlikely to be simply flawed (C). The most cautious and appropriate scientific interpretation is that the evidence for the original effect is now much weaker. The original finding might have been a Type I error (a false positive), or the effect might depend on subtle contextual variables not captured in the methods section and therefore not reproduced in the replication attempt.
- A 1970s study found that feeling anonymous (by wearing hoods and robes) increased participants' willingness to administer fake electric shocks. A modern researcher conducts a study finding that participants who are part of an anonymous online forum are more likely to use aggressive language. If the modern study is successful, it is best described as a conceptual replication.
Answer: It increases confidence in the generalizability of the theory linking anonymity to deindividuation and aggression.
A conceptual replication tests the same underlying hypothesis as an original study but uses different methods, populations, or operationalizations. The primary value of a successful conceptual replication is that it enhances the generalizability of the underlying theory—in this case, that anonymity promotes aggressive or antisocial behavior. It shows the principle is robust across different contexts. It does not directly test whether the original study was a false positive (A), nor does it provide a comparable effect size due to the different methods (B).
- A researcher analyzes a large public dataset without a specific hypothesis. She discovers that individuals who report eating breakfast daily also report higher levels of life satisfaction. Which of the following subsequent actions would be the clearest example of HARKing (Hypothesizing After the Results are Known)?
Answer: Writing the manuscript's introduction as if the study was originally designed to test a theory about breakfast's role in well-being.
HARKing is the act of presenting a post-hoc finding as if it were an a priori prediction. Actions A, B, and D represent good scientific practice. (A) is the proper way to follow up on an exploratory finding. (B) is the transparent way to report the finding itself. (D) is standard practice for contextualizing a finding. Action (C) is the only one that involves misrepresenting the discovery process by creating a new hypothesis to fit the existing data and pretending it came first.
- A researcher conducts an experiment on a new attention-enhancing supplement. They measure performance on five different cognitive tasks: sustained attention, selective attention, task switching, working memory, and response inhibition. The supplement only shows a statistically significant improvement (p = .04) on the task-switching measure. The researcher then publishes a paper titled "A New Supplement Specifically Enhances Cognitive Flexibility."
Answer: The research practice dramatically increases the probability of committing a Type I error.
This scenario describes a form of p-hacking where a researcher tests multiple dependent variables but only reports the one that yields a significant result. By conducting multiple statistical tests (in this case, five), the overall probability of finding at least one significant result by chance alone (a Type I error) increases well above the nominal alpha level of .05. The primary distortion is statistical, not an issue of causal mechanism, fraud, or construct validity.
- A psychologist is testing whether a brief mindfulness intervention reduces anxiety. After testing the first 40 participants, the p-value for the group difference is .08. The psychologist tests an additional 10 participants, and the p-value becomes .06. Ten more are added, and the p-value finally drops to .045. The psychologist stops data collection and reports a significant finding.
Answer: It capitalizes on random fluctuations in the data to obtain a significant result.
This practice, known as optional stopping or sequential testing, is a questionable research practice (QRP). It allows the researcher to continue collecting data until random chance produces a dataset that crosses the significance threshold. This inflates the Type I error rate because the decision to stop is dependent on the result, capitalizing on random noise rather than reflecting a true effect. While adding participants increases power (making C incorrect), the stopping rule is what makes the practice problematic.
- Consider the following sequence of studies: Study 1, an initial experiment, finds a large effect of a 'growth mindset' intervention on academic performance. Study 2, a large, multi-site, pre-registered direct replication, finds no effect on academic performance. Study 3, a conceptual replication, finds the intervention does increase students' self-reported persistence on difficult tasks.
Answer: The original effect on academic performance is likely not reliable, but the intervention might influence a related psychological process.
This scenario requires synthesizing multiple, seemingly conflicting, results. The high-quality direct replication (Study 2) failing to find an effect on performance casts serious doubt on the reliability of Study 1's main finding. However, Study 3 suggests the intervention is not inert; it affects a related construct (persistence). The most nuanced conclusion is that the originally claimed outcome (performance) may not be replicable, but the intervention has a different, more subtle effect (on persistence). This integrates all three findings without dismissing any of them outright.
- An initial study reports that a specific gene variant, GENE-A, is associated with superior memory (p = .02). A second, larger study attempts to replicate this and fails (p = .35). A subsequent massive genome-wide association study (GWAS) with 100,000 participants finds no link for GENE-A, but finds a robust association between a different variant, GENE-B, and memory.
Answer: The original finding was most likely a Type I error, and the cumulative evidence weighs against the claim.
This question requires integrating three pieces of evidence. The initial finding was significant but was followed by a failed replication. The massive GWAS, which has immense statistical power, not only failed to find the GENE-A link but found a different one. The most parsimonious conclusion is that the original, small study produced a false positive (Type I error). The idea that the effect is real but tiny (A) is directly contradicted by the powerful GWAS. A moderation effect (B) is a post-hoc complication that is less likely than the simpler explanation of a false positive.
- If questionable research practices such as p-hacking and selective reporting of positive results are widespread in a field, what is the most direct consequence for the published literature of that field?
Answer: The rate of false positives will be substantially higher than the nominal alpha level.
The nominal alpha level (usually .05) is the acceptable rate of Type I errors (false positives) under the assumption that the null hypothesis is true and all assumptions are met. QRPs like p-hacking and selective reporting systematically violate these assumptions, specifically by creating a system where non-significant results are either converted into significant ones or hidden from view. This directly inflates the proportion of published findings that are false positives, meaning the true Type I error rate in the literature is much higher than the stated 5%.
- An initial study on a new therapy reports a very large effect size (Cohen's d = 0.9). A subsequent, large-scale replication study confirms the therapy has a statistically significant effect, but finds a much smaller effect size (d = 0.25). Assuming both studies were conducted with high integrity, what is the most likely explanation for this discrepancy?
Answer: The initial, large effect size was likely an overestimation due to random sampling error combined with publication bias.
This pattern is often explained by the "winner's curse" and publication bias. Initial studies that get published are often those that, by chance, found a larger-than-average effect. Studies that by chance found a smaller (or null) effect are less likely to be published. Therefore, the first published effect size for a phenomenon is often an overestimate of the true effect. A large, high-powered replication provides a more precise estimate, which is typically smaller and closer to the true value. This is a systemic issue, not necessarily a flaw in either study.
- A meta-analyst creates a p-curve, which is a distribution of statistically significant p-values from a set of published studies on a single topic. The resulting graph shows a strong right skew. Refer to the graph below.
Answer: The evidential value of the literature is weak and likely inflated by p-hacking.
A p-curve for a literature with a true effect should be left-skewed, with more p-values close to 0 (e.g., .001, .01) than close to .05. The right-skewed curve shown in the graph, with a 'bump' of p-values just below the .05 threshold, is a classic sign of p-hacking. It suggests that researchers used questionable research practices to push non-significant results (e.g., p = .06) just over the line into significance (e.g., p = .045). This indicates that the evidential value is weak and the reported effects are likely inflated.
- A pharmaceutical company sponsors ten independent studies on its new anxiolytic drug. Two studies find a statistically significant benefit, while the other eight find no significant effect. The company only submits the two successful studies for publication.
Answer: It leads to a biased overestimation of the drug's true efficacy in the published literature.
This scenario describes the "file drawer problem," a form of publication bias. When only studies with statistically significant results are published, the public record becomes skewed. A meta-analysis of the published literature would suggest the drug is effective, substantially overestimating its true effect size because the null findings from the other eight studies are hidden. This practice is a QRP, not data fabrication (C), and while it makes replication context difficult (A), the primary consequence is the distortion of the evidence base (D).
- Even when researchers are acting in good faith and not engaging in QRPs, conducting a successful direct replication of a previous study can be challenging. Which of the following is a primary methodological reason for this difficulty?
Answer: Method sections in papers often omit crucial procedural details or information about the study's context.
While institutional factors like funding (B) and author cooperation (A) are real challenges, a core methodological reason for replication difficulty is the problem of "tacit knowledge." Method sections, due to space limitations and convention, cannot possibly describe every single detail of a procedure, the demeanor of the experimenter, or subtle characteristics of the sample or setting. A replication attempt may fail not because the original finding was false, but because the replication did not faithfully reproduce a critical, unstated element of the original procedure.
