- To establish the construct validity of a new self-report measure of 'emotional intelligence,' a researcher correlates scores on the new measure with scores from three existing scales: a well-validated emotional intelligence test, a measure of social desirability, and a test of general cognitive ability (IQ). Which pattern of correlations would provide the strongest evidence for the new measure's construct validity?
Answer: Strong positive correlation with the validated test, near-zero correlation with social desirability, and moderate positive correlation with IQ.
Establishing construct validity involves demonstrating both convergent and discriminant validity. Convergent validity is shown by a strong positive correlation with an existing, validated measure of the same construct. Discriminant (or divergent) validity is shown by a weak or near-zero correlation with measures of unrelated or distinct constructs. Social desirability should be unrelated. While some theories link EI and IQ, a moderate correlation is more plausible than a very strong one, showing they are related but distinct. Option A shows the best combination of these.
- Based on the information in the passage, the new survey most clearly demonstrates adequate _______ but poor _______.
Answer: content validity; criterion validity
The expert panel's agreement that the items cover the theoretical domain of engagement provides evidence for strong content validity. However, the survey's failure to correlate with relevant work outcomes (performance ratings, productivity) indicates poor criterion validity. The high Cronbach's alpha (.91) indicates good internal consistency, but this is not what's being asked about in the correct answer.
- A school district implements a new anti-bullying program in its middle schools. To identify the schools most in need, they select the 10 schools with the highest reported incidents of bullying in the previous year to receive the program. At the end of the school year, these 10 schools show a marked decrease in bullying incidents. The superintendent claims the program was a success. A methodologist might temper this conclusion by citing which threat to internal validity?
Answer: Regression to the mean
Regression to the mean is a statistical phenomenon where an extreme score on a first measurement tends to be closer to the average on a second measurement. By selecting the schools with the highest (most extreme) rates of bullying, it is statistically likely that their rates would have decreased somewhat in the following year even without any intervention. This makes it difficult to attribute the entire change to the program itself.
- A research team develops a new scale to measure 'scholarly grit.' The scale is administered to a group of graduate students twice, one month apart, and the scores show a high correlation (r = .89). However, scores on the scale do not correlate with existing measures of perseverance, conscientiousness, or long-term goal attainment. Which statement best describes the psychometric properties of this new scale?
Answer: High test-retest reliability, but low construct validity.
The high correlation between scores from two different administrations indicates high test-retest reliability (consistency over time). However, the scale's failure to correlate with theoretically related constructs (perseverance, conscientiousness) and relevant outcomes (goal attainment) suggests it lacks construct validity; it does not appear to be measuring the intended concept of 'scholarly grit.'
- To avoid socially desirable responding on a new test for hiring police officers, a psychologist includes many subtle, indirect items such as preferences for certain geometric shapes and colors. The test proves to be a very strong predictor of which recruits will receive commendations for bravery two years after being hired. This test most likely has:
Answer: low face validity and high predictive validity.
Face validity refers to whether a test appears, on the surface, to measure what it is supposed to measure. A test about shapes and colors does not look like a test for police work, so it has low face validity. Predictive validity is a form of criterion validity that refers to how well a test predicts future outcomes. Since the test strongly predicts future commendations, it has high predictive validity. This is a common trade-off in psychological testing.
- A cognitive psychologist studies the impact of a new 'brain-training' game on fluid intelligence. An experimental group plays the game for 30 minutes daily for four weeks, while a control group does not. Both groups are given an IQ test before and after the four-week period. The experimental group shows a significantly greater increase in IQ scores. A critic argues that the improvement may be due to the experimental group becoming more comfortable with the testing situation itself. This criticism refers to which potential confound?
Answer: Testing effect
A testing effect occurs when the act of taking a pre-test influences the scores on a post-test. In this case, the experimental group's improvement might not be due to the brain-training game but to practice or familiarity gained from the first IQ test. While this effect would typically apply to both groups, the criticism implies an interaction where the experimental group benefits more, but the core concept being invoked is the testing effect. It's a plausible alternative explanation for the observed score increase.
- A researcher investigating the effect of caffeine on motor skills tells the experimental group, 'You are receiving a high dose of caffeine that should improve your performance.' He tells the control group, 'You are receiving a placebo.' The researcher is not blinded to the group assignments. He observes that the experimental group performs better. The outcome is potentially compromised by which two confounds?
Answer: Demand characteristics and experimenter bias
Demand characteristics occur when participants form an interpretation of the experiment's purpose and subconsciously change their behavior to fit that interpretation. Telling the experimental group to expect improvement creates such a demand. Experimenter bias (or observer-expectancy effect) occurs when a researcher's expectations influence the outcome. Since the researcher is not blinded, his knowledge of who got the caffeine could unconsciously affect how he observes or interacts with the participants, thereby influencing their performance.
- Two clinical psychologists independently watch a video of a client's therapy session and rate the client's 'expression of affect' on a 1-to-7 scale for 20 distinct one-minute intervals. A comparison of their ratings yields a very low correlation (r = .23). Which of the following is the most direct threat to the validity of the study's conclusions about the client's affect?
Answer: The measure has low inter-rater reliability, which limits its potential validity.
Inter-rater reliability is the degree of agreement between independent observers. A low correlation indicates poor agreement, meaning the ratings are inconsistent and likely contain a large amount of measurement error. A fundamental principle of psychometrics is that a measure cannot be valid if it is not reliable. If the raters cannot agree on what they are seeing, the ratings cannot be an accurate (valid) measure of the client's affect.
- Which of the following scenarios describes a measure with high reliability but low validity?
Answer: A broken thermometer consistently reads five degrees hotter than the actual temperature.
Reliability is consistency, and validity is accuracy. The broken thermometer is highly reliable because it is consistent; it always gives the same wrong answer under the same conditions (i.e., it adds five degrees every time). However, it is not valid because it does not accurately reflect the true temperature. This is a classic example of systematic error, which affects validity but not reliability.
- A high school physics teacher creates a cumulative final exam. The course curriculum was divided equally among four topics: mechanics, thermodynamics, electromagnetism, and optics. However, over 80% of the exam questions are about mechanics. A review by the department head would most likely identify a primary weakness in the exam's:
Answer: content validity.
Content validity refers to the extent to which a measure or test represents all facets of a given construct or content domain. The final exam is supposed to cover the entire curriculum. By focusing almost exclusively on one of the four topics, the exam fails to adequately sample the content domain, thus giving it poor content validity.
- A social psychologist publishes a study concluding that people are less likely to help a stranger in need when other bystanders are present. The study was conducted on a public plaza using only American college students as participants. Which of the following critiques most directly addresses the study's external validity?
Answer: The finding may not generalize to different cultures or to non-student populations.
External validity concerns the extent to which the results of a study can be generalized to other populations, settings, and times. The critique that the findings may not apply beyond American college students directly questions the generalizability of the results to a broader population, which is a core component of external validity. The other options are threats to internal validity (demand characteristics, observer bias, confounding variables).
- To validate a new leadership aptitude test for corporate managers, a company administers the test to its entire current management team. The researchers then gather the most recent annual performance evaluation scores for each manager and find a strong positive correlation between the test scores and the performance ratings. This method of validation primarily gathers evidence for which specific type of validity?
Answer: Concurrent validity
This is an example of concurrent validity, a type of criterion-related validity. It is called 'concurrent' because the test is administered at roughly the same time as the criterion (performance evaluation) data is collected. If the company had administered the test to new hires and then waited a year to collect performance data to correlate with the initial test scores, it would be an assessment of predictive validity.
- A psychologist administers a new intelligence test to a sample of 50 participants on two occasions, separated by one week. The scatter plot below shows the scores from Time 1 versus the scores from Time 2, along with a line of perfect agreement (y=x). Based on the scatter plot shown:
Answer: The test demonstrates high test-retest reliability.
The scatter plot displays data for assessing test-retest reliability, which measures the consistency of results over time. The points on the plot are tightly clustered in a positive linear pattern around the y=x line. This indicates that participants who scored high at Time 1 also scored high at Time 2, and those who scored low at Time 1 also scored low at Time 2. This strong correspondence demonstrates high test-retest reliability. The plot provides no information about construct validity or internal validity.
- A clinical researcher designs an experiment to test a new therapy for phobias. To ensure maximum control, the study is conducted in a sterile laboratory environment, uses a highly standardized protocol, and includes a very specific sample of participants (18-22 year old females with a diagnosed fear of spiders). The therapy shows a strong positive effect. These design choices most directly enhance the study's ​ at the potential cost of its ​.
Answer: internal validity; external validity
Internal validity refers to the degree to which a study can establish a cause-and-effect relationship. The high level of control (lab setting, standardized protocol) strengthens the confidence that the therapy, and not some other variable, caused the observed effect. However, these same choices (sterile environment, homogenous sample) make it uncertain whether the results will generalize to other people in other settings, thus potentially weakening the study's external validity.
- A researcher uses a poorly calibrated stopwatch that consistently runs 10% fast to measure participants' reaction times in a cognitive experiment. Every participant's recorded time is therefore 10% longer than their actual time. How does this systematic error affect the psychometric properties of the reaction time measurement?
Answer: It decreases the validity of the measure but does not affect its reliability.
Reliability refers to consistency. Since the stopwatch is consistently 10% fast, the measurements, though inaccurate, will be consistent. If a person is measured twice and has the same true reaction time, the stopwatch will produce the same incorrect measurement. Thus, reliability is unaffected. Validity refers to accuracy. Since the measurements do not reflect the true reaction times, the validity is compromised. Systematic error affects validity, while random error affects reliability.
- A researcher conducts a six-month study to assess the impact of a mindfulness meditation program on employee stress. Three months into the study, the company undergoes a major restructuring, leading to widespread job uncertainty and layoffs. The researcher notes that stress levels, as measured by a self-report questionnaire, are significantly higher at the end of the study than at the beginning. Attributing this increase in stress solely to the failure of the program would ignore which threat to internal validity?
Answer: History effect
A history effect is a threat to internal validity that occurs when an external event, unrelated to the treatment, happens during the course of a study and affects the dependent variable. In this case, the company restructuring is a major external event that could have caused the increase in stress, making it impossible to isolate the effect (or lack thereof) of the mindfulness program.
- A researcher is conducting a longitudinal study on cognitive decline. At the first data collection point, she uses a standard paper-and-pencil version of a cognitive assessment. A year later, for the second data collection point, she switches to a new, computerized version of the same assessment because it is more efficient. She finds a significant drop in scores. This apparent decline may be an artifact of which threat to validity?
Answer: Instrumentation
An instrumentation threat occurs when there is a change in the measurement instrument or procedure over time. Switching from a paper-and-pencil version to a computerized version of a test is a change in instrumentation. The observed drop in scores could be due to this change (e.g., participants' unfamiliarity with the computer interface) rather than a true cognitive decline, confounding the study's results.
- A researcher is assessing the split-half reliability of a new 40-item survey. She correlates the sum of scores on the 20 odd-numbered items with the sum of scores on the 20 even-numbered items, yielding a correlation coefficient of r = .70. What is the most appropriate next step and conclusion?
Answer: The .70 correlation must be adjusted using the Spearman-Brown formula to estimate the reliability of the full-length test.
The correlation between two halves of a test (.70 in this case) underestimates the reliability of the full test, because reliability is partly a function of test length. The Spearman-Brown prophecy formula is used to correct this correlation and estimate what the reliability would be for the full 40-item test. The resulting corrected value will be higher than .70, likely indicating good internal consistency.
- A psychologist develops a 30-item questionnaire to measure trait conscientiousness. An analysis reveals that the items on the first half of the test are not strongly correlated with the items on the second half. Further analysis shows that several individual items have weak or negative correlations with the total score of the other 29 items. These findings present the most direct challenge to the questionnaire's:
Answer: internal consistency.
Internal consistency is the extent to which all items on a scale measure the same underlying construct. The finding that different parts of the test (the two halves) are not well-correlated, and that individual items do not correlate with the total score, indicates that the items are not 'hanging together' to measure a single, unified concept. This is a direct measure of poor internal consistency.
- A researcher hypothesizes that a new educational software program will improve students' math scores. She allows parents to volunteer their children for the program. The children of parents who volunteer are placed in the treatment group, and children of parents who do not volunteer are placed in the control group. The researcher, who developed the software, personally administers the post-test to both groups. This research design is most seriously threatened by:
Answer: selection bias and experimenter bias.
This design has two major flaws. First, allowing participants to self-select into groups creates a selection bias; the parents who volunteered may be different from those who did not (e.g., more motivated, more resources), and these differences, rather than the software, could account for any observed effects. Second, having the invested researcher administer the tests introduces experimenter bias; her expectations could unconsciously influence her interactions with the students or her scoring of the tests.
