Reliability & Validity — I can explain reliability and validity in psychological measurement and identify threats to each. - Psychology

SECTION 1

Historical Context & Motivation

Psychology has always faced a unique challenge compared to sciences like physics or chemistry: how do you accurately measure something you cannot directly see or touch, like intelligence, anxiety, or personality? Early psychologists in the late 1800s recognized that if their field was going to be taken seriously as a science, they needed tools that produced trustworthy results. This drive led to the development of two foundational concepts — reliability and validity — that serve as the quality-control standards for every psychological test, survey, and measurement ever created.

1890 • Cattell's Mental Tests

James McKeen Cattell coined the term "mental test" and began measuring reaction times and sensory abilities, raising immediate questions about whether these tests were consistent and meaningful.

1904 • Spearman's Correlation Work

Charles Spearman developed statistical methods — including the correlation coefficient — that allowed researchers to quantify how consistent a test's scores were, laying the mathematical groundwork for reliability analysis.

1954 • APA Technical Standards

The American Psychological Association published its first official technical standards for psychological tests, formally defining types of validity (content, criterion, and construct) that test developers must demonstrate.

1955 • Cronbach & Meehl on Construct Validity

Lee Cronbach and Paul Meehl published a landmark paper arguing that construct validity — whether a test truly measures the abstract concept it claims to — is the most important type of validity in psychology.

2014 • Modern Standards Update

The latest edition of the Standards for Educational and Psychological Testing was released, reflecting decades of refinement in how psychologists think about measurement quality in an era of computerized and adaptive testing.

The central question that drove all of this progress is deceptively simple: How can we be confident that a psychological measurement is both consistent and accurate? Without reliability, a test gives random, unpredictable results. Without validity, a test might be perfectly consistent but measure the wrong thing entirely. Understanding these two concepts is essential for evaluating any claim in psychology — from an IQ score on a report card to a diagnosis of a mental health condition.

SECTION 2

Core Principles & Definitions

At its core, reliability refers to the consistency of a measurement. A reliable test produces similar results under similar conditions. If you step on a bathroom scale three times in one minute, you expect to see roughly the same weight each time. If the scale shows 130 lbs, then 145 lbs, then 118 lbs, that scale is unreliable — you can't trust any single reading. Validity refers to accuracy — whether a measurement tool actually measures what it claims to measure. A clock may be perfectly reliable, ticking at the exact same rate every day, but using it to measure your intelligence would make it completely invalid for that purpose.

Reliability = Consistency

A measurement is reliable when it produces stable, repeatable results across time, items, or raters. Think of it as the precision of a dart throw — are the darts landing in the same cluster?

Validity = Accuracy

A measurement is valid when it actually measures the concept it is supposed to measure. This is like asking whether the darts are hitting the bullseye — are they landing in the right place?

Reliability Is Necessary but Not Sufficient

A test must be reliable before it can be valid, but reliability alone does not guarantee validity. A broken thermometer could consistently read 5° too high — reliable but not valid.

Multiple Types of Each

There are several subtypes of reliability (test-retest, internal consistency, inter-rater) and validity (content, criterion, construct). Each assesses a different dimension of measurement quality.

Threats Reduce Quality

Many factors can threaten reliability (fatigue, ambiguous items, environmental changes) and validity (biased items, confounding variables, poor sampling). Identifying threats is a critical research skill.

✦

KEY TAKEAWAY

Think of reliability and validity like a GPS device. Reliability is like the GPS giving you the same coordinates every time you stand in the same spot. Validity is whether those coordinates actually match your real location on the map. A GPS that always says you're two miles north of where you really are is reliable (consistent) but not valid (accurate). You need both for the device to be truly useful.

SECTION 3

Visual Explanation — The Target Analogy

The most famous way to visualize the relationship between reliability and validity is the target (dartboard) analogy. Imagine four different dartboards, each representing a different combination of high or low reliability and validity. The bullseye represents the true value of whatever you are trying to measure, and each dart represents one measurement attempt.

The three targets show key combinations. The cyan target (left) shows ideal measurement — darts are tightly clustered on the bullseye. The violet target (center) shows a test that is reliable but not valid — consistent results that miss the mark. The pink target (right) shows neither reliability nor validity — random scatter everywhere.

Notice an important logical relationship in the diagram: you cannot have high validity without high reliability. If your darts are scattered all over the board (unreliable), they can't all be on the bullseye (valid). This is why psychologists say reliability is a necessary but not sufficient condition for validity. You must achieve consistency first, then confirm that the consistent results are actually measuring the right thing.

SECTION 4

How Reliability & Validity Are Assessed

Assessing Reliability

Psychologists use several methods to evaluate reliability, each targeting a different source of potential inconsistency. The strength of reliability is typically expressed as a correlation coefficient — a number between 0 and 1, where values closer to 1 indicate stronger consistency. A reliability coefficient of 0.80 or higher is generally considered acceptable for psychological tests.

RELIABILITY COEFFICIENT

r = correlation between two sets of scores (range: 0.00 to 1.00)

Where r represents the degree of consistency. An r of 0.00 means no consistency (pure randomness), while an r of 1.00 means perfect consistency. Most well-designed psychological tests achieve reliability coefficients between 0.70 and 0.95.

Type of Reliability	What It Measures	How It's Done
Test-Retest	Consistency over time — does the test give the same results when administered again later?	Give the same test to the same group on two different occasions and correlate the scores.
Internal Consistency	Consistency across items — do all the questions on the test measure the same underlying concept?	Use split-half method (correlate first half with second half) or calculate Cronbach's alpha across all items.
Inter-Rater	Consistency across scorers — do different people grading or observing the same behavior assign similar scores?	Have two or more raters independently score the same responses and correlate their judgments.
Parallel Forms	Consistency across equivalent versions — do two different versions of the test produce similar scores?	Create two equivalent test forms, administer both to the same group, and correlate the results.

Four major types of reliability in psychological testing

Assessing Validity

While reliability asks "Is this test consistent?", validity asks the deeper question: "Is this test actually measuring what we think it's measuring?" There are three major categories of validity evidence that psychologists look for when evaluating a measurement tool.

Type of Validity	Core Question	Example
Content Validity	Does the test cover a representative sample of the topic it's supposed to measure?	A history final exam that only covers Chapter 1 lacks content validity — it doesn't represent the whole course.
Criterion Validity	Does the test predict or correlate with an external criterion (real-world outcome)?	SAT scores that correlate with college GPA show criterion validity. Can be concurrent (measured now) or predictive (measured later).
Construct Validity	Does the test truly capture the abstract psychological concept (construct) it claims to?	An anxiety questionnaire should correlate with other anxiety measures (convergent) and NOT correlate with unrelated traits like athletic ability (discriminant).

Three major types of validity evidence

💡

Face Validity — A Common Misconception

Face validity refers to whether a test looks like it measures what it claims to. While it can help with test-taker cooperation, face validity is not considered true scientific validity because appearances can be deceiving. A test could look relevant but still measure the wrong thing.

SECTION 5

Threats to Reliability & Validity

Even well-designed tests can have their reliability or validity undermined by various threats — factors that introduce error or bias into the measurement process. Recognizing these threats is one of the most important skills in evaluating psychological research. The diagram below organizes the major threats into two categories.

The left column (blue) lists five major threats to reliability, all of which introduce random or inconsistent error. The right column (violet) lists five major threats to validity, all of which cause systematic bias or inaccuracy in what the test measures.

A key distinction to remember is that threats to reliability tend to involve random error — unpredictable fluctuations that make scores bounce around. Threats to validity, on the other hand, involve systematic error — consistent biases that push scores in one direction. For example, if a math test includes very complex English vocabulary, non-native English speakers might consistently score lower not because they lack math ability, but because the test is accidentally measuring English proficiency. That's a validity threat — the test is systematically measuring the wrong thing.

SECTION 6

Worked Example — Evaluating a New Anxiety Questionnaire

Let's walk through a realistic scenario. Imagine a school psychologist develops a 20-item questionnaire called the "Student Anxiety Scale" (SAS) to measure test anxiety in high school students. She wants to determine if the SAS is both reliable and valid before using it to identify students who might need support.

Evaluating the Student Anxiety Scale (SAS)

1
Step 1 — Check Test-Retest Reliability

The psychologist administers the SAS to 100 students, waits two weeks, and then gives the same test to the same students. She calculates the correlation between the two sets of scores.

Correlation r = 0.85. This is above the 0.80 threshold, so the SAS demonstrates good test-retest reliability. Students' scores are consistent over time.
2
Step 2 — Check Internal Consistency

She calculates Cronbach's alpha for the 20 items to see if all the questions are measuring the same underlying construct (test anxiety). Items that don't correlate well with the others may need to be revised or removed.

Cronbach's α = 0.91. This is excellent internal consistency, meaning the 20 items are consistently measuring the same concept.
3
Step 3 — Check Content Validity

She asks a panel of five psychology teachers and counselors to review the 20 items and judge whether they represent the full range of test anxiety symptoms (physical symptoms, worried thoughts, avoidance behaviors, etc.).

The panel notes that the SAS includes items about worry and physical symptoms but has no items about avoidance behaviors (like skipping class before a test). This is a content validity concern — the test underrepresents the construct.
4
Step 4 — Check Criterion Validity

She compares SAS scores with students' actual exam performance and with observations of anxiety-related behaviors during test-taking (fidgeting, leaving early, asking to use the restroom).

SAS scores correlate r = −0.42 with exam grades (higher anxiety, lower grades) and r = 0.55 with observed anxious behaviors. Both correlations are in the expected direction, supporting moderate criterion validity.
5
Step 5 — Identify Remaining Threats & Recommend Improvements

The psychologist considers potential threats. Could social desirability bias (students underreporting anxiety to look "tough") hurt validity? Could the fact that the questionnaire is given right before an exam inflate anxiety scores for the retest, hurting generalizability?

Recommendation: Add items about avoidance behaviors to improve content validity. Include reverse-scored items to reduce response bias. Consider administering the test at a neutral time, not right before exams, to control for situational confounds.

SECTION 7

Reliability vs. Validity — Side-by-Side Comparison

Students sometimes confuse reliability and validity because both involve evaluating the quality of a test. The table below provides a direct comparison to help you keep them straight.

Feature	Reliability	Validity
Central question	"Does this test give consistent results?"	"Does this test measure what it claims to?"
Analogy	A scale that gives the same weight each time	A scale that shows your actual weight
Type of error	Random error (inconsistency)	Systematic error (bias)
Can exist without the other?	Yes — a test can be reliable but not valid	No — a test cannot be valid without being reliable
How it's measured	Correlation coefficients (test-retest, split-half, inter-rater, Cronbach's alpha)	Expert review, correlations with criteria, theoretical analysis
Easier to establish?	Yes — mostly statistical, can be calculated directly	No — requires judgment, multiple sources of evidence over time

Key differences between reliability and validity

✦

KEY TAKEAWAY

Here's a memory trick: Reliability is about repeatability (both start with "re"). Validity is about value — is the test actually valuable because it measures the right thing? A clock that's always five minutes fast is repeatable (reliable) but doesn't give you the true time (not valid).

SECTION 8

Connections to Advanced Measurement Theory

The concepts of reliability and validity you've learned here form the foundation of a larger field called psychometrics — the science of psychological measurement. In college-level psychology and statistics courses, these ideas are explored with much greater mathematical depth. The table below previews how these introductory concepts connect to more advanced topics.

What You Know Now	Advanced Extension
Reliability as a correlation between two sets of scores (r = 0.85)	Classical Test Theory (CTT) formally defines an observed score as the sum of a true score plus error: X = T + E. Reliability equals the ratio of true score variance to total variance.
Internal consistency measured by Cronbach's alpha	Item Response Theory (IRT) models each item individually, estimating item difficulty and discrimination parameters. This allows adaptive testing where different students get different questions.
Content, criterion, and construct validity as separate categories	Modern validity theory (Messick, 1989) treats validity as a unified concept — all evidence contributes to one overarching judgment about the appropriateness of test-score interpretations.
Threats like cultural bias and construct-irrelevant variance	Differential Item Functioning (DIF) analysis statistically detects items that perform differently across demographic groups, even when those groups have equal ability levels.

From introductory concepts to advanced psychometrics

You don't need to master these advanced topics now, but it's worth knowing that the reliability and validity framework you're building is the gateway to a sophisticated and mathematically rich area of psychology. These tools are used in high-stakes settings every day — from college admissions testing (SAT, ACT) to clinical diagnosis (depression screeners, ADHD assessments) to personnel selection in organizations. The principles remain the same regardless of the complexity of the math: good measurement requires both consistency and accuracy.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain, in your own words, the difference between reliability and validity. Then explain why a test can be reliable without being valid, but cannot be valid without being reliable.

Show answer

Reliability refers to the consistency of a test — whether it produces similar results under similar conditions. Validity refers to accuracy — whether the test measures what it is supposed to measure. A test can be reliable without being valid because it might consistently measure the wrong thing (like a clock that is always exactly five minutes fast). However, a test cannot be valid without being reliable because if scores fluctuate randomly each time, they cannot consistently hit the 'true' value. Consistency is a prerequisite for accuracy.

Question 2: BASIC CALCULATION

A researcher gives the same depression questionnaire to 50 participants on two occasions, two weeks apart. The test-retest correlation is r = 0.62. Is this considered acceptable reliability? Justify your answer using the standard threshold discussed in the lesson.

Show answer

A test-retest correlation of r = 0.62 falls below the generally accepted threshold of r = 0.80 for psychological tests. This means there is a concerning amount of inconsistency between the two administrations — participants' scores changed substantially over the two-week period. The researcher should investigate possible reasons: did participants' actual depression levels change, were testing conditions different, or are some items poorly worded? Before using this questionnaire for important decisions, reliability should be improved, perhaps by adding more items, clarifying ambiguous wording, or standardizing administration procedures.

Question 3: INTERMEDIATE

A school develops a new "Critical Thinking Assessment" for 10th graders. The test has high internal consistency (Cronbach's α = 0.89) and good test-retest reliability (r = 0.83). However, students' scores on this assessment correlate r = 0.78 with their reading comprehension scores and only r = 0.25 with their performance on logic puzzles. What type of validity concern does this pattern suggest? Explain your reasoning.

Show answer

This pattern suggests a construct validity problem, specifically construct-irrelevant variance. If the test truly measured critical thinking, you would expect a strong correlation with logic puzzles (which require critical thinking) — but the correlation is only r = 0.25. Meanwhile, the high correlation with reading comprehension (r = 0.78) suggests the test is heavily influenced by reading ability rather than pure critical thinking. The test appears to be accidentally measuring reading skill more than reasoning skill. This is a classic example of construct-irrelevant variance: the reading demands of the test items are contaminating the scores, making the test invalid as a measure of critical thinking for students with different reading levels.

Question 4: APPLIED

A company uses a personality test to screen job applicants for a customer service position. The test was originally developed and validated using a sample of college students in the United States. The company now wants to use it to hire employees in Japan. Identify at least two specific threats to reliability or validity that might arise in this situation, and suggest one way to address each threat.

Show answer

Threat 1 — Cultural and language bias (validity threat): The test was developed with American college students, so items may reflect Western cultural values and communication styles. For example, a question like 'I enjoy being the center of attention' may measure extraversion in the U.S. but could be interpreted differently in Japanese culture, where self-promotion is less valued. Solution: Conduct a cultural adaptation process, including translating the test with back-translation, reviewing items with Japanese psychologists, and running a validation study with a Japanese sample. Threat 2 — Construct-irrelevant variance (validity threat): If the test is simply translated without adaptation, language difficulty could affect scores independently of actual personality traits. Solution: Use native Japanese-speaking psychologists to ensure that translated items are natural and at an appropriate reading level, and pilot-test the translated version to check for comprehension issues. Threat 3 — Restricted range or sampling bias (validity threat): The original validation sample (American college students) may not represent the broader working population in Japan. The criterion validity evidence from the U.S. study may not generalize. Solution: Conduct a new criterion validity study correlating test scores with job performance ratings in the Japanese context.

Question 5: CRITICAL THINKING

Consider this argument: 'Standardized intelligence tests like the IQ test have very high reliability coefficients (often above 0.90), which proves they are excellent measures of intelligence.' Do you agree or disagree with this argument? Use the concepts from this lesson to construct a thorough response that addresses both what this evidence does and does not tell us.

Show answer

This argument is partially correct but ultimately flawed because it confuses reliability with validity. A high reliability coefficient (r > 0.90) tells us that IQ tests produce very consistent scores — if you take the test today and again next month, your scores will be very similar. This is important and necessary, but it does not prove the test is an 'excellent measure of intelligence.' Validity is the separate question of whether the test actually captures the full meaning of intelligence. Several validity concerns remain even with perfect reliability. First, there is the question of construct underrepresentation: most IQ tests emphasize logical reasoning, verbal ability, and processing speed, but many psychologists argue that intelligence also includes creativity, practical wisdom, emotional intelligence, and other abilities not captured by these tests. Second, construct-irrelevant variance can affect scores — factors like test anxiety, cultural background, familiarity with timed testing, and socioeconomic access to educational resources can influence IQ scores independently of actual cognitive ability. Third, the very definition of 'intelligence' is debated among psychologists (e.g., Sternberg's triarchic theory, Gardner's multiple intelligences), so the construct itself may not be fully agreed upon. In summary, high reliability is necessary but not sufficient evidence for claiming a test is an excellent measure. We also need strong validity evidence — and for IQ tests, that evidence is the subject of ongoing scientific debate.

SUMMARY

Lesson Summary

Summary

Reliability and validity are the two pillars of quality in psychological measurement. Reliability refers to the consistency of a measurement — assessed through test-retest, internal consistency, inter-rater, and parallel forms methods. Validity refers to accuracy — whether the test measures its intended construct — and includes content validity, criterion validity, and construct validity. A test must be reliable before it can be valid, but reliability alone does not guarantee validity.

Common threats to reliability include test-taker variability, ambiguous items, environmental changes, and scorer subjectivity. Threats to validity include construct underrepresentation, construct-irrelevant variance, cultural bias, and confounding variables. Understanding these concepts equips you to critically evaluate any psychological test, research study, or measurement claim you encounter — a skill that extends far beyond the psychology classroom into everyday life.