Reliability And Validity — Differentiate reliability types and validity forms in test interpretation - Eppp Knowledge

SECTION 1

Historical Context & Motivation

The formal study of psychological measurement has its roots in the late nineteenth and early twentieth centuries, when pioneers of mental testing recognized that a test is only useful if it produces consistent scores and actually measures the construct it claims to measure. Early intelligence tests developed by Alfred Binet and later adapted by Lewis Terman brought these concerns to the forefront: clinicians needed assurance that scores were not merely artifacts of random fluctuation, and that the tests captured genuine cognitive ability rather than some confounding variable. This dual concern—consistency and accuracy—gave rise to the twin pillars of psychometric theory: reliability and validity.

1904 • Spearman's Reliability Theory

Charles Spearman introduced classical test theory (CTT) and the concept of the reliability coefficient, establishing a mathematical framework for separating true score variance from error variance.

1937 • Kuder-Richardson Formulas

Kuder and Richardson published their KR-20 and KR-21 formulas for estimating internal consistency reliability of dichotomously scored tests, laying groundwork for later generalized methods.

1951 • Cronbach's Alpha

Lee Cronbach generalized the Kuder-Richardson formula to coefficient alpha, applicable to tests with items scored on continuous or polytomous scales, making it the most widely reported reliability index in behavioral health research.

1954–1955 • APA Standards & the Validity Trifecta

The APA Technical Recommendations introduced the classic triad of content, criterion-related, and construct validity, formalizing how test developers must justify what their instruments measure.

1999–2014 • Unified Validity Framework

Samuel Messick and subsequent Standards for Educational and Psychological Testing reconceptualized validity as a unitary construct—all evidence supports the overall validity of score interpretations, rather than separate 'types' of validity.

For clinicians preparing for the EPPP, these developments are not merely historical footnotes. Every time you administer a standardized assessment—whether the MMPI-3, the WAIS-IV, or the Beck Depression Inventory—you must evaluate whether the instrument produces stable, replicable results (reliability) and whether those results actually reflect the psychological construct you intend to assess (validity). The central question this lesson addresses is: How do you differentiate among the major types of reliability and forms of validity evidence, and what does each tell you about the quality of a psychological test?

SECTION 2

Core Principles & Definitions

Before distinguishing among specific reliability types and validity forms, it is essential to grasp the foundational distinction between the two constructs. Reliability refers to the degree to which test scores are free from measurement error—it is about consistency, stability, and reproducibility. Validity refers to the degree to which accumulated evidence and theory support specific interpretations of test scores for given purposes—it is about meaning, accuracy, and justification. A critical axiom in psychometrics is that reliability is necessary but not sufficient for validity: a test can produce perfectly consistent scores that are consistently wrong in what they claim to measure.

Classical Test Theory (CTT)

Every observed score (X) is composed of a true score (T) plus error (E). Reliability is the proportion of observed score variance attributable to true score variance. The higher the reliability coefficient (ranging from 0 to 1), the less error contaminates the measurement.

Sources of Measurement Error

Different reliability types address different error sources: temporal instability (test-retest), item sampling variability (internal consistency), and rater subjectivity (inter-rater). Identifying the relevant error source guides your choice of reliability estimate.

Validity as Evidential Argument

Modern psychometrics treats validity not as a property of a test but as a property of test score interpretations. Evidence for validity accumulates from multiple sources—content, internal structure, relationships with other variables, response processes, and consequences.

Reliability Sets the Ceiling for Validity

The validity coefficient of a test cannot exceed the square root of its reliability coefficient. Unreliable tests attenuate correlations with criterion measures, artificially lowering apparent validity. This mathematical relationship underscores why reliability assessment is always the first step.

Standard Error of Measurement (SEM)

The SEM quantifies the expected range within which a person's true score falls. It links reliability to clinical decision-making: a test with low reliability produces wide confidence intervals, undermining the precision needed for diagnosis or treatment planning.

✦

KEY TAKEAWAY

Think of reliability and validity like a bathroom scale. Reliability means the scale gives you the same reading each time you step on it within the same minute. Validity means the number it shows actually corresponds to your true weight. A scale that consistently reads 10 pounds too heavy is reliable but not valid. A scale that shows different numbers every time is neither reliable nor valid. You need both properties for the measurement to be clinically useful.

SECTION 3

Visual Explanation — Reliability Types at a Glance

This diagram illustrates the four major types of reliability, each targeting a distinct source of measurement error. Note that all four yield a reliability coefficient between 0 and 1, and all feed into the overarching question of whether a test produces sufficiently consistent scores for its intended clinical use.

The diagram above organizes the reliability landscape according to the specific source of measurement error each type addresses. Test-retest reliability examines whether scores remain stable across time, making temporal instability the targeted error source. Alternate-forms (parallel-forms) reliability assesses whether two versions of a test produce comparable scores, isolating content sampling error. Internal consistency reliability evaluates whether items within a single test cohere, and inter-rater reliability determines agreement across different scorers. For EPPP preparation, it is essential to match each reliability type with its error source, statistical index, and appropriate use case.

SECTION 4

Mathematical Framework

Classical test theory provides the mathematical backbone for understanding reliability. The equations below formalize the relationship between observed scores, true scores, error, and derived reliability metrics. Mastery of these formulas is essential for interpreting test manuals and evaluating assessment instruments in clinical practice.

CLASSICAL TEST THEORY EQUATION

X = T + E

X = observed score; T = true score (the score a person would obtain if there were no measurement error); E = error. Error is assumed to be random, with a mean of zero across repeated testing.

RELIABILITY COEFFICIENT

rₓₓ = σ²ₜ / σ²ₓ = 1 − (σ²ₑ / σ²ₓ)

rₓₓ = reliability coefficient; σ²ₜ = true score variance; σ²ₓ = observed score variance; σ²ₑ = error variance. Reliability is the proportion of observed variance that is attributable to true score variance.

STANDARD ERROR OF MEASUREMENT

SEM = SDₓ × √(1 − rₓₓ)

SEM = standard error of measurement; SDₓ = standard deviation of observed scores; rₓₓ = reliability coefficient. The SEM creates a confidence interval around an individual's observed score to estimate where the true score likely falls. A 95% confidence interval = Observed Score ± 1.96 × SEM.

CRONBACH'S ALPHA

α = (k / (k − 1)) × (1 − (Σσ²ᵢ / σ²ₓ))

α = Cronbach's alpha; k = number of items; σ²ᵢ = variance of item i; σ²ₓ = total test score variance. Alpha represents the average of all possible split-half reliability coefficients. For dichotomous items, this formula reduces to the Kuder-Richardson 20 (KR-20).

💡

Clinical Rule of Thumb

For individual clinical decisions (e.g., diagnosis, placement), a reliability coefficient of ≥ 0.90 is preferred. For research and group-level decisions, ≥ 0.80 is generally considered acceptable. The higher the stakes, the more stringent the reliability requirement.

SECTION 5

Detailed Breakdown — Forms of Validity Evidence

While the EPPP frequently references the traditional triad of content, criterion-related, and construct validity, the current Standards for Educational and Psychological Testing (AERA, APA, & NCME, 2014) conceptualizes validity as a unitary construct supported by five categories of evidence. Understanding both frameworks is essential for the exam. The table and diagram below cross-reference the traditional labels with the modern evidence categories.

This diagram illustrates the modern unitary validity framework. All five categories of evidence—test content, response processes, internal structure, relations with other variables, and consequences—feed into the overarching construct validity argument represented by the dashed ellipse. Traditional validity labels are mapped to their modern counterparts at the bottom.

Traditional Label	Modern Evidence Category	Key Methods	Clinical Example
Content Validity	Evidence based on test content	Expert panel review, content blueprints, item-objective alignment	EPPP items are mapped to a content domain blueprint to ensure adequate coverage of all knowledge areas.
Criterion-Related (Predictive)	Relations with other variables	Correlation with future criterion measured later in time	GRE scores correlate with first-year graduate GPA (criterion measured months/years later).
Criterion-Related (Concurrent)	Relations with other variables	Correlation with criterion measured at the same time	A new depression scale correlates .85 with the BDI-II administered simultaneously.
Construct Validity	All five evidence categories	Convergent/discriminant evidence, factor analysis, known-groups comparisons, MTMM matrix	An anxiety measure correlates highly with other anxiety measures (convergent) and weakly with unrelated constructs (discriminant).
Face Validity	Not a formal evidence category	Subjective judgment: does the test 'look like' it measures what it claims?	A depression questionnaire asks about mood and sleep. Face validity affects examinee buy-in but is not formal psychometric evidence.

Traditional Validity Labels Mapped to Modern Evidence Categories

💡

EPPP Exam Tip: Convergent vs. Discriminant

Convergent and discriminant validity are subtypes of construct validity, often assessed using Campbell and Fiske's Multitrait-Multimethod (MTMM) matrix. Convergent validity is demonstrated when measures of theoretically related constructs correlate highly. Discriminant validity is demonstrated when measures of theoretically unrelated constructs show low correlations. The EPPP frequently tests your ability to distinguish between these two concepts.

SECTION 6

Worked Example — Evaluating a Depression Screener

Imagine you are a psychologist selecting a new depression screening instrument, the Hypothetical Depression Scale (HDS), for use in a community mental health center. The test manual reports the following psychometric data: the HDS has 20 items scored on a 4-point Likert scale, coefficient alpha of 0.88, test-retest reliability (2-week interval) of 0.82, a correlation of 0.79 with the BDI-II (concurrent criterion), and a standard deviation of 10. Let us walk through how to interpret these data.

Evaluating the Reliability and Validity of the HDS

1
Step 1 — Evaluate Internal Consistency

Cronbach's alpha is reported as 0.88. This exceeds the 0.80 threshold for research purposes, suggesting that the 20 items are measuring a reasonably homogeneous construct. However, for individual clinical decisions (such as making a diagnosis), we ideally want α ≥ 0.90. Thus, the HDS is adequate for screening but may lack the precision needed for sole reliance in diagnostic decision-making.

α = 0.88 → Adequate for screening, marginal for high-stakes individual decisions.
2
Step 2 — Evaluate Temporal Stability

The test-retest reliability over a 2-week interval is 0.82. Because depression symptoms can fluctuate, a moderate test-retest coefficient is expected for state-dependent constructs. If the construct were a stable trait (e.g., personality), we would expect higher test-retest values (≥ 0.85). The 2-week interval is appropriate—too short risks practice effects; too long introduces genuine construct change.

r = 0.82 over 2 weeks → Acceptable for a state-dependent measure.
3
Step 3 — Calculate the Standard Error of Measurement

Using the SEM formula: SEM = SDₓ × √(1 − rₓₓ). With SDₓ = 10 and rₓₓ = 0.88, we calculate SEM = 10 × √(1 − 0.88) = 10 × √0.12 = 10 × 0.346 ≈ 3.46. This means a 95% confidence interval around any individual's score spans approximately ±6.8 points (1.96 × 3.46). If a client scores 25, their true score likely falls between 18.2 and 31.8.

SEM ≈ 3.46 → 95% CI for a score of 25: [18.2, 31.8]
4
Step 4 — Evaluate Concurrent Criterion Validity

The HDS correlates 0.79 with the BDI-II, administered simultaneously. This constitutes concurrent criterion-related validity evidence. The correlation is strong, suggesting the HDS measures a construct highly related to what the BDI-II measures. To determine how much variance the HDS shares with the BDI-II, we square the correlation: r² = 0.79² = 0.624, meaning approximately 62% shared variance.

r = 0.79 with BDI-II → 62% shared variance → Strong concurrent validity evidence.
5
Step 5 — Consider the Validity Ceiling

The maximum possible validity coefficient is limited by reliability: max r(validity) = √rₓₓ = √0.88 ≈ 0.938. The obtained concurrent validity coefficient of 0.79 is well within this ceiling (0.79 / 0.938 = 84% of the maximum possible validity), suggesting that measurement error is not unduly suppressing the validity estimate. Had the reliability been lower—say 0.60—the maximum validity would be only √0.60 ≈ 0.775, severely limiting the instrument's potential.

Max validity = √0.88 ≈ 0.94 → Obtained 0.79 uses 84% of theoretical maximum.

SECTION 7

Strengths, Limitations, and Common Confusions

Reliability Type	Strengths	Limitations / Pitfalls
Test-Retest	Directly assesses temporal stability; easy to understand and compute.	Practice effects inflate estimates with short intervals; genuine change in the construct (e.g., mood states) deflates estimates with long intervals. Inappropriate for unstable constructs.
Alternate Forms	Controls for practice effects and memory; addresses content sampling error.	Creating truly parallel forms is difficult and expensive. If administered at different times, confounds temporal instability with content sampling.
Internal Consistency	Requires only one administration; efficient and widely reported.	Inappropriate for speeded tests (inflated estimates). Overly heterogeneous or multidimensional tests may show misleadingly low alpha. Does not address temporal stability.
Inter-Rater	Essential for subjectively scored assessments; Cohen's kappa corrects for chance agreement.	Percentage agreement alone is misleading because it ignores chance. Kappa can be paradoxically low when base rates are extreme. Requires training raters, which adds cost.
Split-Half	Single administration; Spearman-Brown correction adjusts for test length reduction.	Result depends on how the test is split; only one of many possible splits is evaluated. Superseded by Cronbach's alpha, which averages all possible split-halves.

Comparison of Reliability Types: Strengths and Limitations

✦

KEY TAKEAWAY

Think of choosing a reliability estimate like selecting a diagnostic lab test in medicine. A blood glucose test taken once tells you about current levels (analogous to internal consistency from a single administration), while hemoglobin A1c assessed across months tells you about stability over time (analogous to test-retest reliability). Different clinical questions demand different reliability evidence, just as different medical questions demand different lab panels. No single reliability estimate answers every question about a test's consistency.

SECTION 8

Connection to Advanced Psychometric Theory

Classical test theory (CTT), while foundational, has notable limitations that advanced psychometric frameworks seek to address. The EPPP does not require deep expertise in these advanced methods, but awareness of their existence and relationship to CTT concepts of reliability and validity is increasingly important. Two advanced frameworks deserve mention: Generalizability Theory (G-Theory) and Item Response Theory (IRT).

Feature	Classical Test Theory (CTT)	Generalizability Theory (G-Theory)	Item Response Theory (IRT)
Error model	Single undifferentiated error term (E)	Multiple error facets (raters, items, occasions) simultaneously estimated via ANOVA	Item-level error; reliability varies by trait level (information function)
Reliability index	Single coefficient (α, r)	Generalizability coefficient (Eρ²) and dependability coefficient (Φ)	Test information function I(θ) — precision varies across the trait continuum
Sample dependency	Statistics are sample-dependent; reliability varies across populations	Also sample-dependent but can model multiple error facets in a single study	Item parameters are sample-independent (invariance); enables adaptive testing
Clinical application	Most test manuals report CTT indices; used for standard clinical instruments	Used when multiple error sources matter (e.g., OSCE medical exams with raters × stations)	Computer adaptive testing (e.g., CAT-MH); items tailored to examinee's ability level

CTT vs. Advanced Psychometric Frameworks

For the EPPP, the most critical advanced concept is understanding that Generalizability Theory extends CTT by partitioning error variance into multiple facets simultaneously, rather than treating all error as a monolithic term. In clinical settings, this is increasingly relevant when behavioral health assessments involve multiple raters observing multiple clients across multiple occasions—a scenario common in behavioral observation systems and competency-based clinical evaluations. IRT, meanwhile, offers the concept of measurement precision that varies along the trait continuum, unlike CTT's single reliability coefficient. This is especially valuable for instruments designed to discriminate at clinical cutoff points, where precision at the decision threshold matters most.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A psychologist administers a personality inventory to the same group of clients two weeks apart and correlates the two sets of scores. The resulting correlation coefficient of 0.85 is an estimate of which type of reliability? Additionally, what specific source of measurement error does this reliability type address?

Show answer

This is an estimate of test-retest reliability, also called temporal stability. The specific source of measurement error it addresses is temporal instability—fluctuations in scores that occur due to changes in the examinee's state, testing conditions, or other time-related factors between the two administrations. Because the same instrument is given to the same individuals across two time points, the Pearson correlation between the two score sets captures how stable the trait measurement is over the two-week interval. A coefficient of 0.85 suggests reasonably good temporal stability, particularly for a personality measure that is expected to tap into relatively enduring characteristics.

Question 2: BASIC CALCULATION

A cognitive screening test has a reliability coefficient of 0.91 and a standard deviation of 15. Calculate the Standard Error of Measurement (SEM) and construct a 95% confidence interval around an observed score of 100.

Show answer

Using the SEM formula: SEM = SDₓ × √(1 − rₓₓ) = 15 × √(1 − 0.91) = 15 × √0.09 = 15 × 0.30 = 4.50. The 95% confidence interval is computed as: Observed Score ± 1.96 × SEM = 100 ± 1.96 × 4.50 = 100 ± 8.82. Therefore, the 95% confidence interval is [91.18, 108.82]. Clinically, this means we can be 95% confident that the individual's true score falls between approximately 91 and 109. The relatively narrow confidence interval reflects the high reliability of the instrument (0.91). Had the reliability been lower—say 0.70—the SEM would be 8.22 and the confidence interval would span nearly 32 points, dramatically reducing diagnostic precision.

Question 3: INTERMEDIATE

A researcher develops a new anxiety measure and reports that it correlates 0.82 with the State-Trait Anxiety Inventory (STAI) and 0.15 with a measure of extroversion. The researcher claims these findings support the construct validity of the new measure. Evaluate this claim, specifying what types of construct validity evidence each correlation represents and whether the evidence is adequate.

Show answer

The researcher's claim is reasonably well-supported. The high correlation of 0.82 with the STAI represents convergent validity evidence: the new anxiety measure relates strongly to an established measure of the same construct (anxiety), suggesting they share substantial common variance (r² = 0.67, or 67% shared variance). The low correlation of 0.15 with extroversion represents discriminant validity evidence: the new measure is largely independent of a theoretically unrelated construct, as expected. Together, convergent and discriminant evidence form the foundation of construct validity as operationalized in Campbell and Fiske's multitrait-multimethod framework. However, the claim could be strengthened by additional evidence, such as factor analytic data demonstrating the expected internal structure, known-groups comparisons showing that clinically anxious individuals score higher than non-anxious controls, and examination of response processes to ensure examinees interpret items as intended.

Question 4: APPLIED

You are a clinical psychologist working in a forensic setting. You must select a risk assessment instrument for predicting violent recidivism among parolees. You have narrowed your choice to two instruments: Instrument A reports high internal consistency (α = 0.92) but no predictive validity data, while Instrument B reports moderate internal consistency (α = 0.78) but strong predictive validity (r = 0.65 with violent recidivism over 5 years). Which instrument would you choose and why? What additional psychometric information would you request?

Show answer

Instrument B is the better choice for this application. While Instrument A has superior internal consistency, the purpose of risk assessment is to predict a future criterion—violent recidivism. Without predictive validity evidence, Instrument A's high alpha merely tells us that its items cohere well, not that they predict the outcome of interest. Instrument B's predictive validity coefficient of 0.65 is strong for behavioral prediction, accounting for approximately 42% of variance in recidivism outcomes (r² = 0.42). Its moderate alpha of 0.78 is understandable for a risk instrument that likely samples multiple heterogeneous risk factors (e.g., history, substance use, antisocial attitudes), which may actually reduce alpha while improving criterion prediction. Additional information to request includes: (1) specificity and sensitivity data at recommended cutoff scores, (2) the base rate of violence in the normative sample and your forensic population, (3) incremental validity over existing methods, (4) cross-validation studies across diverse forensic populations, and (5) inter-rater reliability if the instrument involves scored interviews.

Question 5: CRITICAL THINKING

A test manual reports that a 50-item depression measure has a Cronbach's alpha of 0.95. The test publisher uses this as evidence of excellent measurement quality. Critically evaluate this claim. Under what circumstances might an alpha of 0.95 actually signal a problem with the instrument? How does this relate to the distinction between reliability and validity?

Show answer

While α = 0.95 superficially indicates excellent internal consistency, an extremely high alpha can signal item redundancy rather than measurement quality. When items are so similar that they are essentially paraphrases of one another, alpha is inflated because inter-item covariances are high—but the test may be measuring a narrow bandwidth of the construct, sacrificing content validity. For depression, which is a multifaceted construct encompassing cognitive, somatic, affective, and behavioral dimensions, a truly comprehensive instrument might actually show a lower alpha (e.g., 0.80–0.90) because items tap diverse manifestations of the syndrome. An alpha of 0.95 with 50 items may indicate that many items could be eliminated without meaningful information loss, creating an unnecessarily long test that increases respondent burden. Furthermore, alpha is influenced by test length (more items → higher alpha by the Spearman-Brown prophecy) and by sample heterogeneity (more variable samples produce higher alpha). The fundamental lesson is that reliability and validity serve different purposes: high internal consistency does not guarantee that the test comprehensively and accurately measures the target construct. A shorter, more heterogeneous instrument with α = 0.85 might actually offer superior content validity and clinical utility. This exemplifies the axiom that reliability is necessary but not sufficient for validity—and that pursuing maximal reliability can paradoxically undermine the quality of measurement.

SUMMARY

Summary — Reliability and Validity in Psychological Assessment

Summary

Reliability refers to the consistency and stability of test scores, rooted in classical test theory's decomposition of observed scores into true score plus error (X = T + E). Four major types address distinct error sources: test-retest (temporal instability), alternate forms (content sampling), internal consistency (item heterogeneity, indexed by Cronbach's alpha, KR-20, or split-half methods), and inter-rater reliability (rater subjectivity, indexed by Cohen's kappa or ICC). The Standard Error of Measurement (SEM) translates the reliability coefficient into a clinically meaningful confidence interval around an individual's observed score.

Validity is the degree to which evidence supports the intended interpretation of test scores. The modern framework identifies five evidence sources: test content, response processes, internal structure, relations with other variables (encompassing convergent, discriminant, predictive, and concurrent evidence), and consequences of testing. The cardinal rule is that reliability is necessary but not sufficient for validity: a test cannot accurately measure a construct if its scores are inconsistent, but perfectly consistent scores can still miss the intended construct entirely. When selecting or evaluating assessment instruments, clinicians must examine both reliability and validity evidence in light of the specific purpose, population, and stakes of the assessment.