Assessment Technical Vocabulary — 2. Knows basic technical vocabulary associated with assessments (e.g., criterion-referenced, norm-referenced, reliability, validity, fairness, and equity) - Praxis Reading Specialist 5302

SECTION 1

Historical Context & Motivation

The vocabulary of educational assessment did not emerge overnight; it evolved across more than a century of scholarship in psychometrics, educational psychology, and civil rights law. Understanding where these terms originated helps reading specialists appreciate why precision in assessment language matters so profoundly. Early intelligence testing in the late nineteenth century relied on imprecise, subjective judgments, and the consequences—misplacement of students, reinforcement of social inequities—were severe. The drive to formalize assessment concepts like reliability, validity, and fairness was, at its heart, a drive to protect students from harm.

1904 • Spearman's Reliability Theory

Charles Spearman published foundational work on the correlation coefficient and measurement error, introducing the concept of reliability as consistency of scores across administrations—a concept that would become central to modern psychometrics.

1954 • APA Technical Recommendations

The American Psychological Association published the first formal Technical Recommendations for Psychological Tests and Diagnostic Techniques, codifying standards for validity (content, criterion-related, and construct) and reliability.

1963 • Glaser Coins 'Criterion-Referenced'

Robert Glaser introduced the term criterion-referenced testing to distinguish assessments that measure student mastery of specific skills from those that compare students to one another.

1999 • Joint Standards for Testing

AERA, APA, and NCME released the landmark Standards for Educational and Psychological Testing, formally embedding fairness and equity as ethical requirements in every stage of assessment design and use.

2014 • Revised Standards — Fairness as a Chapter

The 2014 revision elevated fairness from a subsection to a full stand-alone chapter, signaling the profession's recognition that equitable assessment is not an optional addendum but a foundational pillar alongside validity and reliability.

This historical arc reveals a central question that frames the entire lesson: How do we ensure that what we measure, how consistently we measure it, and whom we serve with those measurements all meet defensible professional standards? Answering that question requires fluency in the technical vocabulary of assessment—terminology that will appear repeatedly on the Praxis 5302 and, more importantly, in your daily practice as a reading specialist.

SECTION 2

Core Principles & Definitions

Assessment technical vocabulary can be organized around a few anchoring concepts. At the highest level, every assessment question you encounter on the Praxis—and every assessment decision you make in practice—connects to at least one of these foundational ideas. The six terms below form an interconnected web: validity depends on reliability, fairness influences both, and the distinction between criterion-referenced and norm-referenced frameworks shapes what kinds of validity evidence are most important.

Criterion-Referenced Assessment

Measures a student's performance against a fixed standard or set of learning objectives. The question is, 'Can the student do X?' rather than 'How does this student rank?' Examples include state reading benchmarks and curriculum-based measures (CBMs).

Norm-Referenced Assessment

Compares a student's score to the performance of a norming group—a large, representative sample. The question is, 'How does this student perform relative to peers?' Percentile ranks, stanines, and grade equivalents are common norm-referenced scores.

Reliability

The degree to which an assessment yields consistent, stable results across occasions, raters, or forms. Key types include test-retest, inter-rater, internal consistency (Cronbach's α), and alternate-form reliability.

Validity

The degree to which evidence supports the appropriateness of inferences made from test scores for a stated purpose. Modern validity theory (Messick, 1989) treats validity as a unitary concept supported by multiple sources of evidence: content, criterion, and construct.

Fairness & Equity

Fairness ensures that assessment content, administration, and score interpretation do not advantage or disadvantage any group based on characteristics irrelevant to the construct measured. Equity goes further, requiring that assessment systems actively reduce systemic barriers so that all students have a genuine opportunity to demonstrate their knowledge.

✦

KEY TAKEAWAY

Think of assessment like a bathroom scale. Reliability asks whether the scale gives you the same reading when you step on and off repeatedly. Validity asks whether the scale is actually measuring weight (not, say, air pressure). Fairness asks whether the scale works the same for every person—regardless of how they stand on it. A scale can be perfectly reliable (consistent) yet completely invalid (it measures the wrong thing). A test cannot be valid, however, if it is unreliable; you can't draw accurate inferences from inconsistent data.

SECTION 3

Visual Explanation — The Assessment Quality Framework

The diagram illustrates the hierarchical and interdependent relationships among key assessment concepts. Validity sits at the top because it is the overarching goal: drawing defensible inferences from scores. It depends on reliability, which feeds into both criterion-referenced and norm-referenced interpretations. Fairness and equity underpin the entire framework, represented by dashed lines connecting to every other element.

Notice that the diagram places validity at the apex because the 2014 Standards for Educational and Psychological Testing treat it as the most important property of an assessment: without sound evidence that score interpretations are meaningful, even perfectly consistent (reliable) scores are useless. Meanwhile, fairness and equity form the base because inequitable assessment practices can undermine validity itself—if a test systematically disadvantages a subgroup for reasons unrelated to the construct, the inferences drawn from those scores are, by definition, invalid for that group. This interconnection is a favorite testing point on the Praxis 5302: you should expect scenarios that require you to recognize how one concept (e.g., reliability) constrains or supports another (e.g., validity).

SECTION 4

How These Concepts Work in Practice

Types of Reliability

Reliability is not a single, monolithic property; it is assessed through several methods, each addressing a different source of measurement error. Test-retest reliability evaluates stability over time by administering the same test to the same group on two occasions and correlating the scores. Alternate-form reliability uses two equivalent versions of a test to determine whether form differences introduce error. Internal consistency methods, such as Cronbach's alpha (α) or split-half reliability, estimate how well items within a single administration measure the same construct. Finally, inter-rater reliability quantifies agreement among scorers, which is especially critical for subjective assessments like running records or writing rubrics.

INTERNAL CONSISTENCY — CRONBACH'S ALPHA

α = (k / (k − 1)) × (1 − (Σσ²ᵢ / σ²ₜ))

where k = number of items, σ²ᵢ = variance of each item, and σ²ₜ = total test variance. Values above 0.80 are generally considered acceptable for educational decisions; values above 0.90 are preferred for high-stakes interpretations.

Types of Validity Evidence

Modern validity theory (Messick, 1989; AERA/APA/NCME, 2014) describes validity as a unitary concept supported by multiple sources of evidence. Content validity evidence demonstrates that test items adequately represent the domain of knowledge the test claims to measure. Criterion-related validity evidence (concurrent or predictive) shows that test scores correlate with an external outcome, such as future reading performance. Construct validity evidence examines whether the test actually measures the theoretical construct it purports to measure. On the Praxis, you may be asked to identify which type of validity evidence is most relevant to a given scenario—for instance, whether a new phonics screener actually correlates with later decoding ability (predictive criterion-related evidence).

STANDARD ERROR OF MEASUREMENT (SEM)

SEM = σ × √(1 − r)

where σ = standard deviation of test scores and r = reliability coefficient. A smaller SEM means greater precision. Notice that as reliability (r) approaches 1.0, the SEM approaches 0—perfect reliability would mean zero measurement error.

💡

Praxis Tip

The Praxis 5302 will not ask you to compute Cronbach's alpha, but it will expect you to interpret reliability coefficients (e.g., 'A test with a reliability of 0.65 is likely inadequate for placement decisions') and to explain why high reliability is a necessary but not sufficient condition for validity.

SECTION 5

Criterion-Referenced vs. Norm-Referenced — A Detailed Comparison

Perhaps no distinction in assessment vocabulary is tested more frequently on the Praxis 5302 than the difference between criterion-referenced and norm-referenced assessments. These two frameworks answer fundamentally different questions and yield fundamentally different kinds of scores. Confusing them can lead to misinterpretation of data, inappropriate instructional decisions, and even inequitable placement of students.

This side-by-side comparison highlights the fundamental differences in purpose, score types, exemplar instruments, and best uses. Notice that some contemporary assessments straddle both categories, offering both cut scores (criterion-referenced) and percentile ranks (norm-referenced).

Feature	Criterion-Referenced	Norm-Referenced
Interpretation basis	Fixed performance standard (criterion)	Comparison to norming sample
Item selection	Items matched to specific learning objectives	Items selected to maximize score spread (discrimination)
Score distribution goal	All students could theoretically pass	Produces a normal (bell-shaped) distribution
Sensitivity to instruction	High — designed to detect growth toward mastery	Lower — designed for stable rank ordering
Typical use in RTI/MTSS	Tier 1 benchmarking and Tiers 2–3 progress monitoring	Universal screening and eligibility determination

Key distinctions between criterion- and norm-referenced assessments

SECTION 6

Worked Example — Selecting and Evaluating an Assessment

Imagine you are a reading specialist choosing a diagnostic assessment for second-grade students who are not meeting the benchmark on your school's universal screener. Walk through the following scenario to apply the vocabulary concepts from this lesson.

Scenario: Evaluating the 'ReadCheck Diagnostic' for Tier 2 Placement

1
Step 1 — Identify the Purpose

You need to determine specific skill deficits (phonemic awareness, phonics, fluency) so you can plan targeted intervention. Because you are asking 'Can this student decode CVC words?' rather than 'How does this student rank among peers?', you need a criterion-referenced assessment.

Assessment type needed: criterion-referenced diagnostic
2
Step 2 — Check Reliability Evidence

The ReadCheck manual reports a test-retest reliability of 0.72 and an internal consistency (Cronbach's α) of 0.78. For a diagnostic instrument that will guide instructional decisions, you want reliability of at least 0.80 and preferably 0.90+. These values are marginal—scores may fluctuate enough to lead to misclassification.

Reliability judgment: Marginal (below recommended threshold)
3
Step 3 — Evaluate Validity Evidence

The manual reports strong content validity evidence: items were developed by reading experts and align with the Common Core phonics standards. However, no predictive criterion-related validity evidence is provided—there is no study showing that ReadCheck scores predict later reading achievement. You note this gap because predictive evidence would strengthen confidence that low scores actually signal students who will struggle without intervention.

Validity judgment: Content evidence adequate; criterion-related evidence missing
4
Step 4 — Assess Fairness and Equity

You examine whether the norming sample (even for criterion-referenced tests, benchmark cut scores are often field-tested on a sample) represents the demographics of your student population, which is 60% multilingual learners. The manual states that English learners made up only 8% of the field-test sample. This raises a fairness concern: items may contain linguistic bias, and cut scores may not generalize to your population.

Fairness judgment: Potential bias — underrepresentation of ELLs in field-test sample
5
Step 5 — Make a Decision

Based on this analysis, you conclude that the ReadCheck has useful content but insufficient reliability, missing predictive validity evidence, and potential fairness concerns for your population. You recommend supplementing it with a more established criterion-referenced tool (e.g., DIBELS subtests) and conducting follow-up progress monitoring to triangulate the data before making placement decisions.

Decision: Do not use ReadCheck as the sole placement instrument; supplement with additional measures.

SECTION 7

Strengths and Limitations of Key Assessment Concepts

No single assessment property guarantees high-quality measurement. Each concept in the assessment vocabulary toolkit has strengths and limitations, and a reading specialist must weigh them in context. The table below summarizes the practical trade-offs associated with each concept.

Concept	Strengths	Limitations / Cautions
Criterion-Referenced	Directly linked to instructional objectives; all students can potentially 'pass'; sensitive to instruction	Cut scores can be arbitrary; does not indicate relative standing; limited utility for identifying giftedness
Norm-Referenced	Provides relative standing; well-established psychometric properties; useful for eligibility decisions	Norms can become outdated; may not align with local curriculum; encourages rank-ordering rather than growth
Reliability	Essential for consistent decision-making; can be quantified; multiple methods allow triangulation	High reliability alone does not ensure validity; some constructs (e.g., motivation) are inherently less stable
Validity	Most important quality; protects against misuse; modern framework is comprehensive	Always a matter of degree, not all-or-nothing; validation is ongoing and context-specific
Fairness & Equity	Ensures ethical use; protects vulnerable populations; strengthens overall validity	Achieving full equity requires systemic changes beyond the test itself; bias detection is complex

Strengths and limitations of core assessment concepts

✦

KEY TAKEAWAY

Think of these assessment concepts like the instruments on an airplane dashboard. No single gauge tells the pilot everything—airspeed without altitude is dangerous. Similarly, a reading specialist who considers only reliability without examining validity, or who uses a norm-referenced test to measure mastery of a specific skill, is flying partially blind. The Praxis 5302 tests your ability to read the whole dashboard and make integrated, evidence-based judgments.

SECTION 8

Connecting to Advanced Assessment Theory

The basic vocabulary covered in this lesson connects to more sophisticated frameworks that you may encounter in graduate coursework or in advanced Praxis items. Understanding these connections helps you see each term not as an isolated definition but as part of a living, evolving measurement science. The table below maps the foundational concepts to their advanced extensions.

Foundational Concept	Advanced Extension	Why It Matters for Reading Specialists
Reliability (Classical)	Generalizability Theory (G-theory) — accounts for multiple sources of error simultaneously (items, raters, occasions)	Helps you understand why a running record scored by two different teachers might yield different results—and how to design scoring protocols that minimize such error.
Validity (Messick)	Argument-Based Validity (Kane, 2006) — structures validity as a logical argument with claims, warrants, and rebuttals	When selecting a new screener, you can systematically evaluate each claim the publisher makes about the test's interpretive use.
Criterion-Referenced	Standard-Setting Methods (Angoff, Bookmark) — systematic processes for establishing cut scores	Understanding how a 'benchmark' score on DIBELS was determined (and its uncertainty) helps you make more nuanced decisions at the margins.
Norm-Referenced	Item Response Theory (IRT) — models item difficulty and student ability on a common scale, improving norm tables	IRT-based adaptive tests (e.g., MAP Growth) produce more precise ability estimates with fewer items, saving instructional time.
Fairness / Equity	Differential Item Functioning (DIF) — statistical detection of items that function differently across demographic groups	If a reading comprehension item about skiing is harder for urban students not because of reading ability but because of background knowledge, DIF analysis can flag it for removal.

Foundational vocabulary mapped to advanced assessment theory

While the Praxis 5302 will not require you to perform DIF analyses or construct a Kane validity argument, it will expect you to recognize that assessment quality is multidimensional and that each foundational concept opens a doorway to deeper inquiry. Reading specialists who can speak this language fluently are better advocates for their students, better consumers of test manuals, and better collaborators with school psychologists and district assessment coordinators.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A reading specialist says, 'This test is reliable, so it must be valid.' Explain why this statement is problematic, using the relationship between reliability and validity to support your answer.

Show answer

The statement conflates two distinct concepts. Reliability is a necessary but not sufficient condition for validity. A test can produce highly consistent (reliable) scores and still fail to measure the construct it claims to measure. For example, a phonics test that actually assesses memorized sight words may yield consistent results (high reliability) but provide misleading inferences about phonics ability (low validity). Validity requires evidence that score interpretations are appropriate for the intended purpose; reliability merely ensures that the scores are stable enough to interpret at all.

Question 2: BASIC CALCULATION

A fluency assessment has a standard deviation (σ) of 15 words correct per minute and a reliability coefficient (r) of 0.91. Calculate the Standard Error of Measurement (SEM) and explain what it means in practical terms.

Show answer

Using the formula SEM = σ × √(1 − r), we substitute: SEM = 15 × √(1 − 0.91) = 15 × √0.09 = 15 × 0.30 = 4.5 WCPM. This means that if a student scores 85 WCPM, the 'true score' likely falls within approximately one SEM above or below the observed score (roughly 80.5 to 89.5 WCPM about 68% of the time). If the benchmark cut score is 90 WCPM, this student's score is within the margin of error, so the specialist should not rely on a single administration to make a high-stakes decision.

Question 3: INTERMEDIATE

A district adopts a new reading comprehension test and reports strong content validity evidence. A teacher notices that the test correlates only weakly (r = 0.35) with students' grades in reading class. Which type of validity evidence does this correlation represent, and what might explain the weak correlation without necessarily invalidating the test?

Show answer

The correlation between test scores and reading class grades represents concurrent criterion-related validity evidence because it compares the test to an external criterion measured at approximately the same time. A weak correlation does not automatically invalidate the test. Several explanations are possible: reading class grades may reflect effort, homework completion, and participation—constructs distinct from reading comprehension. The test may be measuring a narrower construct (e.g., inferential comprehension) than the grade encompasses. Additionally, grades often suffer from restriction of range (most students receive B's and C's), which attenuates correlations. The specialist should seek additional validity evidence (e.g., predictive or construct-related) before drawing conclusions.

Question 4: APPLIED

You are selecting a universal screener for an elementary school where 45% of students are English learners. The screener manual reports a reliability of 0.92 and strong content validity, but the norming sample included only 12% English learners. The manual provides no Differential Item Functioning (DIF) data. Using the concepts of fairness, equity, and validity, construct an argument for or against adopting this screener.

Show answer

While the screener's reliability (0.92) and content validity evidence are strong, the significant underrepresentation of English learners in the norming sample raises serious fairness and equity concerns. Norms derived primarily from monolingual English speakers may not produce valid percentile ranks for ELLs, potentially leading to over-identification for intervention (if language barriers depress scores) or under-identification (if certain items are culturally biased in complex ways). The absence of DIF data means the publisher has not demonstrated that individual items function equivalently across language groups. From a validity standpoint, if 45% of your students are inadequately represented, the inferences drawn from their scores are questionable for that subgroup—validity is always context- and population-specific. The recommendation would be to request supplementary DIF data from the publisher, pilot the screener with your ELL population, and use additional data sources before making placement decisions.

Question 5: CRITICAL THINKING

A colleague argues that criterion-referenced assessments are inherently 'fairer' than norm-referenced assessments because every student is judged against the same standard rather than compared to others. Construct a nuanced response that identifies both the merit and the flaw in this argument, referencing at least three assessment vocabulary terms from this lesson.

Show answer

The colleague's argument has surface merit: criterion-referenced assessments do avoid the zero-sum logic of norm-referenced tests, where by definition some students must fall below the median. In principle, all students can meet a criterion, which aligns with an equity orientation. However, the argument is flawed because fairness is not determined solely by the interpretive framework—it depends on the test content, the standard-setting process, and the population. A criterion-referenced test can still contain culturally biased items, have a cut score that was established using a non-representative sample, or lack adequate validity evidence for diverse populations. Moreover, if the 'fixed standard' itself reflects the experiences of only the dominant culture, then measuring everyone against it is not equitable—it is merely consistently inequitable. True fairness requires examining item content, administration conditions, and score use regardless of the assessment's interpretive framework.

SUMMARY

Lesson Summary

Summary

This lesson explored the foundational technical vocabulary that every reading specialist must command. Criterion-referenced assessments measure student performance against fixed learning standards, while norm-referenced assessments compare performance to a representative peer group. Reliability refers to the consistency and stability of scores across administrations, raters, or forms, commonly quantified by Cronbach's alpha or test-retest correlations. Validity is the degree to which evidence supports the appropriateness of inferences made from test scores, and it is the single most important property of any assessment. Critically, reliability is necessary but not sufficient for validity—a test cannot yield meaningful inferences if its scores are inconsistent, but consistency alone does not guarantee that the test measures what it claims to measure.

Fairness requires that assessments do not advantage or disadvantage any group for reasons irrelevant to the measured construct, and equity extends this principle by demanding that assessment systems actively reduce systemic barriers. On the Praxis 5302, expect scenario-based questions that ask you to evaluate assessments using these interconnected concepts—for example, identifying why a reliable test might lack validity for a specific population, or explaining why a criterion-referenced screener may still raise fairness concerns if its items contain cultural bias. Mastering this vocabulary is not merely an exercise in memorization; it is the foundation for every data-driven decision you will make as a reading specialist.