Data Reconciliation — Reconcile discrepancies across assessment sources

The challenge of reconciling inconsistent information across assessment sources has been a fundamental concern in clinical psychology since the field's earliest attempts at systematic evaluation. Throughout much of the nineteenth century, psychological assessment relied almost exclusively on a single clinician's interview impressions, which meant that discrepancies between data sources were rarely encountered—because only one source existed. As the discipline matured, however, practitioners began supplementing clinical interviews with standardized tests, behavioral observations, collateral reports, and physiological measures, each of which introduced its own perspective on the client's functioning and, inevitably, its own potential for disagreement with other sources.

The concept of data reconciliation emerged from this growing awareness that no single assessment instrument or informant captures the full picture of a person's psychological experience. When a self-report measure suggests moderate depression but a behavioral observation reveals minimal psychomotor retardation, or when a parent describes severe conduct problems that a teacher does not observe, the clinician must determine how to integrate—and sometimes prioritize—these competing narratives. This section traces the historical milestones that shaped our modern approach to managing discrepant assessment data.

The central question this history raises is both practical and epistemological: when multiple sources of clinical data disagree, how should a practitioner determine which data most accurately reflect the client's true psychological state? Addressing this question requires an understanding of the sources of discrepancy, systematic frameworks for evaluating data quality, and clinical judgment grounded in psychometric principles.

Effective data reconciliation rests on several foundational principles that guide clinicians in making sense of conflicting information. These principles draw from psychometric theory, systems thinking, and evidence-based clinical reasoning. Understanding them transforms the experience of encountering discrepant data from a confusing obstacle into a clinically informative process that deepens the overall assessment.

The following diagram illustrates the multi-source assessment convergence model, showing how different data streams feed into a reconciliation process. At the center, the clinician evaluates areas of convergence and divergence, weighing each source according to its psychometric properties and contextual relevance before arriving at an integrated clinical formulation.

As the diagram illustrates, the reconciliation process is not merely additive. The clinician does not simply average across sources. Instead, each finding is categorized based on whether it converges with, diverges from, or directly contradicts other sources. Convergent findings across multiple methods represent the most reliable clinical conclusions. Divergent findings often reveal important context-dependent variation—for instance, a child who exhibits oppositional behavior only at home but not at school. Contradictory findings demand careful hypothesis testing: the clinician must consider response bias, measure validity, situational factors, and diagnostic alternatives before resolving the inconsistency.

To reconcile discrepancies effectively, one must first understand the mechanisms that produce them. In classical test theory, any observed score on a psychological measure is a composite of the true score and error. When we extend this framework across multiple methods and informants, we recognize that different sources introduce systematically different types of error, which accounts for much of the discrepancy clinicians encounter in practice.

Sources of Discrepancy Variance

The method variance component, first articulated in the multi-trait multi-method matrix framework, refers to systematic error attributable to the assessment method itself rather than the construct being measured. For example, self-report inventories are subject to social desirability bias, denial, and limited insight, while projective techniques may introduce examiner interpretation bias. Structured interviews improve inter-rater reliability but may miss nuances captured in free-form clinical dialogue.

Informant variance represents another major source of discrepancy. Achenbach, McConaughy, and Howell (1987) conducted a landmark meta-analysis demonstrating that the average correlation between different informants rating the same child's behavior was only r = .28. This low agreement does not necessarily indicate that one informant is wrong; rather, it reflects the reality that different informants observe the individual in different settings, at different times, and through different relational lenses. Parents observe the child at home during evenings and weekends, teachers observe in structured academic environments, and the child's own self-report captures internal experiences inaccessible to external observers.

Additional sources of discrepancy include temporal variance (symptoms fluctuate over time, so assessments administered days or weeks apart may capture different clinical states), setting variance (behavior is context-dependent), and construct underrepresentation (a measure may tap only a narrow facet of a multidimensional construct). Understanding these variance sources is essential for determining whether a discrepancy reflects genuine clinical complexity or artifactual measurement noise.

When faced with discrepant assessment data, clinicians benefit from a structured decision framework that guides the reconciliation process. The following diagram presents a decision tree that practitioners can use to systematically evaluate and resolve inconsistencies. This framework integrates psychometric considerations, contextual factors, and clinical reasoning into a stepwise process that moves from detection of the discrepancy through to its resolution.

At Step 1, the clinician evaluates the psychometric properties of each discrepant source—internal consistency, test-retest reliability, criterion validity, and normative adequacy. A source with poor psychometric properties receives less weight in the reconciliation. At Step 2, the clinician examines validity indicators embedded within specific measures—for instance, the MMPI-3 validity scales (F, L, K, VRIN, TRIN) may reveal overreporting, underreporting, or random responding that explains the discrepancy. At Step 3, contextual and setting differences are assessed: the same client may genuinely function differently across environments, making the discrepancy itself a valid clinical finding. Step 4 integrates cultural and developmental considerations, recognizing that cultural norms around emotional expression or developmental stage may systematically affect certain assessment modalities more than others. Finally, at Step 5, the clinician generates specific clinical hypotheses to explain the discrepancy and seeks additional data to test them, ultimately arriving at an integrated clinical formulation.

Consider the following clinical scenario, which demonstrates the reconciliation framework in practice. This example involves a 10-year-old boy, Marcus, referred for evaluation due to academic underperformance and behavioral concerns.

Clinicians employ several distinct strategies when reconciling discrepant data, and each carries its own advantages and potential pitfalls. The table below summarizes the major reconciliation strategies, their appropriate applications, and their limitations. Awareness of these trade-offs helps practitioners select the most appropriate approach for each clinical situation.

Data reconciliation as practiced in routine assessment is the foundation for more advanced integrative practices in clinical psychology. As clinicians develop expertise, they move from the structured, stepwise reconciliation described above toward a more fluid, pattern-recognition-based approach that draws on extensive clinical experience and deep knowledge of psychopathology. The table below contrasts the foundational reconciliation skills covered in this lesson with the advanced integrative competencies that emerge with clinical maturation.

Looking ahead, advances in computational modeling and machine learning are beginning to offer algorithmic approaches to data reconciliation that complement clinical judgment. Latent variable models, for instance, can estimate a latent trait (the "true" construct) from multiple fallible indicators, statistically modeling the measurement error associated with each source. Bayesian approaches allow clinicians to update their diagnostic confidence as each new data source is incorporated, providing a formal framework for the intuitive process of hypothesis revision. While these methods are currently more common in research than in clinical practice, they represent the direction in which evidence-based assessment is moving and underscore the importance of the foundational reconciliation competencies covered in this lesson.