Residual Analysis & Influence — Residual Analysis and Outlier/Influence Concepts

The method of least squares was independently developed by Adrien-Marie Legendre and Carl Friedrich Gauss in the early nineteenth century, giving researchers a principled way to fit a line through scattered data. Yet for more than a century, practitioners largely trusted the global fit without examining whether individual observations might be distorting the results. As datasets grew in complexity through the twentieth century, statisticians realized that a single aberrant point could drastically alter slope estimates, confidence intervals, and hypothesis tests. This recognition catalyzed the development of formal diagnostic tools designed to detect problematic observations and assess their impact on regression output.

The central question driving this topic is deceptively simple: can we trust our regression line? A well-fitting model might conceal the fact that one or two observations are pulling the line toward themselves, masking the true pattern in the remaining data. Residual analysis provides the lens through which we inspect each observation's contribution to the model, while influence diagnostics quantify the degree to which removing a single point would alter the conclusions. Together, these tools form an indispensable part of responsible regression modeling.

Before examining computational formulas, it is essential to distinguish the key concepts that underpin residual analysis. A residual is defined as the difference between an observed response value and the value predicted by the fitted regression model. When we examine the collection of all residuals — their distribution, spread, and patterns — we gain critical information about whether our modeling assumptions (linearity, constant variance, normality, independence) are satisfied. Beyond assumption checking, specific observations may warrant special scrutiny because they are unusual in predictable ways.

The most intuitive way to understand residuals is to see them. The diagram below displays a simple linear regression scenario with several data points, the fitted regression line, and vertical line segments representing each residual. Notice how points above the line have positive residuals and points below the line have negative residuals. The diagram also highlights three special cases: a high-leverage point at an extreme x-value, an outlier with a large residual, and an influential observation that exhibits both properties simultaneously.

Observe that high leverage alone does not necessarily distort the regression line — the cyan point sits close to where the line already passes. Similarly, the pink outlier has a large residual but is located near the center of the x-distribution, giving it limited mechanical advantage to tilt the line. The amber point, however, sits far from the x-mean and far from the line, creating a "lever arm" effect that can substantially shift the estimated slope and intercept. This is why influence is the product of outlyingness and leverage — both ingredients are necessary to meaningfully alter the regression fit.

The mathematical underpinnings of residual analysis rest on the hat matrix H, which projects the observed response vector y onto the column space of the design matrix X. In the ordinary least squares framework, the vector of fitted values is ŷ = Hy = X(X^TX)⁻¹X^Ty. The diagonal elements hᵢᵢ of H — called leverage values — measure how far observation i's predictor values are from the centroid of the predictor space, thereby quantifying its potential to influence the fit.

Beyond Cook's distance, several complementary diagnostics exist. DFFITS measures the change in the fitted value for observation i when it is deleted, expressed in units of the standard error of ŷᵢ. DFBETAS provides a separate measure for each regression coefficient, quantifying how much bⱼ changes (in units of its standard error) when observation i is removed. Leverage values (hᵢᵢ) from the hat matrix provide a purely predictor-space measure of unusualness. The table below summarizes these diagnostics, their formulas, and commonly used thresholds.

This four-quadrant framework is the conceptual backbone of influence analysis. In practice, you will use numerical diagnostics (Cook's distance, DFFITS) rather than visual quadrant placement, but the intuition remains: an observation must be unusual in both predictor space and response space to exert meaningful influence on the regression coefficients. A point that is only a leverage point or only an outlier may merit attention for other reasons (model misspecification, data recording errors) but typically does not distort the overall fit as severely.

Suppose we fit the simple linear regression ŷ = 2.5 + 1.8x to n = 20 observations with p = 2 parameters (intercept + slope). We obtain RMSE s = 3.2 from the full model. Consider observation i = 14, which has x₁₄ = 15 (the sample mean of x is 8) and y₁₄ = 38. The model predicts ŷ₁₄ = 2.5 + 1.8(15) = 29.5. Additional information: the leverage value h₁₄,₁₄ = 0.31, and the RMSE computed without observation 14 is s₍₋₁₄₎ = 2.9. We will compute the key diagnostics.

No single diagnostic captures every facet of data quality. Each tool has particular strengths and blind spots, so experienced practitioners typically examine several diagnostics in concert rather than relying on any one measure. The table below compares the most widely used diagnostics across several evaluative dimensions.

The diagnostic framework presented here extends naturally into more advanced settings. In multiple regression with many predictors, leverage reflects the Mahalanobis distance of an observation's predictor vector from the centroid of the predictor space, accounting for correlations among predictors. In generalized linear models (GLMs), residuals are replaced by deviance residuals or Pearson residuals, and the hat matrix is replaced by the generalized hat matrix derived from the iteratively reweighted least squares algorithm. Robust regression methods, such as M-estimation and least trimmed squares, are designed to automatically downweight or exclude influential observations, providing an alternative to the detect-then-decide approach of classical diagnostics.

Understanding the classical residual and influence framework thoroughly prepares you for these extensions. The underlying logic — quantify how much an individual observation affects the model fit, then decide whether that effect is substantively important — remains unchanged regardless of the model class. In courses on generalized linear models, survival analysis, or Bayesian statistics, you will encounter analogous diagnostics built on the same conceptual foundation, making this material a gateway to advanced model-checking methodology.