Regression Diagnostics — Diagnostics: leverage, influence,...

SECTION 1

Historical Context & Motivation

The development of ordinary least squares regression by Legendre and Gauss in the early nineteenth century provided a powerful tool for modeling relationships between variables, yet for over a century practitioners had no systematic framework for evaluating whether individual observations were unduly controlling the fit. Early statisticians relied on visual inspection of scatter plots and subjective judgment to flag problematic data points, an approach that became untenable as datasets grew in dimensionality. The formalization of regression diagnostics as a rigorous subdiscipline emerged primarily in the 1970s and 1980s, when statisticians began exploiting the algebraic structure of the hat matrix and developing quantitative measures of leverage and influence. These tools address a fundamental question in applied modeling: even when a regression passes global tests such as the F-test or R² thresholds, can a small number of observations be disproportionately responsible for the estimated coefficients and apparent goodness of fit?

1805–1809

Foundations of Least Squares

Legendre (1805) and Gauss (1809) independently develop ordinary least squares, establishing the regression framework but without formal diagnostics for individual observations.

1972

The Hat Matrix and Leverage

Hoaglin and Welsch popularize the hat matrix H = X(X^TX)⁻¹X^T as a diagnostic tool, with its diagonal elements h_ii quantifying leverage.

1977

Cook's Distance

R. Dennis Cook introduces Cook's distance, the first widely adopted scalar measure that combines leverage and residual magnitude to assess each observation's influence on the entire coefficient vector.

1980s

DFFITS, DFBETAS, and Residual Plots

Belsley, Kuh, and Welsch (1980) introduce DFFITS and DFBETAS for coefficient-level diagnostics. Standardized, studentized, and externally studentized residuals become standard tools for detecting heteroscedasticity and non-normality.

2000s–Present

Diagnostics for GLMs and Mixed Models

Extensions of classical leverage and influence diagnostics to generalized linear models, mixed-effects models, and high-dimensional settings emerge, using deviance residuals, Pearson residuals, and analogues of the hat matrix.

The central question that regression diagnostics address is: which observations exert disproportionate control over the fitted model, and how do the residuals reveal systematic departures from model assumptions? Without answering this question, a practitioner risks drawing inferences that rest on the peculiarities of a handful of data points rather than on genuine population-level relationships.

SECTION 2

Core Principles & Definitions

Regression diagnostics revolve around three interconnected concepts. Leverage describes how far an observation's predictor values lie from the center of the predictor space, thereby measuring its potential to affect the fitted values. Influence captures the actual effect an observation has on the estimated coefficients or fitted values when it is included versus excluded from the analysis. Residual structure refers to the patterns and distributional properties of the differences between observed and fitted responses, which serve as primary evidence for or against the validity of model assumptions such as linearity, constant variance, and normality. Together, these three pillars allow a modeler to interrogate a regression fit at the observation level, moving beyond aggregate summaries like R² or the F-statistic.

Leverage (h_ii)

The diagonal element of the hat matrix for observation i. It depends only on the predictor values and measures the observation's distance from the centroid of X-space. High leverage means the observation could potentially distort the fit, regardless of its response value.

Influence (Cook's D, DFFITS)

A measure of the actual change in the fitted model when observation i is deleted. Influence combines leverage and residual size: an observation must be both high-leverage and have a large residual to be truly influential.

Residual Types

Ordinary residuals (e_i = y_i − ŷ_i), standardized residuals (dividing by estimated σ), and studentized (externally studentized) residuals (using leave-one-out variance). Each type has distinct diagnostic utility.

Outlier vs. Influential Point

An outlier has a large residual but may or may not change the fit materially. An influential point actually shifts the regression surface. A high-leverage outlier is the most dangerous combination.

Residual Structure & Assumptions

Patterns in residual plots—fanning, curvature, clustering—signal violations of linearity, homoscedasticity, or independence. Residuals should appear as a structureless random scatter when assumptions hold.

✦ KEY TAKEAWAY

Think of a regression fit like a see-saw balanced on a fulcrum. Leverage is how far from the fulcrum an observation sits—farther out means greater mechanical advantage. Influence is the actual tilt the observation causes, which depends on both its distance from the fulcrum (leverage) and how heavy it is (residual magnitude). An observation sitting far out on the see-saw but weighing nothing causes no tilt; a heavy observation near the center barely budges it either. Only a heavy observation far from the fulcrum truly tips the balance.

SECTION 3

Visual Explanation — Leverage vs. Influence

Point A (pink) is an outlier with a large residual but sits near the center of x-space, giving it low leverage and therefore low influence. Point B (amber) has high leverage because its x-value is extreme, but it falls near the regression line, so its influence is small. Point C (red) combines high leverage with a large residual, producing a dramatically different regression line (dashed red) and high influence.

The diagram above captures the essential taxonomy of unusual observations. Leverage is a function of the predictor configuration alone—it tells us where the observation lives in predictor space relative to the centroid of the design matrix. An observation can have extreme x-values (high leverage) yet conform perfectly to the linear trend, in which case it stabilizes rather than distorts the fit. Conversely, an observation near the centroid of x-space has limited ability to rotate the regression hyperplane even if its y-value departs wildly from the fitted value. The truly problematic observations are those that combine both features: they sit in an extreme region of predictor space and deviate substantially from the pattern established by the remaining data.

SECTION 4

Mathematical Framework

The Hat Matrix and Leverage

In the standard linear model y = Xβ + ε, the vector of fitted values is ŷ = Hy, where H is the hat matrix. This matrix is the orthogonal projection onto the column space of X and plays a central role in all diagnostic measures.

HAT MATRIX

H = X(X⊤X)⁻¹X⊤

X is the n × p design matrix. H is n × n, symmetric, and idempotent (H² = H). The diagonal element h_ii is the leverage of observation i. Properties: 0 ≤ h_ii ≤ 1, and Σh_ii = p (trace of H equals the number of parameters). A common rule of thumb flags h_ii > 2p/n as high leverage.

Residual Types

ORDINARY RESIDUAL

eᵢ = yᵢ − ŷᵢ

The raw difference between observed and fitted values. Under the model, Var(eᵢ) = σ²(1 − h_ii), so high-leverage points have smaller residual variance, making raw residuals misleading for diagnostic purposes.

INTERNALLY STUDENTIZED RESIDUAL

rᵢ = eᵢ / (σ̂ √(1 − hᵢᵢ))

Divides the residual by its estimated standard deviation using the full-sample estimate σ̂. This standardizes residuals to have approximately unit variance, but the estimate of σ still includes observation i.

EXTERNALLY STUDENTIZED RESIDUAL

tᵢ = eᵢ / (σ̂₍₋ᵢ₎ √(1 − hᵢᵢ))

Uses σ̂₍₋ᵢ₎, the estimate of σ computed without observation i. Under normality, tᵢ follows a t-distribution with n − p − 1 degrees of freedom, enabling formal outlier tests (Bonferroni-corrected).

Influence Measures

COOK'S DISTANCE

Dᵢ = (β̂ − β̂₍₋ᵢ₎)⊤(X⊤X)(β̂ − β̂₍₋ᵢ₎) / (p × σ̂²) = rᵢ² × hᵢᵢ / (p × (1 − hᵢᵢ))

Cook's distance measures the shift in the entire coefficient vector when observation i is deleted. The second form reveals that D_i is a product of residual size (r_i²) and leverage (h_ii/(1 − h_ii)). Common threshold: D_i > 4/n or D_i > 1.

DFFITS

DFFITSᵢ = tᵢ × √(hᵢᵢ / (1 − hᵢᵢ))

DFFITS measures the change in the fitted value ŷ_i when observation i is deleted, standardized by the leave-one-out standard error. Threshold: |DFFITS_i| > 2√(p/n).

SECTION 5

Residual Structure & Diagnostic Plots

While scalar measures like Cook's distance and DFFITS flag individual observations, the full diagnostic picture requires examination of residual plots that reveal systematic departures from model assumptions. The four most informative diagnostic plots are: (1) residuals versus fitted values, which reveals non-linearity and heteroscedasticity; (2) the scale-location plot (√|standardized residuals| vs. fitted values), which focuses on variance structure; (3) the Q–Q plot of standardized residuals against theoretical normal quantiles, which assesses the normality assumption; and (4) the residuals-versus-leverage plot, which overlays Cook's distance contours to identify influential points. Together, these are often called the "regression diagnostic quartet" and are generated by default in statistical software packages such as R's plot(lm_object).

The diagnostic quartet. Plot 1 (residuals vs. fitted) shows a well-behaved random scatter. Plot 2 (scale-location) reveals fanning indicative of heteroscedasticity. Plot 3 (Q–Q) shows points closely following the reference line, consistent with normality. Plot 4 (residuals vs. leverage) identifies one observation (red star) beyond the Cook's distance D = 0.5 contour, flagging it as influential.

Summary of the four standard diagnostic plots and their interpretation.

Diagnostic Plot	What It Detects	Warning Signs
Residuals vs. Fitted	Non-linearity, heteroscedasticity	Curved smoothing line, funnel/megaphone pattern
Scale-Location (√\|Std. Resid.\| vs. Fitted)	Non-constant variance	Upward trending smoothing line, systematic spread changes
Normal Q–Q	Non-normality of residuals	S-shaped curve (heavy tails), banana curve (skewness), detached points
Residuals vs. Leverage	Influential observations	Points beyond Cook's distance contours (D > 0.5 or D > 1)

⚠ Correlated Residuals

In time-series or spatial regression contexts, residuals may exhibit autocorrelation even when the model is correctly specified for the mean function. The Durbin–Watson test and plots of residuals against time or lag order help detect serial correlation, which violates the independence assumption and leads to underestimated standard errors.

SECTION 6

Worked Example

Suppose we fit a simple linear regression y = β₀ + β₁x + ε to n = 10 observations and obtain the following information for observation i = 7: the hat matrix diagonal is h₇₇ = 0.45, the ordinary residual is e₇ = 4.2, the full-sample root mean squared error is σ̂ = 2.1, the leave-one-out root mean squared error is σ̂₍₋₇₎ = 1.8, and p = 2 parameters are estimated. We want to compute the leverage flag, the internally studentized residual, the externally studentized residual, Cook's distance, and DFFITS.

Diagnostic Measures for Observation 7

Step 1 — Check Leverage ThresholdThe leverage threshold is 2p/n = 2(2)/10 = 0.40. Since h₇₇ = 0.45 > 0.40, observation 7 is flagged as a high-leverage point. Its predictor value is far from the centroid of x.

h₇₇ = 0.45 > 2p/n = 0.40 → High leverage

Step 2 — Internally Studentized ResidualCompute r₇ = e₇ / (σ̂ × √(1 − h₇₇)) = 4.2 / (2.1 × √(1 − 0.45)) = 4.2 / (2.1 × √0.55) = 4.2 / (2.1 × 0.7416) = 4.2 / 1.5574 ≈ 2.70. An internally studentized residual exceeding 2 in absolute value is noteworthy, so this observation has a large residual.

r₇ ≈ 2.70

Step 3 — Externally Studentized ResidualCompute t₇ = e₇ / (σ̂₍₋₇₎ × √(1 − h₇₇)) = 4.2 / (1.8 × 0.7416) = 4.2 / 1.3349 ≈ 3.15. Under normality, t₇ follows a t-distribution with n − p − 1 = 10 − 2 − 1 = 7 degrees of freedom. Comparing |t₇| = 3.15 against the Bonferroni-corrected threshold t₀.₀₂₅/₁₀,₇ ≈ 3.50, the observation narrowly avoids formal outlier rejection, but is clearly suspicious.

t₇ ≈ 3.15 (borderline outlier)

Step 4 — Cook's DistanceUsing the formula D₇ = r₇² × h₇₇ / (p × (1 − h₇₇)) = (2.70)² × 0.45 / (2 × 0.55) = 7.29 × 0.45 / 1.10 = 3.2805 / 1.10 ≈ 2.98. This vastly exceeds both the 4/n = 0.40 threshold and the conventional D > 1 benchmark, indicating that observation 7 is highly influential on the entire set of regression coefficients.

D₇ ≈ 2.98 → Highly influential

Step 5 — DFFITSDFFITS₇ = t₇ × √(h₇₇ / (1 − h₇₇)) = 3.15 × √(0.45 / 0.55) = 3.15 × √0.8182 = 3.15 × 0.9045 ≈ 2.85. The threshold is 2√(p/n) = 2√(2/10) = 2 × 0.4472 ≈ 0.89. Since |DFFITS₇| = 2.85 far exceeds 0.89, this confirms that observation 7 has a dramatic effect on its own fitted value.

DFFITS₇ ≈ 2.85 ≫ 0.89 threshold

✦ INTERPRETING THE RESULTS

Observation 7 is both high-leverage and has a large residual, producing extreme values of Cook's distance and DFFITS. The appropriate response is not automatic deletion; rather, the analyst should investigate whether this observation reflects a data entry error, belongs to a different population, or represents a genuine but extreme case. If legitimate, robust regression methods or sensitivity analyses are warranted.

SECTION 7

Strengths & Limitations of Diagnostic Measures

No single diagnostic statistic is sufficient for comprehensive model checking. Each measure captures a particular aspect of an observation's role in the regression, and understanding the strengths and limitations of the main diagnostics guides effective practice.

Comparison of key diagnostic measures: what each captures and where each falls short.

Diagnostic Measure	Strengths	Limitations
Leverage (h_ii)	Computed from X alone, no model fitting required; identifies design space extremes	Does not account for the response; a high-leverage point on the regression line is benign
Cook's Distance	Combines leverage and residual size into one scalar; measures global coefficient change	Summarizes influence on all coefficients jointly, which may mask influence on a single coefficient; cutoffs are approximate
DFFITS	Uses externally studentized residuals; measures impact on the observation's own fitted value	Focused on the fitted value at the specific observation, not on any particular coefficient
DFBETAS	Identifies which specific coefficient(s) are most affected by each observation	Produces p values per observation, creating a large amount of output; harder to summarize
Externally Studentized Residuals	Has a known t-distribution under normality; enables formal outlier testing	Masking and swamping effects: multiple outliers can distort each other's residuals

✦ BEST PRACTICE

Effective regression diagnostics require a battery of complementary tools rather than reliance on a single measure. Scalar statistics like Cook's distance should be examined alongside residual plots to detect global assumption violations. When influential points are identified, the first response should be investigation, not deletion: check for recording errors, consider whether the observation belongs to a different subpopulation, and conduct sensitivity analyses comparing models with and without the flagged observations.

SECTION 8

Extensions to GLMs and Beyond

The diagnostic framework developed for the normal linear model extends—with modifications—to generalized linear models (GLMs) and mixed-effects models. In GLMs, the hat matrix is replaced by its generalized analogue computed from the final iteration of the iteratively reweighted least squares (IRLS) algorithm, and residuals take specialized forms—Pearson residuals, deviance residuals, and Anscombe residuals—each correcting for the non-constant variance inherent in non-Gaussian responses. Cook's distance and DFFITS are redefined using the appropriate residual and leverage quantities, but the conceptual interpretation remains the same: influence = leverage × residual magnitude.

Diagnostics in the normal linear model versus generalized linear models.

Feature	Normal Linear Model	GLMs
Hat matrix	H = X(X⊤X)⁻¹X⊤, fixed	H = W^(1/2)X(X⊤WX)⁻¹X⊤W^(1/2), depends on fitted values via weight matrix W
Residual types	Ordinary, standardized, externally studentized	Pearson, deviance, Anscombe, working residuals
Variance structure	Var(y) = σ² (constant)	Var(y) = φ × V(μ), variance function of the mean
Cook's distance	Exact deletion formula available	One-step approximation based on IRLS; exact computation requires re-fitting
Normality of residuals	Expected under correct model	Deviance residuals are approximately normal for large samples; Pearson residuals can be skewed

In modern high-dimensional settings where p may approach or exceed n, classical leverage becomes less informative because many observations have leverage close to 1. Research in this area has developed random-projection-based leverage approximations and regularization-aware influence diagnostics. Similarly, for mixed-effects models, influence can be assessed at both the observation level and the group level, using tools such as the likelihood displacement and conditional Cook's distance. As modeling methodologies continue to evolve, so too does the toolkit for diagnosing whether the conclusions drawn from a fitted model are robust or fragile.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why a high-leverage observation does not necessarily have a large residual. Under what conditions can a high-leverage point actually have a residual of exactly zero?

PROBLEM 2 — BASIC CALCULATION

A regression with n = 20 observations and p = 4 parameters yields h₅₅ = 0.35, e₅ = 3.0, and σ̂ = 1.5. Compute the internally studentized residual r₅ and determine whether observation 5 is flagged as high leverage.

PROBLEM 3 — INTERMEDIATE

For a model with n = 25 and p = 3, observation 12 has h₁₂,₁₂ = 0.42, an internally studentized residual of r₁₂ = 1.8, and σ̂₍₋₁₂₎ = 2.0. Compute Cook's distance D₁₂ and DFFITS₁₂. Is this observation influential by standard criteria? Also compute the externally studentized residual if e₁₂ = 3.24 is given.

PROBLEM 4 — APPLIED

A researcher fits a multiple regression predicting house prices from square footage, number of bedrooms, and neighborhood income (n = 200, p = 4). The residuals-versus-fitted-values plot shows a clear funnel shape (variance increasing with fitted values). One observation—a mansion with 8,000 square feet in a wealthy neighborhood—has h = 0.15, a studentized residual of 0.3, but the sale price of $2.1M is actually well predicted. Should this observation be deleted? What does the funnel pattern suggest, and how would you address it?

PROBLEM 5 — CRITICAL THINKING

Consider the phenomenon of 'masking': when two or more outliers are present, their joint effect can cause each individual observation's diagnostic measures to appear benign. Explain mechanistically why this occurs in terms of the hat matrix and residual computation, and propose a diagnostic strategy to detect masked outliers.

SUMMARY

Summary

Regression diagnostics provide the critical toolset for moving beyond aggregate model summaries to observation-level scrutiny. Leverage, quantified by the hat matrix diagonal h_ii, identifies observations occupying extreme positions in predictor space. Influence measures such as Cook's distance and DFFITS combine leverage with residual magnitude to assess the actual distortion each observation causes. The key insight is that influence = leverage × discrepancy: an observation must be both extreme in X-space and poorly fit to substantially alter the model.

Residual structure analysis through the diagnostic quartet—residuals versus fitted values, scale-location plots, Q–Q plots, and residuals versus leverage—reveals violations of linearity, homoscedasticity, and normality. These tools extend naturally to GLMs through generalized residuals and weight-adjusted hat matrices. In all contexts, the diagnostic workflow should be investigative rather than mechanical: flagged observations warrant substantive inquiry, not automatic deletion.