Deviance & Goodness-of-Fit — Deviance and goodness-of-fit...

SECTION 1

Historical Context & Motivation

In classical linear regression, assessing model fit is relatively straightforward: residual sums of squares and the coefficient of determination R² provide intuitive summaries of how well predicted values track observed outcomes. However, as statisticians began extending regression techniques to non-normal response distributions—binary outcomes, count data, survival times—the familiar least-squares machinery no longer applied. The fundamental challenge was clear: how does one measure the discrepancy between a fitted model and observed data when the error structure is no longer Gaussian and the response is no longer continuous? The answer emerged through the unifying framework of maximum likelihood estimation and the concept of deviance, which generalizes the residual sum of squares by comparing log-likelihoods rather than squared residuals.

1922

Fisher's Likelihood Framework

R. A. Fisher formalizes maximum likelihood estimation, providing the inferential foundation upon which deviance and likelihood ratio tests would later be built.

1938

Neyman–Pearson Likelihood Ratio Tests

Jerzy Neyman and Egon Pearson develop the likelihood ratio test as an optimal framework for hypothesis testing, establishing the −2 log-likelihood ratio as a pivotal statistic.

1972

Nelder & Wedderburn's GLM Unification

John Nelder and Robert Wedderburn publish their landmark paper unifying regression, ANOVA, logistic regression, and log-linear models under the generalized linear model (GLM) framework, introducing deviance as the canonical measure of goodness-of-fit.

1983

McCullagh & Nelder's Treatise

The publication of Generalized Linear Models provides a comprehensive treatment of deviance, scaled deviance, and residual analysis, becoming the definitive reference for GLM practitioners.

1990s–Present

Information-Theoretic Extensions

AIC, BIC, and cross-validation techniques complement deviance-based assessment, enabling principled model comparison in complex settings including mixed models and Bayesian frameworks.

The central question that deviance addresses is deceptively simple: given a fitted model, how much information is lost compared to a model that perfectly reproduces the observed data? By framing goodness-of-fit as a comparison of log-likelihoods, deviance provides a single, interpretable statistic that works uniformly across all members of the exponential family—normal, binomial, Poisson, gamma, and beyond. Understanding deviance is therefore essential for anyone working with GLMs, as it underpins model diagnostics, nested model comparisons via likelihood ratio tests, and the broader logic of statistical model selection.

SECTION 2

Core Principles & Definitions

The conceptual architecture of deviance rests on a few interconnected ideas. At its core, deviance measures the distance between two models using the language of likelihood. To appreciate this, one must first understand the roles played by the saturated model, the fitted model, and the null model. These three reference points define the landscape within which goodness-of-fit is assessed.

Saturated Model

A model with as many parameters as observations, fitting the data perfectly. It achieves the maximum attainable log-likelihood ℓ(ŷ = y) and serves as the upper reference point against which all fitted models are compared.

Deviance (D)

Defined as D = −2[ℓ(fitted) − ℓ(saturated)], deviance quantifies the log-likelihood gap between the fitted model and the saturated model. Smaller deviance indicates better fit. It is always non-negative.

Null Deviance

The deviance of the intercept-only model (no predictors). Comparing null deviance to residual deviance reveals the proportion of likelihood-based 'variation' explained by the predictors.

Likelihood Ratio Test (LRT)

Nested models can be compared via the difference in deviances, which under H₀ follows a χ² distribution with degrees of freedom equal to the difference in the number of parameters.

Scaled Deviance

When the dispersion parameter ϕ is not unity (e.g., in gamma or quasi-Poisson models), the scaled deviance D* = D / ϕ is used instead, ensuring the asymptotic χ² reference distribution remains valid.

✦ KEY TAKEAWAY

Think of deviance as a generalized 'residual sum of squares' for GLMs. In ordinary least squares, the RSS measures how far predictions are from observations in squared-Euclidean distance. Deviance does the same thing but measures distance in log-likelihood units. Just as a small RSS relative to the total SS indicates a good linear fit, a small residual deviance relative to the null deviance indicates that predictors have substantially improved the model's ability to explain the data through the lens of the assumed distribution.

SECTION 3

Visual Explanation

The Deviance Landscape: Saturated, Fitted, and Null Models

The three bars represent log-likelihoods for the saturated (perfect-fit), fitted (proposed), and null (intercept-only) models. Residual deviance D is the gap between the saturated and fitted models, while null deviance D₀ spans from saturated to null. The difference ΔD = D₀ − D captures how much the predictors improve fit.

The diagram above illustrates the fundamental geometry of deviance. The saturated model sits at the top, representing the best any model from the same distributional family could achieve. It has n parameters for n observations, so it reproduces the data exactly. Below it, the fitted model uses p < n parameters, trading perfect fit for parsimony. The gap between these two log-likelihoods, multiplied by −2, gives the residual deviance. At the bottom lies the null model, which ignores all predictors. The null deviance captures the total amount of 'explainable' discrepancy, analogous to the total sum of squares in OLS. By comparing residual deviance to null deviance, one can gauge how much predictive power the covariates contribute.

SECTION 4

Mathematical Framework

The mathematical formulation of deviance is rooted in the exponential family of distributions. Recall that for a response Y with probability density or mass function belonging to the exponential dispersion family, the log-likelihood contribution of a single observation can be written in canonical form. The deviance statistic emerges naturally as twice the log-likelihood ratio between the saturated and fitted models.

EXPONENTIAL FAMILY LOG-LIKELIHOOD

f(y; θ, ϕ) = exp{ [yθ − b(θ)] / a(ϕ) + c(y, ϕ) }

Here θ is the canonical parameter, ϕ is the dispersion parameter, b(θ) is the cumulant function, and a(ϕ) and c(y, ϕ) are known functions. The mean is μ = b′(θ) and Var(Y) = b″(θ) · a(ϕ).

DEVIANCE (GENERAL FORM)

D(y; μ̂) = 2 Σᵢ [ ℓᵢ(ŷᵢ = yᵢ) − ℓᵢ(μ̂ᵢ) ] · a(ϕ)

Equivalently, D = 2 Σᵢ [ yᵢ(θ̃ᵢ − θ̂ᵢ) − b(θ̃ᵢ) + b(θ̂ᵢ) ], where θ̃ᵢ is the canonical parameter under the saturated model (setting μ̃ᵢ = yᵢ) and θ̂ᵢ is the canonical parameter under the fitted model. The factor a(ϕ) is typically absorbed when ϕ is known.

DEVIANCE FOR POISSON GLM

D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) − (yᵢ − μ̂ᵢ) ]

For Poisson data with log link, the canonical parameter is θ = ln(μ). When yᵢ = 0, the convention 0 · ln(0) = 0 is applied. The dispersion parameter ϕ = 1 for Poisson, so scaled and unscaled deviance coincide.

DEVIANCE FOR BINOMIAL GLM

D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) + (nᵢ − yᵢ) ln((nᵢ − yᵢ) / (nᵢ − μ̂ᵢ)) ]

Here yᵢ is the number of successes, nᵢ is the number of trials, and μ̂ᵢ = nᵢπ̂ᵢ where π̂ᵢ is the estimated success probability. Again ϕ = 1 for binomial, so D is directly comparable to a χ² distribution under regularity conditions.

⚠ Asymptotic Distribution

Under the null hypothesis that the fitted model is correct, the residual deviance D is asymptotically distributed as χ²(n − p), where n is the number of observations and p is the number of estimated parameters. This holds rigorously when a(ϕ) = ϕ/wᵢ with known ϕ. For binomial and Poisson models (ϕ = 1), the approximation improves with increasing group sizes or rates, but can be poor for sparse data—particularly in logistic regression with individual binary observations.

SECTION 5

Deviance Across Exponential Family Members

One of the most elegant features of the deviance framework is its universality: every member of the exponential dispersion family has a specific deviance formula derived from the general expression. Understanding these family-specific forms builds intuition for what deviance is actually measuring in each context, and reveals why deviance reduces to familiar quantities in certain special cases.

Each card shows the deviance formula for a major exponential family member. Note that the Normal deviance reduces to the residual sum of squares, confirming that ordinary least squares is a special case of the deviance framework.

Deviance contributions by exponential family member

Family	Canonical Link	ϕ	Deviance Unit Contribution dᵢ
Normal	Identity: g(μ) = μ	σ² (estimated)	(yᵢ − μ̂ᵢ)²
Poisson	Log: g(μ) = ln(μ)	1 (fixed)	2[yᵢ ln(yᵢ/μ̂ᵢ) − (yᵢ − μ̂ᵢ)]
Binomial	Logit: g(π) = ln[π/(1−π)]	1 (fixed)	2[yᵢ ln(yᵢ/μ̂ᵢ) + (nᵢ−yᵢ)ln((nᵢ−yᵢ)/(nᵢ−μ̂ᵢ))]
Gamma	Inverse: g(μ) = 1/μ	1/ν (estimated)	2[−ln(yᵢ/μ̂ᵢ) + (yᵢ − μ̂ᵢ)/μ̂ᵢ]

SECTION 6

Worked Example: Poisson Regression Deviance

Suppose a researcher models the count of insurance claims yᵢ as a function of driver age using a Poisson GLM with a log link. The model is fitted to n = 5 age groups with observed counts y = (2, 5, 8, 4, 1) and fitted values μ̂ = (2.5, 4.2, 6.8, 3.8, 1.7). We will compute the residual deviance, compare it to a χ² distribution, and interpret the result.

Computing Poisson Deviance

Step 1 — State the Deviance FormulaFor a Poisson GLM, the deviance is D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) − (yᵢ − μ̂ᵢ) ]. We evaluate this sum over i = 1, …, 5 using the observed and fitted values.

Step 2 — Compute Each Unit Deviance dᵢFor observation 1: d₁ = 2[2 ln(2/2.5) − (2 − 2.5)] = 2[2(−0.2231) + 0.5] = 2[−0.4463 + 0.5] = 2(0.0537) = 0.1075. For observation 2: d₂ = 2[5 ln(5/4.2) − (5 − 4.2)] = 2[5(0.1744) − 0.8] = 2[0.8720 − 0.8] = 2(0.0720) = 0.1440. For observation 3: d₃ = 2[8 ln(8/6.8) − (8 − 6.8)] = 2[8(0.1625) − 1.2] = 2[1.3000 − 1.2] = 2(0.1000) = 0.2000. For observation 4: d₄ = 2[4 ln(4/3.8) − (4 − 3.8)] = 2[4(0.0513) − 0.2] = 2[0.2053 − 0.2] = 2(0.0053) = 0.0105. For observation 5: d₅ = 2[1 ln(1/1.7) − (1 − 1.7)] = 2[1(−0.5306) + 0.7] = 2(0.1694) = 0.3388.

d = (0.1075, 0.1440, 0.2000, 0.0105, 0.3388)

Step 3 — Sum to Get Residual DevianceD = Σ dᵢ = 0.1075 + 0.1440 + 0.2000 + 0.0105 + 0.3388 = 0.8008.

D = 0.801

Step 4 — Determine Degrees of FreedomThe model has n = 5 observations and p = 2 parameters (intercept + age coefficient). The residual degrees of freedom are n − p = 5 − 2 = 3.

df = 3

Step 5 — Compare to χ²(3) and InterpretUnder the null hypothesis that the Poisson model is adequate, D ~ χ²(3) approximately. The critical value at α = 0.05 is χ²₀.₉₅(3) = 7.815. Since D = 0.801 ≪ 7.815, we have no evidence of lack of fit. The p-value is P(χ²(3) > 0.801) ≈ 0.849, indicating excellent model adequacy. The ratio D/df = 0.801/3 = 0.267 is well below 1, suggesting no overdispersion.

p ≈ 0.849 — No evidence of lack of fit

SECTION 7

Deviance vs. Alternative Goodness-of-Fit Measures

Deviance is not the only goodness-of-fit statistic available for GLMs. The Pearson chi-squared statistic, Akaike Information Criterion (AIC), and Hosmer–Lemeshow test all serve related but distinct purposes. Understanding when each measure is appropriate prevents misdiagnosis of model problems.

Comparison of major goodness-of-fit measures for GLMs

Criterion	Strengths	Limitations
Deviance	Directly linked to likelihood ratio tests; additive over nested models; canonical measure in GLM theory; provides individual deviance residuals for diagnostics.	χ² approximation can be poor for sparse binary data; not meaningful for comparing non-nested models; requires known ϕ for exact test.
Pearson χ²	More robust to sparse data than deviance; familiar quadratic form; useful for estimating dispersion via X²/(n−p).	Not additive for nested model comparisons; less directly tied to likelihood theory; same asymptotic distribution as deviance under regularity conditions.
AIC	Allows comparison of non-nested models; penalizes complexity; grounded in information theory (Kullback–Leibler divergence).	No formal hypothesis test; sensitive to sample size; requires models fit to the same data.
Hosmer–Lemeshow	Specifically designed for logistic regression with binary outcomes; circumvents sparse-data problem by grouping observations.	Arbitrary choice of number of groups; low power; test result depends on grouping strategy; not generalizable beyond binomial models.

✦ KEY TAKEAWAY

Think of deviance as the 'official language' of the GLM framework—it is the native measure that comes directly from the likelihood, just as RSS comes naturally from least squares. Alternative measures like Pearson χ² and AIC are complementary tools for specific situations. When comparing nested models, the difference in deviances is the likelihood ratio test statistic. When comparing non-nested models, switch to AIC or BIC. When checking overall model adequacy for sparse binary data, prefer the Hosmer–Lemeshow test or Pearson χ².

SECTION 8

Connections to Advanced Theory

Deviance sits at a crossroads between classical hypothesis testing and modern model selection theory. Its connections extend in several important directions: toward information criteria, quasi-likelihood methods, Bayesian model comparison, and the analysis of deviance (ANODEV) for factor effects in GLMs.

From deviance fundamentals to advanced extensions

Core Concept	Advanced Extension	Key Relationship
Deviance D	AIC = D + 2p	AIC is deviance plus a complexity penalty; minimizing AIC trades fit against parsimony via an asymptotic estimate of out-of-sample prediction loss.
Scaled deviance D/ϕ	Quasi-likelihood	When the full distribution is unknown, quasi-likelihood methods estimate ϕ from the data and use scaled deviance for inference, relaxing distributional assumptions.
Deviance residuals rᵢᴰ	Residual diagnostics	Deviance residuals rᵢᴰ = sign(yᵢ − μ̂ᵢ)√dᵢ are more symmetrically distributed than Pearson residuals, making QQ-plots and influence diagnostics more reliable.
ΔD for nested models	Analysis of Deviance (ANODEV)	Sequential deviance reduction by adding terms mirrors sequential ANOVA. Type I and Type III deviance tables decompose the contribution of each predictor or factor.
−2 log-likelihood	DIC (Bayesian)	The Deviance Information Criterion extends deviance to Bayesian hierarchical models by replacing p with the effective number of parameters pD estimated from the posterior.

A particularly important extension is the concept of deviance residuals. Defined as rᵢᴰ = sign(yᵢ − μ̂ᵢ) × √dᵢ, where dᵢ is the unit deviance for observation i, these residuals decompose the total deviance into per-observation contributions. Unlike raw or Pearson residuals, deviance residuals tend to be more normally distributed, particularly for Poisson and binomial models. This property makes them invaluable for diagnostic plots: a QQ-plot of deviance residuals against normal quantiles should approximate a straight line when the model is correctly specified. Systematic departures indicate misspecified link functions, omitted predictors, or distributional inadequacy.

🔭 Looking Ahead

In mixed-effects GLMs (GLMMs), the concept of deviance extends but becomes more nuanced. The marginal likelihood integrates over random effects, making standard deviance-based tests inapplicable in their classical form. Restricted maximum likelihood (REML) deviance is used for variance component estimation, while conditional AIC and parametric bootstrap tests replace simple χ² comparisons for testing random effects. Understanding classical deviance thoroughly provides the conceptual foundation for navigating these more complex settings.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

Explain why the deviance of a normal linear model with identity link and known variance σ² reduces to RSS/σ². What does this imply about the relationship between deviance-based inference and classical F-tests in ordinary least squares regression?

PROBLEM 2 — BASIC CALCULATION

A Poisson GLM is fitted to n = 4 observations with observed counts y = (3, 7, 2, 6) and fitted values μ̂ = (3.5, 6.0, 2.8, 5.7). Compute the residual deviance D. Using p = 2 parameters, determine the residual degrees of freedom and assess model fit at α = 0.05.

PROBLEM 3 — INTERMEDIATE

A researcher fits two nested logistic regression models to the same dataset (n = 200). Model A (three predictors) has deviance D_A = 185.3 with p_A = 4 parameters. Model B (one additional predictor) has deviance D_B = 178.6 with p_B = 5 parameters. Perform a likelihood ratio test at α = 0.05 to determine whether the additional predictor significantly improves fit. Also compute the approximate pseudo-R² based on deviance for Model B, given that the null deviance is D₀ = 246.8.

PROBLEM 4 — APPLIED

An epidemiologist models the number of hospital infections per ward using a Poisson GLM with a log link. The fitted model yields a residual deviance of 58.4 on 30 degrees of freedom. The Pearson chi-squared statistic is X² = 62.1. (a) Assess overall model fit. (b) Compute the estimated dispersion parameter using both the deviance-based and Pearson-based estimators. (c) Discuss whether a quasi-Poisson or negative binomial model might be more appropriate.

PROBLEM 5 — CRITICAL THINKING

In a logistic regression with individual binary responses (nᵢ = 1 for all i), the residual deviance is often uninformative as a global goodness-of-fit measure because the χ² approximation fails. (a) Explain why this occurs from a theoretical standpoint. (b) Prove that the saturated model for binary data has log-likelihood exactly zero. (c) Discuss two alternative strategies for assessing goodness-of-fit in this setting and their relative merits.

SUMMARY

Summary

Deviance is the fundamental goodness-of-fit measure for generalized linear models, defined as D = −2[ℓ(fitted) − ℓ(saturated)]. It generalizes the residual sum of squares from ordinary least squares by measuring discrepancy in log-likelihood units rather than squared Euclidean distance. Each member of the exponential family (Normal, Poisson, Binomial, Gamma) produces a specific deviance formula derived from the general canonical-form expression. Under regularity conditions, the residual deviance follows an asymptotic χ² distribution with n − p degrees of freedom, though this approximation can be unreliable for sparse binary data.

The difference in deviances between nested models yields the likelihood ratio test statistic, providing a principled approach to variable selection and model simplification. Complementary measures include the Pearson χ² statistic (better for sparse data), AIC (for non-nested model comparison), and the Hosmer–Lemeshow test (for ungrouped binary outcomes). Deviance residuals decompose the total deviance into per-observation contributions and provide superior distributional properties for diagnostic plots. Mastery of deviance is essential for anyone working with GLMs, as it underpins model comparison, diagnostics, dispersion estimation, and the broader logic of likelihood-based statistical inference.