Deviance & Goodness-of-Fit — Deviance and goodness-of-fit concepts - Statistics Graduate Level

SECTION 1

Historical Context & Motivation

In classical linear regression, assessing model fit is relatively straightforward: residual sums of squares and the coefficient of determination R² provide intuitive summaries of how well predicted values track observed outcomes. However, as statisticians began extending regression techniques to non-normal response distributions—binary outcomes, count data, survival times—the familiar least-squares machinery no longer applied. The fundamental challenge was clear: how does one measure the discrepancy between a fitted model and observed data when the error structure is no longer Gaussian and the response is no longer continuous? The answer emerged through the unifying framework of maximum likelihood estimation and the concept of deviance, which generalizes the residual sum of squares by comparing log-likelihoods rather than squared residuals.

1922 • Fisher's Likelihood Framework

R. A. Fisher formalizes maximum likelihood estimation, providing the inferential foundation upon which deviance and likelihood ratio tests would later be built.

1938 • Neyman–Pearson Likelihood Ratio Tests

Jerzy Neyman and Egon Pearson develop the likelihood ratio test as an optimal framework for hypothesis testing, establishing the −2 log-likelihood ratio as a pivotal statistic.

1972 • Nelder & Wedderburn's GLM Unification

John Nelder and Robert Wedderburn publish their landmark paper unifying regression, ANOVA, logistic regression, and log-linear models under the generalized linear model (GLM) framework, introducing deviance as the canonical measure of goodness-of-fit.

1983 • McCullagh & Nelder's Treatise

The publication of Generalized Linear Models provides a comprehensive treatment of deviance, scaled deviance, and residual analysis, becoming the definitive reference for GLM practitioners.

1990s–Present • Information-Theoretic Extensions

AIC, BIC, and cross-validation techniques complement deviance-based assessment, enabling principled model comparison in complex settings including mixed models and Bayesian frameworks.

The central question that deviance addresses is deceptively simple: given a fitted model, how much information is lost compared to a model that perfectly reproduces the observed data? By framing goodness-of-fit as a comparison of log-likelihoods, deviance provides a single, interpretable statistic that works uniformly across all members of the exponential family—normal, binomial, Poisson, gamma, and beyond. Understanding deviance is therefore essential for anyone working with GLMs, as it underpins model diagnostics, nested model comparisons via likelihood ratio tests, and the broader logic of statistical model selection.

SECTION 2

Core Principles & Definitions

The conceptual architecture of deviance rests on a few interconnected ideas. At its core, deviance measures the distance between two models using the language of likelihood. To appreciate this, one must first understand the roles played by the saturated model, the fitted model, and the null model. These three reference points define the landscape within which goodness-of-fit is assessed.

Saturated Model

A model with as many parameters as observations, fitting the data perfectly. It achieves the maximum attainable log-likelihood ℓ(ŷ = y) and serves as the upper reference point against which all fitted models are compared.

Deviance (D)

Defined as D = −2[ℓ(fitted) − ℓ(saturated)], deviance quantifies the log-likelihood gap between the fitted model and the saturated model. Smaller deviance indicates better fit. It is always non-negative.

Null Deviance

The deviance of the intercept-only model (no predictors). Comparing null deviance to residual deviance reveals the proportion of likelihood-based 'variation' explained by the predictors.

Likelihood Ratio Test (LRT)

Nested models can be compared via the difference in deviances, which under H₀ follows a χ² distribution with degrees of freedom equal to the difference in the number of parameters.

Scaled Deviance

When the dispersion parameter ϕ is not unity (e.g., in gamma or quasi-Poisson models), the scaled deviance D* = D / ϕ is used instead, ensuring the asymptotic χ² reference distribution remains valid.

✦

KEY TAKEAWAY

Think of deviance as a generalized 'residual sum of squares' for GLMs. In ordinary least squares, the RSS measures how far predictions are from observations in squared-Euclidean distance. Deviance does the same thing but measures distance in log-likelihood units. Just as a small RSS relative to the total SS indicates a good linear fit, a small residual deviance relative to the null deviance indicates that predictors have substantially improved the model's ability to explain the data through the lens of the assumed distribution.

SECTION 3

Visual Explanation

The Deviance Landscape: Saturated, Fitted, and Null Models

The three bars represent log-likelihoods for the saturated (perfect-fit), fitted (proposed), and null (intercept-only) models. Residual deviance D is the gap between the saturated and fitted models, while null deviance D₀ spans from saturated to null. The difference ΔD = D₀ − D captures how much the predictors improve fit.

The diagram above illustrates the fundamental geometry of deviance. The saturated model sits at the top, representing the best any model from the same distributional family could achieve. It has n parameters for n observations, so it reproduces the data exactly. Below it, the fitted model uses p < n parameters, trading perfect fit for parsimony. The gap between these two log-likelihoods, multiplied by −2, gives the residual deviance. At the bottom lies the null model, which ignores all predictors. The null deviance captures the total amount of 'explainable' discrepancy, analogous to the total sum of squares in OLS. By comparing residual deviance to null deviance, one can gauge how much predictive power the covariates contribute.

SECTION 4

Mathematical Framework

The mathematical formulation of deviance is rooted in the exponential family of distributions. Recall that for a response Y with probability density or mass function belonging to the exponential dispersion family, the log-likelihood contribution of a single observation can be written in canonical form. The deviance statistic emerges naturally as twice the log-likelihood ratio between the saturated and fitted models.

EXPONENTIAL FAMILY LOG-LIKELIHOOD

f(y; θ, ϕ) = exp{ [yθ − b(θ)] / a(ϕ) + c(y, ϕ) }

Here θ is the canonical parameter, ϕ is the dispersion parameter, b(θ) is the cumulant function, and a(ϕ) and c(y, ϕ) are known functions. The mean is μ = b′(θ) and Var(Y) = b″(θ) · a(ϕ).

DEVIANCE (GENERAL FORM)

D(y; μ̂) = 2 Σᵢ [ ℓᵢ(ŷᵢ = yᵢ) − ℓᵢ(μ̂ᵢ) ] · a(ϕ)

Equivalently, D = 2 Σᵢ [ yᵢ(θ̃ᵢ − θ̂ᵢ) − b(θ̃ᵢ) + b(θ̂ᵢ) ], where θ̃ᵢ is the canonical parameter under the saturated model (setting μ̃ᵢ = yᵢ) and θ̂ᵢ is the canonical parameter under the fitted model. The factor a(ϕ) is typically absorbed when ϕ is known.

DEVIANCE FOR POISSON GLM

D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) − (yᵢ − μ̂ᵢ) ]

For Poisson data with log link, the canonical parameter is θ = ln(μ). When yᵢ = 0, the convention 0 · ln(0) = 0 is applied. The dispersion parameter ϕ = 1 for Poisson, so scaled and unscaled deviance coincide.

DEVIANCE FOR BINOMIAL GLM

D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) + (nᵢ − yᵢ) ln((nᵢ − yᵢ) / (nᵢ − μ̂ᵢ)) ]

Here yᵢ is the number of successes, nᵢ is the number of trials, and μ̂ᵢ = nᵢπ̂ᵢ where π̂ᵢ is the estimated success probability. Again ϕ = 1 for binomial, so D is directly comparable to a χ² distribution under regularity conditions.

⚠

Asymptotic Distribution

Under the null hypothesis that the fitted model is correct, the residual deviance D is asymptotically distributed as χ²(n − p), where n is the number of observations and p is the number of estimated parameters. This holds rigorously when a(ϕ) = ϕ/wᵢ with known ϕ. For binomial and Poisson models (ϕ = 1), the approximation improves with increasing group sizes or rates, but can be poor for sparse data—particularly in logistic regression with individual binary observations.

SECTION 5

Deviance Across Exponential Family Members

One of the most elegant features of the deviance framework is its universality: every member of the exponential dispersion family has a specific deviance formula derived from the general expression. Understanding these family-specific forms builds intuition for what deviance is actually measuring in each context, and reveals why deviance reduces to familiar quantities in certain special cases.

Each card shows the deviance formula for a major exponential family member. Note that the Normal deviance reduces to the residual sum of squares, confirming that ordinary least squares is a special case of the deviance framework.

Family	Canonical Link	ϕ	Deviance Unit Contribution dᵢ
Normal	Identity: g(μ) = μ	σ² (estimated)	(yᵢ − μ̂ᵢ)²
Poisson	Log: g(μ) = ln(μ)	1 (fixed)	2[yᵢ ln(yᵢ/μ̂ᵢ) − (yᵢ − μ̂ᵢ)]
Binomial	Logit: g(π) = ln[π/(1−π)]	1 (fixed)	2[yᵢ ln(yᵢ/μ̂ᵢ) + (nᵢ−yᵢ)ln((nᵢ−yᵢ)/(nᵢ−μ̂ᵢ))]
Gamma	Inverse: g(μ) = 1/μ	1/ν (estimated)	2[−ln(yᵢ/μ̂ᵢ) + (yᵢ − μ̂ᵢ)/μ̂ᵢ]

Deviance contributions by exponential family member

SECTION 6

Worked Example: Poisson Regression Deviance

Suppose a researcher models the count of insurance claims yᵢ as a function of driver age using a Poisson GLM with a log link. The model is fitted to n = 5 age groups with observed counts y = (2, 5, 8, 4, 1) and fitted values μ̂ = (2.5, 4.2, 6.8, 3.8, 1.7). We will compute the residual deviance, compare it to a χ² distribution, and interpret the result.

Computing Poisson Deviance

1
Step 1 — State the Deviance Formula

For a Poisson GLM, the deviance is D = 2 Σᵢ [ yᵢ ln(yᵢ / μ̂ᵢ) − (yᵢ − μ̂ᵢ) ]. We evaluate this sum over i = 1, …, 5 using the observed and fitted values.
2
Step 2 — Compute Each Unit Deviance dᵢ

For observation 1: d₁ = 2[2 ln(2/2.5) − (2 − 2.5)] = 2[2(−0.2231) + 0.5] = 2[−0.4463 + 0.5] = 2(0.0537) = 0.1075. For observation 2: d₂ = 2[5 ln(5/4.2) − (5 − 4.2)] = 2[5(0.1744) − 0.8] = 2[0.8720 − 0.8] = 2(0.0720) = 0.1440. For observation 3: d₃ = 2[8 ln(8/6.8) − (8 − 6.8)] = 2[8(0.1625) − 1.2] = 2[1.3000 − 1.2] = 2(0.1000) = 0.2000. For observation 4: d₄ = 2[4 ln(4/3.8) − (4 − 3.8)] = 2[4(0.0513) − 0.2] = 2[0.2053 − 0.2] = 2(0.0053) = 0.0105. For observation 5: d₅ = 2[1 ln(1/1.7) − (1 − 1.7)] = 2[1(−0.5306) + 0.7] = 2(0.1694) = 0.3388.

d = (0.1075, 0.1440, 0.2000, 0.0105, 0.3388)
3
Step 3 — Sum to Get Residual Deviance

D = Σ dᵢ = 0.1075 + 0.1440 + 0.2000 + 0.0105 + 0.3388 = 0.8008.

D = 0.801
4
Step 4 — Determine Degrees of Freedom

The model has n = 5 observations and p = 2 parameters (intercept + age coefficient). The residual degrees of freedom are n − p = 5 − 2 = 3.

df = 3
5
Step 5 — Compare to χ²(3) and Interpret

Under the null hypothesis that the Poisson model is adequate, D ~ χ²(3) approximately. The critical value at α = 0.05 is χ²₀.₉₅(3) = 7.815. Since D = 0.801 ≪ 7.815, we have no evidence of lack of fit. The p-value is P(χ²(3) > 0.801) ≈ 0.849, indicating excellent model adequacy. The ratio D/df = 0.801/3 = 0.267 is well below 1, suggesting no overdispersion.

p ≈ 0.849 — No evidence of lack of fit

SECTION 7

Deviance vs. Alternative Goodness-of-Fit Measures

Deviance is not the only goodness-of-fit statistic available for GLMs. The Pearson chi-squared statistic, Akaike Information Criterion (AIC), and Hosmer–Lemeshow test all serve related but distinct purposes. Understanding when each measure is appropriate prevents misdiagnosis of model problems.

Criterion	Strengths	Limitations
Deviance	Directly linked to likelihood ratio tests; additive over nested models; canonical measure in GLM theory; provides individual deviance residuals for diagnostics.	χ² approximation can be poor for sparse binary data; not meaningful for comparing non-nested models; requires known ϕ for exact test.
Pearson χ²	More robust to sparse data than deviance; familiar quadratic form; useful for estimating dispersion via X²/(n−p).	Not additive for nested model comparisons; less directly tied to likelihood theory; same asymptotic distribution as deviance under regularity conditions.
AIC	Allows comparison of non-nested models; penalizes complexity; grounded in information theory (Kullback–Leibler divergence).	No formal hypothesis test; sensitive to sample size; requires models fit to the same data.
Hosmer–Lemeshow	Specifically designed for logistic regression with binary outcomes; circumvents sparse-data problem by grouping observations.	Arbitrary choice of number of groups; low power; test result depends on grouping strategy; not generalizable beyond binomial models.

Comparison of major goodness-of-fit measures for GLMs

✦

KEY TAKEAWAY

Think of deviance as the 'official language' of the GLM framework—it is the native measure that comes directly from the likelihood, just as RSS comes naturally from least squares. Alternative measures like Pearson χ² and AIC are complementary tools for specific situations. When comparing nested models, the difference in deviances is the likelihood ratio test statistic. When comparing non-nested models, switch to AIC or BIC. When checking overall model adequacy for sparse binary data, prefer the Hosmer–Lemeshow test or Pearson χ².

SECTION 8

Connections to Advanced Theory

Deviance sits at a crossroads between classical hypothesis testing and modern model selection theory. Its connections extend in several important directions: toward information criteria, quasi-likelihood methods, Bayesian model comparison, and the analysis of deviance (ANODEV) for factor effects in GLMs.

Core Concept	Advanced Extension	Key Relationship
Deviance D	AIC = D + 2p	AIC is deviance plus a complexity penalty; minimizing AIC trades fit against parsimony via an asymptotic estimate of out-of-sample prediction loss.
Scaled deviance D/ϕ	Quasi-likelihood	When the full distribution is unknown, quasi-likelihood methods estimate ϕ from the data and use scaled deviance for inference, relaxing distributional assumptions.
Deviance residuals rᵢᴰ	Residual diagnostics	Deviance residuals rᵢᴰ = sign(yᵢ − μ̂ᵢ)√dᵢ are more symmetrically distributed than Pearson residuals, making QQ-plots and influence diagnostics more reliable.
ΔD for nested models	Analysis of Deviance (ANODEV)	Sequential deviance reduction by adding terms mirrors sequential ANOVA. Type I and Type III deviance tables decompose the contribution of each predictor or factor.
−2 log-likelihood	DIC (Bayesian)	The Deviance Information Criterion extends deviance to Bayesian hierarchical models by replacing p with the effective number of parameters pD estimated from the posterior.

From deviance fundamentals to advanced extensions

A particularly important extension is the concept of deviance residuals. Defined as rᵢᴰ = sign(yᵢ − μ̂ᵢ) × √dᵢ, where dᵢ is the unit deviance for observation i, these residuals decompose the total deviance into per-observation contributions. Unlike raw or Pearson residuals, deviance residuals tend to be more normally distributed, particularly for Poisson and binomial models. This property makes them invaluable for diagnostic plots: a QQ-plot of deviance residuals against normal quantiles should approximate a straight line when the model is correctly specified. Systematic departures indicate misspecified link functions, omitted predictors, or distributional inadequacy.

ℹ

Looking Ahead

In mixed-effects GLMs (GLMMs), the concept of deviance extends but becomes more nuanced. The marginal likelihood integrates over random effects, making standard deviance-based tests inapplicable in their classical form. Restricted maximum likelihood (REML) deviance is used for variance component estimation, while conditional AIC and parametric bootstrap tests replace simple χ² comparisons for testing random effects. Understanding classical deviance thoroughly provides the conceptual foundation for navigating these more complex settings.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain why the deviance of a normal linear model with identity link and known variance σ² reduces to RSS/σ². What does this imply about the relationship between deviance-based inference and classical F-tests in ordinary least squares regression?

Show answer

For the normal distribution, the saturated model sets μ̃ᵢ = yᵢ, so the saturated log-likelihood involves terms Σ(yᵢ − yᵢ)² = 0. The fitted model contributes Σ(yᵢ − μ̂ᵢ)² to the log-likelihood difference. Applying D = 2[ℓ(sat) − ℓ(fit)] yields D = Σ(yᵢ − μ̂ᵢ)²/σ² = RSS/σ². This confirms that ordinary least squares is a special case of GLM deviance analysis. The classical F-test for comparing nested linear models is equivalent to the scaled deviance difference test when the dispersion parameter σ² is estimated from the data—the F-distribution arises from using an estimated ϕ in place of the known ϕ used in the χ² reference.

Question 2: BASIC CALCULATION

A Poisson GLM is fitted to n = 4 observations with observed counts y = (3, 7, 2, 6) and fitted values μ̂ = (3.5, 6.0, 2.8, 5.7). Compute the residual deviance D. Using p = 2 parameters, determine the residual degrees of freedom and assess model fit at α = 0.05.

Show answer

We compute each unit deviance: d₁ = 2[3 ln(3/3.5) − (3−3.5)] = 2[3(−0.1542) + 0.5] = 2(0.0374) = 0.0748. d₂ = 2[7 ln(7/6) − (7−6)] = 2[7(0.1542) − 1] = 2(0.0794) = 0.1588. d₃ = 2[2 ln(2/2.8) − (2−2.8)] = 2[2(−0.3365) + 0.8] = 2(0.1271) = 0.2541. d₄ = 2[6 ln(6/5.7) − (6−5.7)] = 2[6(0.0513) − 0.3] = 2(0.0078) = 0.0155. Total D = 0.0748 + 0.1588 + 0.2541 + 0.0155 = 0.5032. With df = n − p = 4 − 2 = 2, we compare to χ²₀.₉₅(2) = 5.991. Since 0.503 ≪ 5.991, there is no evidence of lack of fit, and the model appears adequate.

Question 3: INTERMEDIATE

A researcher fits two nested logistic regression models to the same dataset (n = 200). Model A (three predictors) has deviance

D_A

= 185.3 with

p_A

= 4 parameters. Model B (one additional predictor) has deviance

D_B

= 178.6 with

p_B

= 5 parameters. Perform a likelihood ratio test at α = 0.05 to determine whether the additional predictor significantly improves fit. Also compute the approximate pseudo-R² based on deviance for Model B, given that the null deviance is D₀ = 246.8.

Show answer

The likelihood ratio test statistic is ΔD =

D_A

−

D_B

= 185.3 − 178.6 = 6.7, with Δdf =

p_B

−

p_A

= 5 − 4 = 1. Under H₀, ΔD ~ χ²(1). The critical value is χ²₀.₉₅(1) = 3.841, and the p-value is P(χ²(1) > 6.7) ≈ 0.0096. Since 6.7 > 3.841 (equivalently, p ≈ 0.010 < 0.05), we reject H₀ and conclude that the additional predictor provides a statistically significant improvement in fit. For the pseudo-R², we compute R²_D = 1 −

D_B

/D₀ = 1 − 178.6/246.8 = 1 − 0.7236 = 0.2764, indicating that the predictors in Model B account for roughly 27.6% of the null deviance, analogous to the proportion of variance explained in OLS.

Question 4: APPLIED

An epidemiologist models the number of hospital infections per ward using a Poisson GLM with a log link. The fitted model yields a residual deviance of 58.4 on 30 degrees of freedom. The Pearson chi-squared statistic is X² = 62.1. (a) Assess overall model fit. (b) Compute the estimated dispersion parameter using both the deviance-based and Pearson-based estimators. (c) Discuss whether a quasi-Poisson or negative binomial model might be more appropriate.

Show answer

(a) For a Poisson model, we expect D ≈ df if the model is adequate. Here D/df = 58.4/30 = 1.947. The p-value for χ²(30) exceeding 58.4 is approximately 0.0014, suggesting statistically significant lack of fit. (b) The deviance-based dispersion estimate is ϕ̂_D = D/(n−p) = 58.4/30 = 1.947, and the Pearson-based estimate is ϕ̂_P = X²/(n−p) = 62.1/30 = 2.070. Both exceed 1, confirming overdispersion. (c) Since the estimated dispersion is roughly 2, a quasi-Poisson model—which scales standard errors by √ϕ̂—would be appropriate to correct inference without changing point estimates. Alternatively, a negative binomial GLM explicitly models extra-Poisson variation through a dispersion parameter and would be preferable if the overdispersion is substantial or if likelihood-based model comparison is needed.

Question 5: CRITICAL THINKING

In a logistic regression with individual binary responses (nᵢ = 1 for all i), the residual deviance is often uninformative as a global goodness-of-fit measure because the χ² approximation fails. (a) Explain why this occurs from a theoretical standpoint. (b) Prove that the saturated model for binary data has log-likelihood exactly zero. (c) Discuss two alternative strategies for assessing goodness-of-fit in this setting and their relative merits.

Show answer

(a) The χ² approximation for deviance requires that each cell (combination of covariate values) contributes sufficient information, which in practice means nᵢ should be reasonably large. For ungrouped binary data, each observation forms its own cell with nᵢ = 1, yielding dᵢ values that can only take one of two forms depending on yᵢ ∈ {0,1}. The central limit theorem does not apply cell-by-cell, so the sum of these contributions does not converge to χ². (b) For yᵢ = 1: the saturated model sets π̃ᵢ = 1, giving ℓᵢ = ln(1) = 0. For yᵢ = 0: π̃ᵢ = 0, giving ℓᵢ = ln(1) = 0. Hence ℓ(saturated) = 0 and D = −2ℓ(fitted). (c) The Hosmer–Lemeshow test groups observations by predicted probability deciles and compares observed to expected counts within each group using a Pearson-type statistic, though it suffers from arbitrary grouping and low power. A more modern approach uses predictive checks or calibration curves: plot observed event rates against predicted probabilities in sliding windows and assess calibration visually. Le Cessie–van Houwelingen's omnibus test provides a formal alternative with better power. Cross-validated log-loss (negative Bernoulli deviance) offers a prediction-oriented assessment that avoids the χ² approximation entirely.

SUMMARY

Summary

Deviance is the fundamental goodness-of-fit measure for generalized linear models, defined as D = −2[ℓ(fitted) − ℓ(saturated)]. It generalizes the residual sum of squares from ordinary least squares by measuring discrepancy in log-likelihood units rather than squared Euclidean distance. Each member of the exponential family (Normal, Poisson, Binomial, Gamma) produces a specific deviance formula derived from the general canonical-form expression. Under regularity conditions, the residual deviance follows an asymptotic χ² distribution with n − p degrees of freedom, though this approximation can be unreliable for sparse binary data.

The difference in deviances between nested models yields the likelihood ratio test statistic, providing a principled approach to variable selection and model simplification. Complementary measures include the Pearson χ² statistic (better for sparse data), AIC (for non-nested model comparison), and the Hosmer–Lemeshow test (for ungrouped binary outcomes). Deviance residuals decompose the total deviance into per-observation contributions and provide superior distributional properties for diagnostic plots. Mastery of deviance is essential for anyone working with GLMs, as it underpins model comparison, diagnostics, dispersion estimation, and the broader logic of likelihood-based statistical inference.