Analyzing Departures from Linearity

SECTION 1

Historical Context & Motivation

The idea of fitting a straight line through data goes back to the early nineteenth century, when least-squares regression was independently developed by Adrien-Marie Legendre and Carl Friedrich Gauss to predict the orbits of celestial bodies. For over a century, practitioners applied linear models almost reflexively, rarely pausing to verify whether the assumed straight-line form was actually appropriate. It was only as data sets grew larger and computational tools became available that statisticians recognized the need for systematic diagnostics that could reveal when a linear model was misleading rather than informative.

1805

Legendre's Least Squares

Adrien-Marie Legendre publishes the method of least squares, providing the first formal algorithm for fitting a linear model to bivariate data.

1885

Galton's Regression to the Mean

Francis Galton introduces 'regression toward mediocrity' in his study of heights, establishing the concept of the regression line and spurring interest in how well linear models capture real relationships.

1973

Anscombe's Quartet

Frank Anscombe constructs four data sets sharing identical regression statistics (slope, intercept, r²) yet displaying wildly different scatter patterns—dramatically demonstrating that numerical summaries alone cannot confirm linearity.

1977

Tukey's EDA Movement

John Tukey's Exploratory Data Analysis formalizes the principle of 'looking at the data first,' championing residual plots and other graphical diagnostics as standard practice.

The central question this lesson addresses is deceptively simple: after you fit a line, how do you know whether the line is actually a good description of the relationship? As Anscombe's Quartet powerfully illustrates, summary statistics like r and r² can mask severe departures from linearity, making graphical residual analysis an indispensable skill for any practicing statistician.

SECTION 2

Core Principles & Definitions

Before diagnosing departures from linearity, you must be fluent in several foundational concepts that underpin residual analysis. A residual is defined as the difference between an observed response value and the value predicted by the regression line: e = y − ŷ. Residuals are the diagnostic currency of regression—they encode every piece of information the model failed to capture. When the linear model is appropriate, residuals should appear structureless; any visible pattern suggests the model is inadequate.

Residual Plot

A scatterplot of residuals (e) on the vertical axis against the explanatory variable (x) or fitted values (ŷ) on the horizontal axis. A well-fitting linear model yields a residual plot with no discernible pattern—only random scatter centered on zero.

Curvature (Nonlinearity)

When the residual plot shows a systematic curved pattern—typically a U-shape or inverted U—the true relationship between x and y is nonlinear, and the straight-line model systematically over- and under-predicts in different regions.

Non-Constant Spread (Heteroscedasticity)

If the vertical spread of residuals fans out (or funnels in) as x increases, the variability of the response is not constant across the range of x, violating a key regression assumption and distorting inference.

Influential Points & Outliers

Points with unusually large residuals (outliers) or extreme x-values (high leverage) can disproportionately shift the regression line. Identifying them is critical before drawing conclusions from any fitted model.

Coefficient of Determination (r²)

r² measures the proportion of variability in y explained by the linear model. A high r² is necessary but not sufficient for linearity—a curved relationship can still produce a high r² while being poorly modeled by a line.

✦ KEY TAKEAWAY

Think of fitting a regression line like using a ruler to trace the path of a winding river on a map. The ruler captures the general direction, but the bends and meanders are lost. Residuals measure precisely how far the river departs from the ruler at every point. If the residuals show a pattern of systematic departures rather than random wobble, the ruler is the wrong tool—you need a flexible curve.

SECTION 3

Residual Plots — The Visual Diagnostic

The single most powerful tool for detecting departures from linearity is the residual plot. The diagram below contrasts three scenarios: a residual plot consistent with a good linear fit, one showing systematic curvature, and one exhibiting non-constant spread. Study the shapes carefully—recognizing these patterns at a glance is essential on the AP exam.

Left: Residuals scatter randomly about the zero line with roughly constant spread—evidence that a linear model is appropriate. Center: A clear U-shaped (or inverted-U) pattern indicates curvature in the original data; the linear model systematically overestimates in the middle and underestimates at the extremes. Right: A fan-shaped spread signals heteroscedasticity, where variability in y increases with x.

When reading a residual plot, focus on two questions. First, is there a systematic shape (curve, wave, or cluster) rather than formless scatter? If so, the linear model is missing a structural feature of the data. Second, does the vertical spread remain roughly the same from left to right? Increasing or decreasing spread (the 'fan' or 'funnel' pattern) indicates that precision of prediction varies across the range of x, which undermines standard inference procedures that assume constant variance.

SECTION 4

Mathematical Framework

The least-squares regression line minimizes the sum of squared residuals. Understanding the formal definitions clarifies exactly what residuals measure and why their behavior reveals model inadequacy.

LEAST-SQUARES REGRESSION LINE

ŷ = b₀ + b₁x

where ŷ is the predicted response, b₀ is the y-intercept, and b₁ is the slope. The slope b₁ = r × (s_y / s_x), and the line always passes through (x̄, ȳ).

RESIDUAL DEFINITION

eᵢ = yᵢ − ŷᵢ

A positive residual means the observed value lies above the regression line (model underestimates), while a negative residual means the observed value lies below (model overestimates). The sum of all residuals for an LSRL equals zero: Σeᵢ = 0.

COEFFICIENT OF DETERMINATION

r² = 1 − (SSE / SST)

SSE = Σ(yᵢ − ŷᵢ)² is the sum of squared residuals. SST = Σ(yᵢ − ȳ)² is the total sum of squares. r² gives the proportion of variation in y explained by the linear model. A high r² does not guarantee linearity—it only measures how much spread is reduced by the model.

STANDARD DEVIATION OF RESIDUALS

s = √[Σeᵢ² / (n − 2)]

Also called the residual standard error, s estimates the typical size of a prediction error. A smaller s indicates predictions are, on average, closer to observed values. We divide by n − 2 because fitting a line consumes two degrees of freedom (one for the slope, one for the intercept).

📝 AP Exam Tip

On the AP Statistics exam, you will frequently be asked to interpret a residual plot and state whether a linear model is appropriate. Always reference the specific pattern you observe—'there is a curved pattern in the residual plot, indicating the relationship is not linear'—rather than simply saying 'the residual plot looks bad.'

SECTION 5

Detailed Diagnostic Breakdown

Beyond recognizing gross patterns, effective residual analysis requires distinguishing among several specific types of departures. The diagram below illustrates additional diagnostic scenarios, including the effect of outliers and influential observations. The accompanying classification table provides a quick-reference guide for interpreting what you see.

The left panel shows a scatterplot where points follow a curve (green) but a straight line (red dashed LSRL) has been fitted. The data might appear 'roughly linear' at first glance, especially with a moderate r². The right panel plots the residuals from that LSRL—the U-shaped pattern unmistakably reveals the curvature that was partially hidden in the original plot.

Residual plot pattern interpretation guide

Residual Plot Pattern	Diagnosis	Action / Remedy
Random scatter, roughly constant spread around e = 0	Linear model is appropriate	Proceed with inference; report slope, r², and s
Curved (U-shape or inverted-U)	Nonlinear relationship; line misses the true pattern	Try a transformation (e.g., log, square root, power) or fit a polynomial/exponential model
Fan or funnel shape (spread changes with x)	Heteroscedasticity (non-constant variance)	Consider a log transformation on y or use weighted regression
One or two points far from the rest	Outlier (large residual) or influential point (high leverage)	Investigate cause; refit without the point and compare results
Clusters or grouping in residuals	Possible lurking variable creating subgroups	Stratify by the grouping variable and fit separate models, or add the variable to a multiple regression model

SECTION 6

Worked Example

A biologist measures the age (in years) and trunk diameter (in centimeters) of 10 oak trees. After fitting a least-squares regression line (ŷ = 2.4 + 1.8x), the computer output gives r² = 0.91 and the following residuals. The goal is to determine whether a linear model is appropriate.

Oak tree age-diameter data with residuals from ŷ = 9 + 1.26x

Age (x)	Diameter (y)	Predicted (ŷ)	Residual (e)
5	14.0	15.3	−1.3
10	22.5	21.6	+0.9
15	28.0	27.9	+0.1
20	35.0	34.2	+0.8
25	40.0	40.5	−0.5
30	48.0	46.9	+1.1
35	52.0	53.2	−1.2

Analyzing the Oak Tree Residuals

Step 1 — Examine r²The computer reports r² = 0.91, which means the linear model explains 91% of the variability in trunk diameter. This is numerically impressive, but as Anscombe's Quartet warns us, a high r² alone does not confirm linearity. We must check the residual plot.

Step 2 — Construct the Residual Plot (mentally or on calculator)Plot the residuals (e) against x (age). At x = 5 and x = 10, residuals are positive (+2.6 and +2.1). From x = 15 through x = 35, residuals are negative and become increasingly so: −1.4, −3.4, −7.4, −8.4, −13.4. This means the model overestimates diameter for older trees and underestimates for younger trees—a clear one-sided pattern.

Step 3 — Identify the PatternThe residuals transition from positive to increasingly negative in a systematic way. This is not the random scatter we expect from a well-fitting linear model. Instead, the pattern is consistent with a concave-down curve: tree diameter growth rate slows with age (diminishing returns), and a straight line fails to capture this deceleration.

Curved pattern detected → departure from linearity

Step 4 — State the ConclusionDespite r² = 0.91, the residual plot reveals a clear curved pattern indicating that the relationship between tree age and trunk diameter is nonlinear. A linear model is not appropriate for these data. A transformation—such as using the square root or natural logarithm of diameter—should be explored to linearize the relationship before fitting a regression model.

A linear model is not appropriate; consider a transformation.

SECTION 7

Strengths & Limitations of Residual Analysis

Strengths and limitations of residual analysis for detecting departures from linearity

Strengths	Limitations
Residual plots can reveal curvature, non-constant spread, and outliers that summary statistics like r² completely miss.	With small sample sizes, random variation can mimic patterns, making it difficult to distinguish real departures from noise.
The technique requires no additional data collection—residuals are automatically produced by any regression software or graphing calculator.	Residual analysis is inherently subjective; two analysts may disagree about whether a pattern is 'systematic enough' to matter.
Residual plots are intuitive and communicate results visually, making them effective for both analysis and presentation.	A clean residual plot does not prove the model is correct—it only fails to reveal a problem. There may be lurking variables or issues not captured by a single predictor.
Identifying influential points via large residuals can flag data entry errors or unusual observations that deserve further investigation.	High-leverage points may not have large residuals precisely because they pull the line toward themselves, masking their own influence.

✦ KEY TAKEAWAY

Residual analysis is like a quality-control inspection on an assembly line. Passing inspection (no pattern in residuals) does not guarantee a flawless product, but failing inspection (clear pattern) provides an unambiguous signal that something is wrong with the manufacturing process. In statistics, 'something is wrong' means the linear model is not capturing the true relationship, and you should investigate further before reporting results.

SECTION 8

Connecting to Transformations & Advanced Models

When residual analysis reveals a departure from linearity, the natural next step is to transform the data so that the relationship between the transformed variables is approximately linear. This strategy is central to the AP Statistics curriculum and connects residual diagnostics to a broader toolkit of modeling techniques. The two most common transformations—logarithmic and power—address the two most common types of curved relationships: exponential growth/decay and power functions.

Comparing the linear model with common transformations used to address departures from linearity

Feature	Linear Model (no transform)	Log Transformation	Power Transformation
Form of equation	ŷ = b₀ + b₁x	log(ŷ) = a + bx → ŷ = 10ᵃ × 10ᵇˣ	log(ŷ) = a + b·log(x) → ŷ = 10ᵃ × xᵇ
Typical pattern in original residual plot	Random scatter (model is fine)	Exponential curve; residuals accelerate	Concave-down curve; residuals level off
What to plot for linearity check	y vs. x	log(y) vs. x	log(y) vs. log(x)
When to try it	Residual plot shows no pattern	Data show multiplicative (percentage) growth	Data follow a physical/biological scaling law

In more advanced statistics courses, you will encounter polynomial regression, spline smoothing, and generalized additive models that handle nonlinearity without requiring you to choose a specific transformation. However, at the AP level, the critical skill is recognizing when the linear model fails via the residual plot and knowing that a logarithmic or power transformation may restore linearity. In the free-response section, clearly stating the evidence (the pattern in the residual plot) and the remedy (a specific transformation) demonstrates the kind of structured reasoning that earns full credit.

SECTION 9

Practice Problems

A student fits a least-squares regression line to a bivariate data set and obtains r² = 0.95. The student claims that because r² is very high, a linear model is definitely appropriate. Which of the following best explains why this reasoning is flawed?

A regression analysis produces the equation ŷ = 12.3 + 4.7x. For one observation, x = 6 and y = 43.0. What is the residual for this observation, and does the model overestimate or underestimate the response?

A researcher fits a linear model to data relating hours of sunlight (x) to daily energy output (kilowatt-hours) for a solar panel array. The residual plot shows residuals that are positive for moderate sunlight values but increasingly negative for both very low and very high sunlight values. Which of the following best describes the likely departure from linearity?

PROBLEM 4 — APPLIED

A city planner collects data on 15 neighborhoods relating population density (people per square mile) to average commute time (minutes). After fitting a linear regression, the planner reports: ŷ = 18.2 + 0.0012x, r² = 0.84, and s = 4.3 minutes. The residual plot reveals a clear curved pattern: residuals are positive for low and high densities, and negative for moderate densities. (a) Explain what the curved pattern in the residual plot tells us about the relationship between population density and commute time. (b) A colleague argues that because r² = 0.84, the linear model is 'good enough.' Provide a statistically sound rebuttal. (c) Suggest a specific transformation that might linearize the relationship and explain how you would verify that the transformation worked. (d) Interpret s = 4.3 in context.

PROBLEM 5 — CRITICAL THINKING

Two students each collect data on the relationship between study hours (x) and exam score (y) for different classes. Both students report r = 0.92 and fit an LSRL. Student A's residual plot shows random scatter with constant spread. Student B's residual plot shows a clear U-shaped pattern, with residuals positive at low and high study hours and negative in the middle. (a) Despite identical correlation coefficients, explain in detail why the two linear models differ fundamentally in their appropriateness. (b) For Student B, sketch what the original scatterplot likely looks like and describe how the true relationship differs from the fitted line. (c) Student B decides to remove the three data points with the largest residuals to 'improve' the model. Explain why this is poor statistical practice and suggest a better alternative. (d) Discuss how reporting only r (or r²) without a residual plot can lead to misleading conclusions in a research context. Reference the concept behind Anscombe's Quartet in your response.

SUMMARY

Lesson Summary

Analyzing departures from linearity begins with fitting a least-squares regression line and computing residuals (e = y − ŷ). The residual plot—residuals plotted against x or ŷ—is the primary diagnostic tool. A good linear fit produces random scatter with constant spread around zero. A curved pattern (U-shape or inverted-U) signals nonlinearity, a fan or funnel shape signals non-constant variance (heteroscedasticity), and isolated extreme points may indicate outliers or influential observations.

Critically, a high r² does not guarantee linearity—as Anscombe's Quartet demonstrated, dramatically different data configurations can share identical numerical summaries. When departures are detected, transformations (logarithmic or power) can often re-express the data into a linear form. Always verify a transformation's success by checking that the new residual plot shows the desired random scatter. On the AP exam, clearly describe the pattern you observe, state whether the linear model is appropriate, and if it is not, recommend a specific next step.