01 / 21

Understanding
Residuals

Learn how residuals measure the gap between observed data and predicted values, and why they are essential for evaluating regression models.

AP Statistics

What Is a Residual?

When we fit a line (or curve) to data, each point rarely falls exactly on the line. The vertical distance between an observed data point and the value predicted by the model is called a residual.

The Residual Formula

The residual for the \(i\)-th observation is:

\( y_i \)
The observed (actual) value of the response variable
\( \hat{y}_i \)
The predicted value from the regression line
\( e_i \)
The residual: positive if above the line, negative if below

Visualizing Residuals on a Scatterplot

x y
\( \hat{y} \)
\( e > 0 \)
\( e < 0 \)
Observed Predicted Residual (neg) Residual (pos)

Positive vs. Negative Residuals

Positive Residual
\( y_i > \hat{y}_i \): the model underestimated the actual value. The point sits above the regression line.
Negative Residual
\( y_i < \hat{y}_i \): the model overestimated the actual value. The point sits below the regression line.

Residuals and Least Squares

The least-squares regression line (LSRL) is chosen specifically to minimize the sum of squared residuals:

By squaring the residuals before summing, the LSRL penalizes large errors heavily. This is why the method is called "least squares": it finds the line that makes the total squared error as small as possible.

Key Properties of Residuals

1
Sum to zero: For the LSRL, \( \sum e_i = 0 \). Positive and negative residuals cancel out exactly.
2
Mean is zero: Since they sum to zero, the average residual is always \( \bar{e} = 0 \).
3
Uncorrelated with \( x \): In a well-fit linear model, residuals show no pattern when plotted against \( x \).
4
Standard deviation of residuals (\( s \)): \( s = \sqrt{\dfrac{\sum e_i^2}{n - 2}} \), a measure of typical prediction error.

The Residual Plot

A residual plot graphs residuals (\( e \)) on the vertical axis against the explanatory variable (\( x \)) or predicted values (\( \hat{y} \)) on the horizontal axis. It is the primary tool for judging whether a linear model is appropriate.

x (or \( \hat{y} \))
Residual 0 Good: Random scatter around 0

Problematic Residual Plots

Curved Pattern
A U-shape or arch suggests the relationship is nonlinear. A straight line is not the right model.
Fanning Out
Spread increases as \( x \) grows, indicating non-constant variance (heteroscedasticity).
Curved Pattern Fanning Out

Residual vs. Error Term

Residual \( e_i \) Error \( \varepsilon_i \)
Definition \( y_i - \hat{y}_i \) \( y_i - (\beta_0 + \beta_1 x_i) \)
Based on Sample regression line True population line
Observable? Yes: computed from data No: theoretical quantity
Sum Always \( 0 \) (for LSRL) Expected value \( 0 \)
Used for Checking model fit Defining the model

How to Calculate a Residual

1
Identify the regression equation: \( \hat{y} = a + bx \) (given or computed).
2
Plug in the \( x \)-value for the observation to get the predicted value \( \hat{y} \).
3
Subtract: \( e = y_{\text{observed}} - \hat{y}_{\text{predicted}} \).
4
Interpret the sign: positive means the model underestimated; negative means it overestimated.

Example 1: Computing a Residual

A study predicts test scores from hours studied. The LSRL is: $$ \hat{y} = 52 + 4.8x $$ A student studied 10 hours and scored 95.

1
Predicted: \( \hat{y} = 52 + 4.8(10) = 52 + 48 = \) 100
2
Residual: \( e = 95 - 100 = \) \(-5\)
3
Interpretation: The model overestimated by 5 points. The student scored 5 points below the predicted value.

Example 2: Interpreting a Residual Plot

Below is a residual plot from a regression of plant height on sunlight hours. What does it tell us?

Sunlight (hrs) Residual

Example 3: Using Residuals to Find Observed Values

A regression equation is \( \hat{y} = 20 + 3x \). At \( x = 8 \), the residual is \( +5 \). What was the actual observed value?

1
Predicted: \( \hat{y} = 20 + 3(8) = 20 + 24 = \) 44
2
Rearrange the formula: \( y = e + \hat{y} = 5 + 44 = \) 49
3
Interpretation: The actual observation was 49, which is 5 above the predicted value of 44.

Common Mistakes

✕ Reversed Subtraction
Computing \( \hat{y} - y \) instead of \( y - \hat{y} \). This flips the sign and reverses interpretation.
✕ Confusing Sign Meaning
Saying a positive residual means "overestimated." Positive means the model underestimated: the real value was higher.
✕ Ignoring Residual Plots
Relying only on \( r^2 \) without checking the residual plot. A high \( r^2 \) can still hide a nonlinear relationship.

Summary: Residual Checklist

Concept Key Fact
Formula \( e_i = y_i - \hat{y}_i \)
Positive residual Model underestimates; point above line
Negative residual Model overestimates; point below line
Sum of residuals \( \sum e_i = 0 \) for LSRL
Good residual plot Random scatter, no pattern
Bad residual plot Curve or fan shape: model is inappropriate
\( s \) (std. dev. of residuals) Typical size of prediction error

Question 1 of 5

A regression line predicts \( \hat{y} = 30 + 2x \). At \( x = 10 \), the observed value is \( y = 55 \). What is the residual?

Question 2 of 5

A residual of \( e = -8 \) means the regression model:

Question 3 of 5

Which of the following is always true for the least-squares regression line?

Question 4 of 5

A residual plot shows a clear U-shaped pattern. What does this indicate?

Question 5 of 5

The regression equation is \( \hat{y} = 15 + 5x \). At \( x = 6 \), the residual is \( e = -3 \). What is the observed value of \( y \)?