01 / 20

Least Squares
Regression

Master the technique of fitting the best line through data by minimizing the sum of squared residuals.

AP Statistics

What is Regression?

Regression is a statistical method for modeling the relationship between two quantitative variables. Given an explanatory variable \(x\) and a response variable \(y\), regression finds a function that best predicts \(y\) from \(x\).

Scatterplots and Linear Patterns

x (explanatory) y (response) LSRL

Before fitting a line, always create a scatterplot. Check for a linear pattern, the direction (positive or negative), and any outliers.

Key Vocabulary

Residual
The difference between the observed and predicted value: \(e_i = y_i - \hat{y}_i\)
Slope \(b\)
The predicted change in \(y\) for each one-unit increase in \(x\)
Intercept \(a\)
The predicted value of \(y\) when \(x = 0\)

The LSRL Formulas

The least squares regression line is: $$ \hat{y} = a + bx $$

Slope
$$ b = r \cdot \frac{s_y}{s_x} $$
Intercept
$$ a = \bar{y} - b\bar{x} $$

Minimizing Squared Residuals

LSRL squares of residuals

The LSRL minimizes: $$ \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} e_i^2 $$ Each red square represents a squared residual. The green line is positioned to make the total area of all squares as small as possible.

Correlation vs Regression

Correlation \(r\) Regression
Purpose Measures strength and direction of linear association Predicts \(y\) from \(x\)
Output A single number: \(-1 \leq r \leq 1\) An equation: \(\hat{y} = a + bx\)
Roles of \(x, y\) Symmetric: switching \(x\) and \(y\) gives same \(r\) Not symmetric: switching changes the line
Units Unitless Slope has units \(y/x\)

The Power of \(r^2\)

The coefficient of determination \(r^2\) tells you the proportion of variation in \(y\) that is explained by the regression on \(x\).

\(r^2 = 0.85\)
85% of the variation in the response variable is explained by the linear model.
\(1 - r^2 = 0.15\)
15% of the variation remains unexplained (captured by residuals).

Reading Residual Plots

Good: random scatter x 0 Bad: curved pattern x 0

A random scatter around the zero line confirms the linear model is appropriate. A curved pattern means a linear model is not a good fit: consider a transformation.

Influential Points and Outliers

Outlier
A point with a large residual: it lies far from the regression line vertically.
High Leverage
A point with an \(x\)-value far from \(\bar{x}\). It could pull the line strongly.
Influential
A point that, when removed, substantially changes the slope, intercept, or \(r^2\).

Step-by-Step Procedure

1
Plot the data. Make a scatterplot of \(y\) versus \(x\) and confirm a linear pattern.
2
Compute summary statistics. Find \(\bar{x}\), \(\bar{y}\), \(s_x\), \(s_y\), and \(r\).
3
Calculate slope and intercept. Use \(b = r \cdot \frac{s_y}{s_x}\) and \(a = \bar{y} - b\bar{x}\).
4
Write the equation. State \(\hat{y} = a + bx\) with context (variable names, units).
5
Check residuals. Plot residuals vs. \(x\). Look for random scatter around zero.
6
Report \(r^2\). Interpret the proportion of variation explained in context.

Example 1: Study Hours

A teacher records the hours studied (\(x\)) and exam score (\(y\)) for 6 students. Summary statistics: \(\bar{x} = 4\), \(\bar{y} = 76\), \(s_x = 1.8\), \(s_y = 10.2\), \(r = 0.92\).

1
Slope: \(b = 0.92 \times \frac{10.2}{1.8} = 0.92 \times 5.667 \approx \)5.21
2
Intercept: \(a = 76 - 5.21(4) = 76 - 20.84 \approx \)55.16
3
Equation: \(\hat{y} = 55.16 + 5.21x\)

Example 2: Making a Prediction

Using the equation from Example 1, \(\hat{y} = 55.16 + 5.21x\), predict the exam score for a student who studies 5.5 hours.

1
Substitute: \(\hat{y} = 55.16 + 5.21(5.5)\)
2
Calculate: \(\hat{y} = 55.16 + 28.655 = \)83.82

Example 3: Computing a Residual

Suppose a student who studied \(x = 3\) hours actually scored \(y = 72\). Using \(\hat{y} = 55.16 + 5.21x\):

1
Predicted: \(\hat{y} = 55.16 + 5.21(3) = 55.16 + 15.63 = \)70.79
2
Residual: \(e = y - \hat{y} = 72 - 70.79 = \)+1.21

Common Mistakes

✕ Extrapolation
Using the LSRL to predict \(y\) for \(x\)-values far outside the observed data range. The linear pattern may not hold there.
✕ Confusing \(r\) and \(r^2\)
\(r = 0.70\) does not mean 70% explained. The correct value is \(r^2 = 0.49\), so only 49% of variation is explained.
✕ Claiming Causation
Regression shows an association. Without a controlled experiment, you cannot conclude that \(x\) causes changes in \(y\).

Summary Reference Table

Concept Formula / Key Idea Interpretation
LSRL Equation \(\hat{y} = a + bx\) Predicted value of \(y\) for a given \(x\)
Slope \(b\) \(b = r \cdot \frac{s_y}{s_x}\) Change in \(\hat{y}\) per unit change in \(x\)
Intercept \(a\) \(a = \bar{y} - b\bar{x}\) Predicted \(y\) when \(x = 0\)
Residual \(e_i = y_i - \hat{y}_i\) Positive: above line; Negative: below line
\(r^2\) Square of correlation % of variation in \(y\) explained by \(x\)

Question 1 of 5

A least squares regression line has the equation \(\hat{y} = 12.4 + 3.2x\). What is the predicted value of \(y\) when \(x = 5\)?

Question 2 of 5

If the correlation between \(x\) and \(y\) is \(r = 0.80\), what is the coefficient of determination \(r^2\)?

Question 3 of 5

The LSRL for predicting weight (\(y\), in kg) from height (\(x\), in cm) is \(\hat{y} = -105 + 1.1x\). A person is 170 cm tall and weighs 85 kg. What is the residual?

Question 4 of 5

A residual plot shows a clear U-shaped pattern. What does this indicate about the linear model?

Question 5 of 5

Given \(\bar{x} = 10\), \(\bar{y} = 50\), \(s_x = 3\), \(s_y = 9\), and \(r = 0.75\), what is the slope of the LSRL?