Least Squares
Regression
Master the technique of fitting the best line through data by minimizing the sum of squared residuals.
What is Regression?
Regression is a statistical method for modeling the relationship between two quantitative variables. Given an explanatory variable \(x\) and a response variable \(y\), regression finds a function that best predicts \(y\) from \(x\).
Scatterplots and Linear Patterns
Before fitting a line, always create a scatterplot. Check for a linear pattern, the direction (positive or negative), and any outliers.
Key Vocabulary
The LSRL Formulas
The least squares regression line is: $$ \hat{y} = a + bx $$
Minimizing Squared Residuals
The LSRL minimizes: $$ \sum_{i=1}^{n} (y_i - \hat{y}_i)^2 = \sum_{i=1}^{n} e_i^2 $$ Each red square represents a squared residual. The green line is positioned to make the total area of all squares as small as possible.
Correlation vs Regression
| Correlation \(r\) | Regression | |
|---|---|---|
| Purpose | Measures strength and direction of linear association | Predicts \(y\) from \(x\) |
| Output | A single number: \(-1 \leq r \leq 1\) | An equation: \(\hat{y} = a + bx\) |
| Roles of \(x, y\) | Symmetric: switching \(x\) and \(y\) gives same \(r\) | Not symmetric: switching changes the line |
| Units | Unitless | Slope has units \(y/x\) |
The Power of \(r^2\)
The coefficient of determination \(r^2\) tells you the proportion of variation in \(y\) that is explained by the regression on \(x\).
Reading Residual Plots
A random scatter around the zero line confirms the linear model is appropriate. A curved pattern means a linear model is not a good fit: consider a transformation.
Influential Points and Outliers
Step-by-Step Procedure
Example 1: Study Hours
A teacher records the hours studied (\(x\)) and exam score (\(y\)) for 6 students. Summary statistics: \(\bar{x} = 4\), \(\bar{y} = 76\), \(s_x = 1.8\), \(s_y = 10.2\), \(r = 0.92\).
Example 2: Making a Prediction
Using the equation from Example 1, \(\hat{y} = 55.16 + 5.21x\), predict the exam score for a student who studies 5.5 hours.
Example 3: Computing a Residual
Suppose a student who studied \(x = 3\) hours actually scored \(y = 72\). Using \(\hat{y} = 55.16 + 5.21x\):
Common Mistakes
Summary Reference Table
| Concept | Formula / Key Idea | Interpretation |
|---|---|---|
| LSRL Equation | \(\hat{y} = a + bx\) | Predicted value of \(y\) for a given \(x\) |
| Slope \(b\) | \(b = r \cdot \frac{s_y}{s_x}\) | Change in \(\hat{y}\) per unit change in \(x\) |
| Intercept \(a\) | \(a = \bar{y} - b\bar{x}\) | Predicted \(y\) when \(x = 0\) |
| Residual | \(e_i = y_i - \hat{y}_i\) | Positive: above line; Negative: below line |
| \(r^2\) | Square of correlation | % of variation in \(y\) explained by \(x\) |
Question 1 of 5
A least squares regression line has the equation \(\hat{y} = 12.4 + 3.2x\). What is the predicted value of \(y\) when \(x = 5\)?
Question 2 of 5
If the correlation between \(x\) and \(y\) is \(r = 0.80\), what is the coefficient of determination \(r^2\)?
Question 3 of 5
The LSRL for predicting weight (\(y\), in kg) from height (\(x\), in cm) is \(\hat{y} = -105 + 1.1x\). A person is 170 cm tall and weighs 85 kg. What is the residual?
Question 4 of 5
A residual plot shows a clear U-shaped pattern. What does this indicate about the linear model?
Question 5 of 5
Given \(\bar{x} = 10\), \(\bar{y} = 50\), \(s_x = 3\), \(s_y = 9\), and \(r = 0.75\), what is the slope of the LSRL?