Least Squares Regression - AP Statistics

SECTION 1

Historical Context & Motivation

The quest to summarize the relationship between two quantitative variables with a single straight line is one of the oldest problems in applied mathematics. Long before modern statistics existed, astronomers and physicists needed a principled way to draw a line through noisy observations so they could predict planetary orbits, track cometary paths, and refine gravitational constants. The challenge was always the same: when data points do not fall perfectly on a line, which line among infinitely many candidates should we choose? The least squares regression method provides an elegant, mathematically optimal answer to that question—one that has endured for over two centuries and remains the backbone of modern statistical modeling.

1805 • Legendre Publishes the Method

Adrien-Marie Legendre publishes Nouvelles méthodes pour la détermination des orbites des comètes, introducing the principle of minimizing the sum of squared deviations—the first formal statement of least squares.

1809 • Gauss Claims Prior Use

Carl Friedrich Gauss asserts he had been using least squares since 1795, connecting the method to the normal distribution of errors in his landmark work on celestial mechanics.

1886 • Galton's Regression to the Mean

Francis Galton coins the term regression while studying the heights of parents and children, noting that extreme values tend to regress toward the average in the next generation.

1896 • Pearson Formalizes Correlation

Karl Pearson develops the product-moment correlation coefficient r, giving regression a direct link to the strength and direction of linear association.

1900s–Present • Computational Revolution

Electronic computers and modern software (TI-84, JMP, R, Python) make least squares regression accessible to every student and researcher, enabling instant computation even for massive data sets.

The central question driving this entire history is deceptively simple: given n paired observations (x, y), how do we find the slope and intercept of a line ŷ = a + bx that best captures the linear trend in the data? The least squares criterion—minimize the sum of the squared vertical distances from each point to the line—turns that qualitative question into one with a unique, closed-form solution.

SECTION 2

Core Principles & Definitions

Before computing any regression line, it is essential to internalize the foundational ideas that make least squares both powerful and interpretable. Each principle below connects directly to how the AP Statistics exam frames regression questions—from identifying explanatory and response variables to interpreting the slope in context.

Explanatory & Response Variables

The explanatory variable (x) is the predictor; the response variable (y) is the outcome. Regression models ŷ as a linear function of x, and the direction of prediction matters.

Residuals

A residual is the vertical distance from an observed point to the regression line: residual = y − ŷ. Positive residuals lie above the line; negative residuals lie below it.

Minimizing Σ(residual²)

The least squares criterion chooses the line that makes Σ(yᵢ − ŷᵢ)² as small as possible. Squaring the residuals penalizes large deviations more heavily and ensures the total is always non-negative.

The Line Passes Through (x̄, ȳ)

A critical algebraic consequence: the least squares regression line (LSRL) always passes through the point of means (x̄, ȳ). This anchors the line to the center of the data cloud.

r² — Coefficient of Determination

The value r² tells us the proportion of variability in y that is explained by the linear relationship with x. It ranges from 0 (no linear fit) to 1 (perfect linear fit).

✦

KEY TAKEAWAY

Think of the LSRL like a tightly stretched rubber band running through a scatter of push-pins on a board. If every pin exerts a pull proportional to the square of its vertical distance from the band, the equilibrium position of the band is exactly the least squares line—it balances the total squared tension. That physics-style analogy explains why the line is unique: there is only one equilibrium.

SECTION 3

Visual Explanation — Scatter Plot & LSRL

Each cyan dot represents an observed (x, y) pair. The violet line is the LSRL. The dashed red segments show residuals—the vertical gaps between each observed y-value and its predicted ŷ. The LSRL is the unique line that minimizes the sum of the squares of these red segments.

Notice how some residual segments are longer than others. Points far from the line contribute disproportionately to the sum of squared residuals because squaring amplifies large deviations. The least squares algorithm, in effect, compromises: it allows slightly larger residuals where many points cluster close to the line in order to pull the line closer to the most deviant observations. This balancing act is why the LSRL passes through (x̄, ȳ)—it anchors itself at the center of the data cloud and rotates to minimize total squared error.

SECTION 4

Mathematical Framework

The LSRL is written ŷ = a + bx, where b is the slope and a is the y-intercept. The formulas below are derived by taking partial derivatives of the sum of squared residuals with respect to a and b, setting each to zero, and solving the resulting system of two linear equations (the normal equations).

SLOPE OF THE LSRL

b = r × (s_y / s_x)

where r is the correlation coefficient, s_y is the standard deviation of y, and s_x is the standard deviation of x. The slope inherits the sign of r and rescales it by the ratio of spread in y to spread in x.

Y-INTERCEPT OF THE LSRL

a = ȳ − b × x̄

This formula guarantees that the LSRL passes through the point (x̄, ȳ). It is derived directly from substituting the mean point into the line equation ŷ = a + bx.

COEFFICIENT OF DETERMINATION

r² = 1 − (SSresid / SStotal)

SS_resid = Σ(yᵢ − ŷᵢ)² is the residual sum of squares; SS_total = Σ(yᵢ − ȳ)² is the total sum of squares. Equivalently, r² equals the square of the correlation coefficient r. It represents the fraction of total variability in y accounted for by the linear model.

STANDARD DEVIATION OF RESIDUALS

s = √[ Σ(yᵢ − ŷᵢ)² / (n − 2) ]

The denominator is n − 2 because two parameters (a and b) have been estimated. This value s measures the typical size of a residual—the typical vertical distance an observed y falls from the LSRL.

💡

AP Exam Tip

When interpreting slope on the AP exam, always include context: "For each additional [unit of x], the predicted [y variable] changes by [b] [units of y]." Graders award separate points for the word predicted (or estimated) and for naming both variables.

SECTION 5

Residual Analysis & Assessing Fit

Computing the LSRL is only half the story; we must also evaluate whether a linear model is appropriate. The primary diagnostic tool is the residual plot, which graphs residuals (y − ŷ) on the vertical axis against either x or ŷ on the horizontal axis. A well-behaved residual plot shows random scatter with no discernible pattern. Conversely, a curved pattern in the residual plot signals that a straight line is not the best model for the data and that a nonlinear transformation or a polynomial model should be considered.

Left: a good residual plot with random scatter about the zero line, indicating a linear model is appropriate. Right: a problematic residual plot showing a curved pattern, indicating the relationship is not linear and a different model or variable transformation should be considered.

Beyond curvature, watch for heteroscedasticity—a fanning pattern where the spread of residuals increases (or decreases) as x increases. This condition does not invalidate the regression equation itself, but it signals that predictions are less reliable at one end of the x-range than the other. On the AP exam, describing what you see in a residual plot (pattern versus no pattern, increasing spread versus constant spread) is essential for earning full credit on free-response questions about model adequacy.

✓ No pattern → linear model is appropriate
✓ Curved pattern → nonlinear relationship; consider transforming x, y, or both
✓ Fan shape → non-constant variance (heteroscedasticity); predictions less reliable at the wider end
✓ One or two extreme residuals → potential outliers or influential points; investigate their effect on slope and r²

SECTION 6

Worked Example

A researcher records the number of hours eight students spent studying for a statistics exam and their resulting exam scores. Use the summary statistics to find the LSRL, interpret the slope and y-intercept in context, and calculate r².

Summary Statistic	Value
n	8
x̄ (mean study hours)	5.0
ȳ (mean exam score)	72.0
sₓ	2.4
$s_y$	10.0
r	0.92

Summary statistics for the study-hours data

Finding and Interpreting the LSRL

1
Step 1 — Compute the Slope (b)

Use the formula b = r × (s_y / s_x). Substituting: b = 0.92 × (10.0 / 2.4) = 0.92 × 4.1667 ≈ 3.833.

b ≈ 3.833
2
Step 2 — Compute the Y-Intercept (a)

Use a = ȳ − b × x̄. Substituting: a = 72.0 − 3.833 × 5.0 = 72.0 − 19.167 ≈ 52.833.

a ≈ 52.833
3
Step 3 — Write the LSRL Equation

Combining the slope and intercept: ŷ = 52.833 + 3.833x, where x = study hours and ŷ = predicted exam score.

ŷ = 52.833 + 3.833x
4
Step 4 — Interpret the Slope in Context

For each additional hour of studying, the predicted exam score increases by approximately 3.833 points. The word 'predicted' is critical—the LSRL gives estimates, not certainties.
5
Step 5 — Interpret the Y-Intercept in Context

When study hours = 0, the predicted exam score is approximately 52.833. Whether this interpretation is meaningful depends on whether x = 0 is within the range of observed data. Here it may be an extrapolation and should be stated with caution.
6
Step 6 — Calculate and Interpret r²

r² = (0.92)² = 0.8464. Interpretation: approximately 84.6% of the variability in exam scores is explained by the linear relationship with study hours.

r² ≈ 0.846 — 84.6% of variability explained

SECTION 7

Strengths, Limitations & Common Pitfalls

Strengths	Limitations
Closed-form solution—fast and unique; always yields a single best line	Only captures linear associations; curved relationships require transformation
Interpretable coefficients: slope and intercept have direct contextual meaning	Sensitive to outliers and influential points, which can dramatically shift the line
r² provides a clear measure of explanatory power	Does not establish causation—association ≠ causation
Residual analysis offers a visual check on model adequacy	Extrapolation beyond the observed x-range is unreliable
Foundation for more complex models (multiple regression, ANOVA)	Assumes homoscedasticity (constant variance of residuals) for inference

Strengths and limitations of least squares regression

⚠

COMMON AP EXAM PITFALLS

Three errors appear repeatedly on AP Statistics exams. First, students forget to say predicted when interpreting slope—this single word distinguishes a regression interpretation from a deterministic statement. Second, students confuse r with r²; remember that r measures direction and strength of linear association, while r² measures the proportion of variability explained. Third, students extrapolate the regression line far beyond the data and make unwarranted predictions—like predicting exam scores for 50 hours of study when the data only goes up to 10.

SECTION 8

Connection to Inference & Advanced Topics

The LSRL you compute from sample data is an estimate of the true population regression line β₀ + β₁x. In the inference unit of AP Statistics, you will test whether the true slope β₁ is significantly different from zero—a t-test for the slope—and construct confidence intervals for β₁. These procedures rely on the same residual standard deviation s introduced in Section 4 and require that residuals be approximately normal with constant variance. Understanding the mechanics of least squares now will make the inferential extension feel natural rather than opaque.

Topic	Descriptive LSRL (This Lesson)	Inference for Regression (Later)
Goal	Summarize the linear relationship in a sample	Test whether the population slope β₁ differs from 0
Key statistic	b (sample slope), r, r²	t = b / SE_b with df = n − 2
Conditions	Linearity (check residual plot)	Linearity, independence, normality of residuals, equal variance
Output	Equation ŷ = a + bx, r², s	p-value, confidence interval for β₁

Descriptive LSRL versus inference for regression

Beyond simple linear regression, the same least squares principle extends to multiple regression (more than one explanatory variable), polynomial regression (fitting curves such as ŷ = a + b₁x + b₂x²), and generalized linear models. Every one of these more advanced techniques builds on the intuition you develop here: minimize a measure of prediction error to find the best-fitting model.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

A least squares regression line is fit to a set of data. Which of the following must always be true?

A. The regression line passes through every data point.
B. The sum of the residuals is zero.
C. The slope of the line is always positive.
D. The value of r² is always greater than 0.5.
E. Every residual is less than one standard deviation of y.

Show answer (B)

B is correct. A mathematical property of the LSRL is that the sum of the residuals Σ(yᵢ − ŷᵢ) always equals zero—positive and negative residuals cancel out exactly. A is wrong because the line minimizes squared residuals, not making them zero; it generally passes through no individual point (except the point of means). C is wrong because slope can be negative when r is negative. D is wrong because r² can be any value between 0 and 1 depending on how well the linear model fits. E is wrong because individual residuals can exceed one standard deviation of y, especially for data points far from the regression line.

Question 2: BASIC CALCULATION

For a data set with x̄ = 10, ȳ = 50, s_x = 3, s_y = 12, and r = −0.80, what is the equation of the LSRL?

A. ŷ = 82.0 − 3.2x
B. ŷ = 50.0 − 3.2x
C. ŷ = 82.0 + 3.2x
D. ŷ = 18.0 + 3.2x
E. ŷ = 70.0 − 2.0x

Show answer (A)

A is correct. First, compute the slope: b = r × (

s_y

s_x

) = −0.80 × (12 / 3) = −0.80 × 4 = −3.2. Then the intercept: a = ȳ − b × x̄ = 50 − (−3.2)(10) = 50 + 32 = 82.0. So ŷ = 82.0 − 3.2x. B incorrectly uses ȳ as the intercept without adjusting for the slope. C has the right intercept but wrong sign on the slope; since r is negative, the slope must be negative. D reverses the intercept calculation (50 − 32 = 18 instead of 50 + 32 = 82) and flips the slope sign. E uses an incorrect slope calculation that does not match r × (

s_y

s_x

Question 3: INTERMEDIATE

The equation of the LSRL for predicting a car's fuel efficiency (miles per gallon) from its weight (thousands of pounds) is ŷ = 48.7 − 8.1x. A car that weighs 3.5 thousand pounds gets 22 mpg. What is the residual for this car, and what does it mean in context?

A. Residual = 1.65; the car gets 1.65 mpg more than predicted.
B. Residual = −1.65; the car gets 1.65 mpg less than predicted.
C. Residual = 1.65; the car weighs 1.65 thousand pounds more than predicted.
D. Residual = −20.35; the car gets 20.35 mpg less than the mean.
E. Residual = 22; no comparison to the predicted value is needed.

Show answer (A)

A is correct. First compute the predicted value: ŷ = 48.7 − 8.1(3.5) = 48.7 − 28.35 = 20.35 mpg. The residual is y − ŷ = 22 − 20.35 = 1.65. A positive residual means the actual mpg is 1.65 higher than the model predicted. B reverses the sign—it should be y − ŷ, not ŷ − y. C misidentifies the residual as a weight difference; residuals are always in the units of the response variable. D confuses the predicted value (20.35) with the residual and subtracts from the wrong reference. E simply states the observed value without computing a residual, which is the difference between observed and predicted.

Question 4: APPLIED

5 pts

A marine biologist studying coral reef health collected data on water temperature (°C) and coral coverage (% of reef area) at 25 sites. Computer output is shown below: Predictor Coef SE Coef T P Constant 118.5 6.8 17.43 0.000 Temperature −3.20 0.27 −11.85 0.000 s = 4.12 R-Sq = 85.9% R-Sq(adj) = 85.3% (a) Write the equation of the least-squares regression line. Define any variables used. (b) Interpret the slope in context. (c) Interpret r² in context. (d) A site has a water temperature of 30°C. Calculate the predicted coral coverage. Then explain why using this model to predict coral coverage at a temperature of 45°C would be problematic.

Show answer

Scoring rubric (5 points): Point 1 — Equation (part a): ŷ = 118.5 − 3.20x, where x = water temperature (°C) and ŷ = predicted coral coverage (%). Must use ŷ (or 'predicted coral coverage'), not y. Point 2 — Slope interpretation (part b): For each additional degree Celsius increase in water temperature, the predicted coral coverage decreases by 3.20 percentage points. Must include 'predicted' (or 'estimated'), identify both variables by name, and state the direction (decrease). Point 3 — r² interpretation (part c): Approximately 85.9% of the variability in coral coverage is explained by the linear relationship with water temperature. Must reference 'variability' (or 'variation') in the response variable and name the explanatory variable. Point 4 — Prediction at 30°C (part d, first part): ŷ = 118.5 − 3.20(30) = 118.5 − 96.0 = 22.5%. The predicted coral coverage at 30°C is 22.5%. Point 5 — Extrapolation concern (part d, second part): Predicting at 45°C would be extrapolation because 45°C is almost certainly far outside the range of water temperatures observed at the 25 coral reef sites. The linear relationship established within the observed data range may not hold at such extreme temperatures, making the prediction unreliable. At 45°C the model would predict ŷ = 118.5 − 3.20(45) = −25.5%, which is a nonsensical negative percentage.

Question 5: CRITICAL THINKING

5 pts

A statistics student claims that because the correlation between hours of TV watched per day and GPA is r = −0.65, watching more TV causes GPA to drop. The student also notes that the LSRL is ŷ = 3.8 − 0.15x, and since r² = 0.4225, the model explains a large amount of variability. (a) Identify and explain the flaw in the student's causal claim. (b) Give a plausible confounding variable and explain how it could produce the observed negative association without TV directly causing lower GPA. (c) The student interprets r² = 0.4225 as meaning the model 'explains most of the variability in GPA.' Evaluate this claim. (d) Using the LSRL, compute the predicted GPA for a student who watches 6 hours of TV per day and the residual if that student's actual GPA is 3.1. Interpret the residual.

Show answer

Scoring rubric (5 points): Point 1 — Causal flaw (part a): Correlation does not imply causation. The LSRL and correlation coefficient describe the strength and direction of a linear association, but an observational study (as this appears to be) cannot establish that TV watching causes GPA to decrease. There may be lurking or confounding variables that explain the association. Point 2 — Confounding variable (part b): A plausible confounding variable is the student's level of motivation or engagement with school. Students with lower academic motivation may both watch more TV and invest less effort in coursework, producing a negative association between TV hours and GPA even if TV watching itself has no direct effect on academic performance. The confound must be linked to both x and y. Point 3 — Evaluating the r² claim (part c): The claim that r² = 0.4225 means the model explains 'most' of the variability is incorrect. An r² of about 42% means that only about 42% of the variability in GPA is explained by the linear relationship with TV hours; the remaining 58% is unexplained. While 42% is nontrivial, it does not constitute 'most' of the variability. GPA is influenced by many other factors. Point 4 — Prediction and residual calculation (part d): Predicted GPA: ŷ = 3.8 − 0.15(6) = 3.8 − 0.9 = 2.9. Residual = y − ŷ = 3.1 − 2.9 = 0.2. Point 5 — Residual interpretation (part d continued): The positive residual of 0.2 means that this student's actual GPA is 0.2 points higher than the GPA predicted by the LSRL for a student who watches 6 hours of TV per day. The observed data point lies above the regression line.

SUMMARY

Lesson Summary

Summary

The least squares regression line (LSRL) is the unique line ŷ = a + bx that minimizes the sum of squared residuals Σ(yᵢ − ŷᵢ)². The slope b = r × ( $s_y$ / $s_x$ ) quantifies the predicted change in y per unit increase in x, and the y-intercept a = ȳ − bx̄ anchors the line through the point of means (x̄, ȳ). The coefficient of determination r² tells us the proportion of variability in the response variable explained by the linear model.

Always check the residual plot for patterns: random scatter supports a linear model, while curvature or fanning signals inadequacy. Remember that correlation does not imply causation, extrapolation beyond the observed data is risky, and influential points can dramatically alter the LSRL. When interpreting slope on the AP exam, always include the word predicted, name both variables in context, and state the direction and magnitude of the change.