Linear Regression via Least Squares - Linear Algebra

SECTION 1

Historical Context & Motivation

Imagine you have a bunch of data points scattered across a graph. Maybe they show how many hours students studied and what scores they earned on a test. You can see a general trend — more studying tends to mean higher scores — but the points don't fall on a perfectly straight line. How do you draw the best possible line through those messy points? That's the question that least squares regression answers, and it's a question that mathematicians have worked on for over two hundred years.

1795 • Gauss's Early Work

At just 18 years old, Carl Friedrich Gauss began using a method of minimizing squared errors to predict the orbits of planets. He didn't publish his work right away, but he was already laying the foundation.

1805 • Legendre Publishes First

French mathematician Adrien-Marie Legendre published the first formal description of the least squares method in his book on determining comet orbits. He gave it the name "méthode des moindres carrés" (method of least squares).

1809 • Gauss Publishes His Derivation

Gauss published his own treatment of the method, connecting it to probability theory and the famous normal distribution (bell curve). This gave the method a deep theoretical justification.

1829 • Gauss and Normal Equations

Gauss introduced the normal equations, a set of algebraic equations that you can solve directly to find the best-fit line. These equations are still used today.

1900s–Today • The Digital Age

With computers, least squares became the backbone of data science, machine learning, and statistics. Nearly every field — from medicine to sports analytics — uses it to find trends in data.

The central question is simple but powerful: given a collection of data points that don't perfectly line up, how do we find the single straight line that comes closest to all of them at once? Least squares gives us a clear, mathematical answer — and it connects beautifully to ideas from linear algebra about projections and orthogonality.

SECTION 2

Core Principles & Definitions

Before we dive into formulas, let's nail down the key ideas. Linear regression means fitting a straight line to data. The word "regression" just means we're trying to describe the relationship between two quantities. The "least squares" part tells us how we choose the best line: we pick the one that makes the sum of the squared differences between each data point and the line as small as possible.

Residual (Error)

The vertical distance between a data point and the line. If the point is above the line, the residual is positive; if below, it's negative. We want residuals to be as small as possible.

Sum of Squared Errors

We square each residual (making them all positive) and add them up. This total is called the sum of squared errors (SSE). The best-fit line minimizes this value.

Best-Fit Line

The unique straight line y = mx + b where the slope m and y-intercept b are chosen so that the SSE is the smallest it can possibly be. No other line does better.

Projection & Orthogonality

In linear algebra, the least squares solution is a projection of the data onto a subspace. The error vector is perpendicular (orthogonal) to that subspace — just like a shadow on the ground.

✦

KEY TAKEAWAY

Think of it like adjusting a ruler over a scatter plot. You tilt and shift it until the total gap between the ruler and every point is as small as possible. Squaring the gaps means a point that's far away from the line gets penalized a lot more than one that's close. This keeps the line from ignoring outliers.

SECTION 3

Visual Explanation

The diagram below shows five data points on a coordinate plane and the best-fit line running through them. The dashed vertical segments represent the residuals — the errors we want to shrink. Notice how some points sit above the line (positive residual) and some sit below (negative residual). The least squares method finds the specific line that makes the total of all those squared residuals as small as possible.

Each pink dashed segment shows the residual — the vertical distance between a violet data point and the cyan best-fit line. The least squares method chooses the line that makes the total of these squared distances as small as possible.

Why do we square the residuals instead of just adding them up? If we simply added the positive and negative residuals together, they would cancel each other out, and a terrible line could look like it has zero total error. Squaring makes every residual positive, and it also punishes large errors much more than small ones. A residual of 10 contributes 100 to the total, while a residual of 2 only contributes 4. This keeps the line honest — it can't just ignore faraway points.

SECTION 4

Mathematical Framework

Now let's see the actual formulas. Suppose we have n data points: (x₁, y₁), (x₂, y₂), …, (xₙ, yₙ). We want to find a line ŷ = mx + b that best fits the data. Here, m is the slope and b is the y-intercept.

SUM OF SQUARED ERRORS

SSE = Σᵢ (yᵢ − (m·xᵢ + b))²

yᵢ is the actual data value, m·xᵢ + b is the predicted value on the line, and Σ means we add this up for every data point i = 1, 2, …, n.

We can rewrite this problem using matrices. Think of it as a system of equations — one equation per data point — that we're trying to solve. In matrix form, the system looks like Ax = b, where A is the design matrix, x holds the unknowns (m and b), and b holds the y-values.

MATRIX FORM

A·x̂ = b → where A = [1 x₁; 1 x₂; …; 1 xₙ], x̂ = [b; m], b = [y₁; y₂; …; yₙ]

Each row of A has a 1 (for the intercept) and the x-value for that data point. The vector x̂ contains the two unknowns we want to find: b and m.

Usually there are more data points than unknowns, so the system is "overdetermined" — there is no exact solution. Instead, we find the closest approximate solution by projecting the vector b onto the column space of A. The magic formula that does this is called the normal equation.

NORMAL EQUATION

AᵀA · x̂ = Aᵀb

Aᵀ means the transpose of A (swap rows and columns). Multiplying both sides by Aᵀ turns the unsolvable system into one that always has a solution. The solution x̂ gives us the slope m and intercept b of the best-fit line.

SLOPE FORMULA (DIRECT)

m = (n·Σxᵢyᵢ − Σxᵢ·Σyᵢ) / (n·Σxᵢ² − (Σxᵢ)²)

This formula comes from solving the normal equation. n is the number of data points. Once you know m, you can find b = ȳ − m·x̄, where x̄ and ȳ are the averages of the x and y values.

SECTION 5

The Geometry of Projection

Here's the beautiful geometric idea hiding inside the math. The vector of actual y-values (b) lives in a high-dimensional space. The set of all possible predictions from a line (Ax̂) forms a flat subspace called the column space of A. The least squares solution projects b onto this column space — it finds the point in the subspace that is closest to b. The leftover error (the residual vector) is perpendicular (orthogonal) to the column space.

The violet vector b represents the actual data. The cyan vector Ax̂ is the closest point on the column space (the shaded plane). The pink dashed error vector e is perpendicular to the plane, shown by the yellow right-angle marker. This orthogonality is what the normal equation AᵀA·x̂ = Aᵀb guarantees.

This is like shining a flashlight straight down onto a tilted table. The shadow of a ball (vector b) on the table is the projection (Ax̂). The string connecting the ball to its shadow hangs straight down — perpendicular to the table. In our problem, "straight down" means the error vector e is orthogonal to every column of A. That perpendicularity condition is exactly what the normal equation AᵀA·x̂ = Aᵀb encodes.

💡

Why Orthogonality Matters

When the error vector is perpendicular to the column space, it means no adjustment to m or b can reduce the error any further. You've found the true minimum. This is the same idea behind dropping a perpendicular from a point to a line in geometry — it gives the shortest distance.

SECTION 6

Worked Example

Let's work through a complete example. Suppose a teacher records how many hours five students studied and the scores they earned:

Student	Hours (x)	Score (y)
A	1	52
B	2	58
C	3	65
D	4	68
E	5	72

Study hours and test scores for five students

Finding the Best-Fit Line: ŷ = mx + b

1
Step 1 — Compute the Needed Sums

We need five sums. With n = 5 data points: Σx = 1 + 2 + 3 + 4 + 5 = 15, Σy = 52 + 58 + 65 + 68 + 72 = 315, Σx² = 1 + 4 + 9 + 16 + 25 = 55, Σxy = 52 + 116 + 195 + 272 + 360 = 995.

n = 5, Σx = 15, Σy = 315, Σx² = 55, Σxy = 995
2
Step 2 — Calculate the Slope (m)

Use the slope formula: m = (n·Σxy − Σx·Σy) / (n·Σx² − (Σx)²). Plug in: m = (5 × 995 − 15 × 315) / (5 × 55 − 15²) = (4975 − 4725) / (275 − 225) = 250 / 50.

m = 5
3
Step 3 — Calculate the Y-Intercept (b)

First find the means: x̄ = 15/5 = 3 and ȳ = 315/5 = 63. Then b = ȳ − m·x̄ = 63 − 5 × 3 = 63 − 15.

b = 48
4
Step 4 — Write the Best-Fit Line

Our least squares regression line is ŷ = 5x + 48. This means that for every additional hour studied, the model predicts an increase of 5 points on the test.

ŷ = 5x + 48
5
Step 5 — Verify with a Prediction

If a student studies for 3 hours, the model predicts ŷ = 5(3) + 48 = 63. The actual score for Student C (who studied 3 hours) was 65, so the residual is 65 − 63 = 2. That's pretty close!

Predicted: 63, Actual: 65, Residual: 2

SECTION 7

Strengths & Limitations

Least squares regression is incredibly useful, but like any tool, it works best in certain situations. Understanding when it shines and when it struggles will make you a smarter data analyst.

Strengths	Limitations
Easy to compute — only basic arithmetic and algebra are needed, and computers can handle millions of data points instantly.	Assumes a linear (straight-line) relationship. If the true pattern is curved, the line will be a poor fit.
Always gives a unique answer (as long as the data isn't degenerate). There's exactly one best-fit line.	Very sensitive to outliers. A single extreme data point can tilt the entire line because squaring magnifies large errors.
Strong theoretical foundation in linear algebra and statistics. It's the best unbiased estimator under certain assumptions (Gauss–Markov theorem).	Only measures vertical distances (residuals). It doesn't account for uncertainty in the x-values.
Extends naturally to multiple variables (multiple regression) and to polynomial curves.	Can be misleading if you don't check whether a linear model is appropriate in the first place (always plot your data first!).

Comparing the strengths and limitations of least squares regression

✦

KEY TAKEAWAY

Least squares is like a GPS that always finds the shortest route — but only on roads (straight lines). If the real path is a winding mountain trail, GPS directions along straight roads won't help much. Always plot your data first to check whether a straight line is the right model before applying least squares.

SECTION 8

Connection to Advanced Theory

The least squares idea you've learned here is just the beginning. As you study more math, you'll discover that it connects to many powerful ideas. Here's a quick preview of where these concepts lead.

What You Learned Here	Where It Leads
Fitting a straight line (ŷ = mx + b)	Multiple regression: fitting planes or hyperplanes with many input variables (x₁, x₂, x₃, …)
The normal equation AᵀA·x̂ = Aᵀb	QR decomposition: a more numerically stable way to solve least squares using orthogonal matrices
Projecting b onto the column space of A	Orthogonal projections in inner product spaces: generalizing to infinite-dimensional function spaces (Fourier series, wavelets)
Minimizing the sum of squared errors	Regularization: adding penalty terms (Ridge regression, Lasso) to prevent overfitting in machine learning

From basic least squares to advanced topics

The key geometric insight — that the best approximation comes from an orthogonal projection — is one of the most universal ideas in all of mathematics. It shows up in signal processing (filtering noise from audio), in quantum mechanics (measuring quantum states), and in statistics (computing correlations). Mastering it here gives you a head start on all of those fields.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Why do we square the residuals when finding the best-fit line, instead of just adding up the absolute values of the residuals? Give two reasons.

Show answer

First, squaring ensures that positive and negative residuals don't cancel each other out, which would happen if we simply added them without taking absolute values. Second, squaring penalizes large errors more than small ones (a residual of 6 contributes 36, while a residual of 2 contributes only 4). This produces a line that avoids being too far from any single point. A bonus mathematical reason: the squared-error function is smooth and differentiable, which makes it easy to find the minimum using algebra or calculus.

Question 2: BASIC CALCULATION

Three data points are given: (1, 3), (2, 5), (3, 9). Compute the slope m and y-intercept b of the least squares line using the formulas m = (n·Σxy − Σx·Σy) / (n·Σx² − (Σx)²) and b = ȳ − m·x̄.

Show answer

First compute the sums: n = 3, Σx = 6, Σy = 17, Σx² = 14, Σxy = 1×3 + 2×5 + 3×9 = 3 + 10 + 27 = 40. Now plug into the slope formula: m = (3 × 40 − 6 × 17) / (3 × 14 − 36) = (120 − 102) / (42 − 36) = 18 / 6 = 3. Next find the means: x̄ = 6/3 = 2, ȳ = 17/3 ≈ 5.67. So b = 5.67 − 3 × 2 = 5.67 − 6 = −0.33. The best-fit line is approximately ŷ = 3x − 0.33.

Question 3: INTERMEDIATE

Using the line ŷ = 3x − 0.33 from Problem 2, calculate the residual for each of the three data points and then compute the Sum of Squared Errors (SSE). Verify that this SSE is smaller than the SSE for the line ŷ = 3x − 1.

Show answer

For ŷ = 3x − 0.33: at x = 1, ŷ = 2.67, residual = 3 − 2.67 = 0.33; at x = 2, ŷ = 5.67, residual = 5 − 5.67 = −0.67; at x = 3, ŷ = 8.67, residual = 9 − 8.67 = 0.33. SSE = 0.33² + 0.67² + 0.33² = 0.109 + 0.449 + 0.109 = 0.667. For ŷ = 3x − 1: at x = 1, ŷ = 2, residual = 1; at x = 2, ŷ = 5, residual = 0; at x = 3, ŷ = 8, residual = 1. SSE = 1 + 0 + 1 = 2. Since 0.667 < 2, the least squares line indeed has a smaller SSE, confirming it is the better fit.

Question 4: APPLIED

A small business tracks advertising spending (in hundreds of dollars) and monthly revenue (in thousands of dollars) over four months: (2, 10), (4, 14), (6, 19), (8, 22). Find the least squares line. Then use it to predict the revenue if the business spends $1,000 (i.e., x = 10) on advertising. Does this prediction seem reliable? Why or why not?

Show answer

Compute: n = 4, Σx = 20, Σy = 65, Σx² = 120, Σxy = 20 + 56 + 114 + 176 = 366. Slope: m = (4 × 366 − 20 × 65) / (4 × 120 − 400) = (1464 − 1300) / (480 − 400) = 164 / 80 = 2.05. Means: x̄ = 5, ȳ = 16.25. Intercept: b = 16.25 − 2.05 × 5 = 16.25 − 10.25 = 6. So ŷ = 2.05x + 6. For x = 10: ŷ = 2.05(10) + 6 = 26.5, meaning $26,500 in predicted revenue. This prediction should be viewed cautiously because x = 10 is outside the range of the original data (2 to 8), which is called extrapolation. The linear trend may not continue outside the observed range.

Question 5: CRITICAL THINKING

The normal equation states that Aᵀ(b − Ax̂) = 0, which means the residual vector e = b − Ax̂ is orthogonal to every column of A. Explain in your own words, using a geometric analogy, why this orthogonality condition guarantees that x̂ minimizes the length of the error vector. What would happen if the error vector were NOT perpendicular to the column space?

Show answer

Think about standing on a flat field and wanting to find the point on the field closest to a kite flying above you. The shortest string from the kite to the field goes straight down — perpendicular to the ground. If the string were tilted (not perpendicular), you could shorten it by sliding the anchor point on the ground in the direction of the tilt. In the same way, if the error vector e had any component that lay within the column space, you could adjust x̂ to cancel out that component and reduce the error. Only when e is completely perpendicular to the column space is there no possible adjustment that could make the error shorter. Mathematically, Aᵀe = 0 means the inner product of e with each column of A is zero — no column of A can 'reach toward' the error — confirming that we've truly minimized it.

SUMMARY

Lesson Summary

Summary

Linear regression via least squares finds the straight line that best fits a set of data points by minimizing the sum of squared residuals. Each residual is the vertical distance between a data point and the line, and squaring them ensures large errors are penalized heavily. The slope m and y-intercept b can be computed directly from the data using the slope formula or the normal equation AᵀA·x̂ = Aᵀb.

From a linear algebra perspective, the least squares solution is an orthogonal projection of the data vector onto the column space of the design matrix A. The error vector is perpendicular (orthogonal) to this column space, which guarantees that no other line can produce a smaller total error. This elegant geometric idea extends to multiple regression, Fourier analysis, and machine learning — making least squares one of the most foundational tools in all of applied mathematics.