Historical Context & Motivation
Imagine you have a bunch of data points scattered across a graph. Maybe they show how many hours students studied and what scores they earned on a test. You can see a general trend — more studying tends to mean higher scores — but the points don't fall on a perfectly straight line. How do you draw the best possible line through those messy points? That's the question that least squares regression answers, and it's a question that mathematicians have worked on for over two hundred years.
The central question is simple but powerful: given a collection of data points that don't perfectly line up, how do we find the single straight line that comes closest to all of them at once? Least squares gives us a clear, mathematical answer — and it connects beautifully to ideas from linear algebra about projections and orthogonality.
Core Principles & Definitions
Before we dive into formulas, let's nail down the key ideas. Linear regression means fitting a straight line to data. The word "regression" just means we're trying to describe the relationship between two quantities. The "least squares" part tells us how we choose the best line: we pick the one that makes the sum of the squared differences between each data point and the line as small as possible.
Residual (Error)
Sum of Squared Errors
Best-Fit Line
Projection & Orthogonality
Visual Explanation
The diagram below shows five data points on a coordinate plane and the best-fit line running through them. The dashed vertical segments represent the residuals — the errors we want to shrink. Notice how some points sit above the line (positive residual) and some sit below (negative residual). The least squares method finds the specific line that makes the total of all those squared residuals as small as possible.
Why do we square the residuals instead of just adding them up? If we simply added the positive and negative residuals together, they would cancel each other out, and a terrible line could look like it has zero total error. Squaring makes every residual positive, and it also punishes large errors much more than small ones. A residual of 10 contributes 100 to the total, while a residual of 2 only contributes 4. This keeps the line honest — it can't just ignore faraway points.
Mathematical Framework
Now let's see the actual formulas. Suppose we have n data points: (x₁, y₁), (x₂, y₂), …, (xₙ, yₙ). We want to find a line ŷ = mx + b that best fits the data. Here, m is the slope and b is the y-intercept.
We can rewrite this problem using matrices. Think of it as a system of equations — one equation per data point — that we're trying to solve. In matrix form, the system looks like Ax = b, where A is the design matrix, x holds the unknowns (m and b), and b holds the y-values.
Usually there are more data points than unknowns, so the system is "overdetermined" — there is no exact solution. Instead, we find the closest approximate solution by projecting the vector b onto the column space of A. The magic formula that does this is called the normal equation.
The Geometry of Projection
Here's the beautiful geometric idea hiding inside the math. The vector of actual y-values (b) lives in a high-dimensional space. The set of all possible predictions from a line (Ax̂) forms a flat subspace called the column space of A. The least squares solution projects b onto this column space — it finds the point in the subspace that is closest to b. The leftover error (the residual vector) is perpendicular (orthogonal) to the column space.
This is like shining a flashlight straight down onto a tilted table. The shadow of a ball (vector b) on the table is the projection (Ax̂). The string connecting the ball to its shadow hangs straight down — perpendicular to the table. In our problem, "straight down" means the error vector e is orthogonal to every column of A. That perpendicularity condition is exactly what the normal equation AᵀA·x̂ = Aᵀb encodes.
Worked Example
Let's work through a complete example. Suppose a teacher records how many hours five students studied and the scores they earned:
| Student | Hours (x) | Score (y) |
|---|---|---|
| A | 1 | 52 |
| B | 2 | 58 |
| C | 3 | 65 |
| D | 4 | 68 |
| E | 5 | 72 |
Strengths & Limitations
Least squares regression is incredibly useful, but like any tool, it works best in certain situations. Understanding when it shines and when it struggles will make you a smarter data analyst.
| Strengths | Limitations |
|---|---|
| Easy to compute — only basic arithmetic and algebra are needed, and computers can handle millions of data points instantly. | Assumes a linear (straight-line) relationship. If the true pattern is curved, the line will be a poor fit. |
| Always gives a unique answer (as long as the data isn't degenerate). There's exactly one best-fit line. | Very sensitive to outliers. A single extreme data point can tilt the entire line because squaring magnifies large errors. |
| Strong theoretical foundation in linear algebra and statistics. It's the best unbiased estimator under certain assumptions (Gauss–Markov theorem). | Only measures vertical distances (residuals). It doesn't account for uncertainty in the x-values. |
| Extends naturally to multiple variables (multiple regression) and to polynomial curves. | Can be misleading if you don't check whether a linear model is appropriate in the first place (always plot your data first!). |
Connection to Advanced Theory
The least squares idea you've learned here is just the beginning. As you study more math, you'll discover that it connects to many powerful ideas. Here's a quick preview of where these concepts lead.
| What You Learned Here | Where It Leads |
|---|---|
| Fitting a straight line (ŷ = mx + b) | Multiple regression: fitting planes or hyperplanes with many input variables (x₁, x₂, x₃, …) |
| The normal equation AᵀA·x̂ = Aᵀb | QR decomposition: a more numerically stable way to solve least squares using orthogonal matrices |
| Projecting b onto the column space of A | Orthogonal projections in inner product spaces: generalizing to infinite-dimensional function spaces (Fourier series, wavelets) |
| Minimizing the sum of squared errors | Regularization: adding penalty terms (Ridge regression, Lasso) to prevent overfitting in machine learning |
The key geometric insight — that the best approximation comes from an orthogonal projection — is one of the most universal ideas in all of mathematics. It shows up in signal processing (filtering noise from audio), in quantum mechanics (measuring quantum states), and in statistics (computing correlations). Mastering it here gives you a head start on all of those fields.
Practice Problems
Lesson Summary
Linear regression via least squares finds the straight line that best fits a set of data points by minimizing the sum of squared residuals. Each residual is the vertical distance between a data point and the line, and squaring them ensures large errors are penalized heavily. The slope m and y-intercept b can be computed directly from the data using the slope formula or the normal equation AᵀA·x̂ = Aᵀb.
From a linear algebra perspective, the least squares solution is an orthogonal projection of the data vector onto the column space of the design matrix A. The error vector is perpendicular (orthogonal) to this column space, which guarantees that no other line can produce a smaller total error. This elegant geometric idea extends to multiple regression, Fourier analysis, and machine learning — making least squares one of the most foundational tools in all of applied mathematics.