01 / 21

Geometric
Applications

Master vector projections and least-squares best fits: the geometric heart of linear algebra that powers everything from 3D graphics to data science.

Linear Algebra

What is a Projection?

Imagine shining a flashlight straight down onto a line. The shadow a vector casts on that line is its projection. In linear algebra, projecting \(\mathbf{b}\) onto \(\mathbf{a}\) means finding the point on the line through \(\mathbf{a}\) that is closest to \(\mathbf{b}\).

\(\mathbf{a}\) \(\mathbf{b}\) \(\text{proj}_{\mathbf{a}}(\mathbf{b})\) \(\mathbf{e}\) O

The Projection Formula

To project \(\mathbf{b}\) onto the direction of \(\mathbf{a}\), we use the dot product to measure how much of \(\mathbf{b}\) points along \(\mathbf{a}\):

The error vector is the part left over:

Why Projections Matter

Best Fits
Least-squares regression finds the line closest to data by projecting onto a column space.
3D Graphics
Shadows, lighting, and camera views all rely on projecting vectors onto planes and directions.
Signal Processing
Fourier analysis decomposes signals by projecting onto sine and cosine basis vectors.

The Projection Theorem

\(W\) \(\mathbf{b}\) \(\hat{\mathbf{b}}\) \(\mathbf{e} = \mathbf{b} - \hat{\mathbf{b}}\) O

The key insight: \(\hat{\mathbf{b}}\) minimizes \(\|\mathbf{b} - \mathbf{x}\|\) for all \(\mathbf{x} \in W\). This is because the shortest distance from a point to a plane is always perpendicular.

Projection onto a Subspace

When projecting onto the column space of a matrix \(A\), we need the projection matrix \(P\):

1
\(P\) is symmetric: \(P^T = P\). The projection "looks the same" from both sides.
2
\(P\) is idempotent: \(P^2 = P\). Projecting twice gives the same result as once.
3
Error is orthogonal: \(\mathbf{e} = \mathbf{b} - P\mathbf{b}\) lies in the left nullspace of \(A\).

Projection vs Reflection

Projection Reflection
Matrix \(P = A(A^T\!A)^{-1}A^T\) \(R = 2P - I\)
Effect Drops component perpendicular to \(W\) Flips component perpendicular to \(W\)
Idempotent? Yes: \(P^2 = P\) No: \(R^2 = I\)
Eigenvalues \(0\) and \(1\) \(-1\) and \(1\)
Preserves length? No (shrinks) Yes (orthogonal)

From Projections to Best Fits

When \(A\mathbf{x} = \mathbf{b}\) has no exact solution (more equations than unknowns), we want the \(\hat{\mathbf{x}}\) that makes \(A\hat{\mathbf{x}}\) as close to \(\mathbf{b}\) as possible.

\(\text{Col}(A)\) \(\mathbf{b}\) \(A\hat{\mathbf{x}} = \hat{\mathbf{b}}\) \(\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}\)

How to Find the Least-Squares Fit

1
Set up \(A\) and \(\mathbf{b}\): Write the system \(A\mathbf{x} = \mathbf{b}\), where \(A\) encodes the model and \(\mathbf{b}\) is the observed data.
2
Compute \(A^T\!A\): Multiply the transpose of \(A\) by \(A\). This is a square, symmetric matrix.
3
Compute \(A^T\!\mathbf{b}\): Multiply the transpose of \(A\) by the data vector \(\mathbf{b}\).
4
Solve the normal equations: \(A^T\!A\,\hat{\mathbf{x}} = A^T\!\mathbf{b}\) for \(\hat{\mathbf{x}}\).
5
Interpret: \(\hat{\mathbf{x}}\) gives the coefficients of the best-fit model. The error is \(\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}\).

Example: Vector Projection

Project \(\mathbf{b} = \begin{pmatrix} 1 \\ 2 \\ 2 \end{pmatrix}\) onto \(\mathbf{a} = \begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix}\).

1
Dot products: \(\mathbf{a} \cdot \mathbf{b} = 1(1) + 0(2) + 1(2) = \) 3,   \(\mathbf{a} \cdot \mathbf{a} = 1 + 0 + 1 = \) 2
2
Scalar factor: \(\displaystyle \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}} = \frac{3}{2}\)
3
Projection: \(\displaystyle \text{proj}_{\mathbf{a}}(\mathbf{b}) = \frac{3}{2}\begin{pmatrix} 1 \\ 0 \\ 1 \end{pmatrix} = \begin{pmatrix} 3/2 \\ 0 \\ 3/2 \end{pmatrix}\)

Example: Best-Fit Line

Find the best-fit line \(y = c + dt\) through the points \((0,1)\), \((1,3)\), \((2,4)\).

1
\(A^T\!A = \begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}\),   \(A^T\!\mathbf{b} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}\)
2
Solve: \(\begin{pmatrix} 3 & 3 \\ 3 & 5 \end{pmatrix}\begin{pmatrix} c \\ d \end{pmatrix} = \begin{pmatrix} 8 \\ 11 \end{pmatrix}\) → \(c = \frac{7}{6}\), \(d = \frac{3}{2}\)
3
Best-fit line: \(y = \frac{7}{6} + \frac{3}{2}\,t\)

Visualizing the Best Fit

\(t\) \(y\) 0 1 2 1 2 3 4 \((0, 1)\) \((1, 3)\) \((2, 4)\) \(y = \frac{7}{6} + \frac{3}{2}t\) residuals

Example: Projection Matrix

Find the projection matrix \(P\) that projects onto the column space of \(A = \begin{pmatrix} 1 \\ 2 \end{pmatrix}\).

1
\(A^T\!A = \begin{pmatrix} 1 & 2 \end{pmatrix}\begin{pmatrix} 1 \\ 2 \end{pmatrix} = 5\)
2
\(P = A(A^T\!A)^{-1}A^T = \frac{1}{5}\begin{pmatrix} 1 \\ 2 \end{pmatrix}\begin{pmatrix} 1 & 2 \end{pmatrix} = \frac{1}{5}\begin{pmatrix} 1 & 2 \\ 2 & 4 \end{pmatrix}\)
3
Verify: \(P^2 = P\) ✓   \(P^T = P\) ✓   Eigenvalues: \(0, 1\) ✓

The Orthogonal Decomposition

Every vector \(\mathbf{b}\) can be uniquely split into two perpendicular components relative to a subspace \(W\):

\(\mathbf{b}\) \(\hat{\mathbf{b}} = \text{proj}_W(\mathbf{b})\) \(\mathbf{e} = \mathbf{b} - \hat{\mathbf{b}}\) \(W\) = +

Common Mistakes to Avoid

✕ Wrong Denominator
Using \(\|\mathbf{a}\|\) instead of \(\mathbf{a} \cdot \mathbf{a}\) in the vector projection formula. Remember: the scalar projection divides by the norm, but the vector projection divides by the squared norm.
✕ Forgetting Perpendicularity
Not checking that \(\mathbf{e} \perp \mathbf{a}\) after computing. Always verify: \(\mathbf{a} \cdot \mathbf{e}\) should equal zero.
✕ Inverting \(A\) Directly
Trying to solve \(A\mathbf{x} = \mathbf{b}\) directly when \(A\) is not square. You must use the normal equations \(A^T\!A\,\hat{\mathbf{x}} = A^T\!\mathbf{b}\).

Summary of Key Formulas

Concept Formula Key Property
Scalar projection \(\frac{\mathbf{a} \cdot \mathbf{b}}{\|\mathbf{a}\|}\) Signed length of shadow
Vector projection \(\frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\,\mathbf{a}\) \(\hat{\mathbf{b}} \in \text{span}(\mathbf{a})\)
Projection matrix \(A(A^T\!A)^{-1}A^T\) \(P^2 = P,\; P^T = P\)
Normal equations \(A^T\!A\,\hat{\mathbf{x}} = A^T\!\mathbf{b}\) Minimizes \(\|\mathbf{b} - A\mathbf{x}\|^2\)
Error vector \(\mathbf{e} = \mathbf{b} - A\hat{\mathbf{x}}\) \(\mathbf{e} \perp \text{Col}(A)\)

Question 1 of 5

What is the projection of \(\mathbf{b} = \begin{pmatrix} 3 \\ 4 \end{pmatrix}\) onto \(\mathbf{a} = \begin{pmatrix} 1 \\ 0 \end{pmatrix}\)?

Question 2 of 5

If \(P\) is a projection matrix, which property must it satisfy?

Question 3 of 5

The least-squares solution \(\hat{\mathbf{x}}\) to \(A\mathbf{x} = \mathbf{b}\) satisfies which equation?

Question 4 of 5

If \(\hat{\mathbf{b}}\) is the orthogonal projection of \(\mathbf{b}\) onto subspace \(W\), and \(\mathbf{e} = \mathbf{b} - \hat{\mathbf{b}}\), what is \(\hat{\mathbf{b}} \cdot \mathbf{e}\)?

Question 5 of 5

What are the eigenvalues of a projection matrix \(P\) that projects onto a 2-dimensional subspace of \(\mathbb{R}^4\)?