01 / 20

Projections &
Decompositions

Learn how to project vectors onto subspaces and decompose them into orthogonal components: the foundation of least squares, signal processing, and dimensionality reduction.

Linear Algebra

What Is a Projection?

A projection takes a vector and "drops" it onto a subspace, finding the closest point in that subspace. Think of shining a flashlight straight down on a vector: the shadow on the floor is the projection.

W
\( \mathbf{b} \)
\( \text{proj}_W \mathbf{b} \)
\( \mathbf{e} \)
0

Why Do Projections Matter?

Projections appear everywhere in applied mathematics. Whenever you need the best approximation of a vector inside a subspace, you are computing a projection.

Least Squares
Finding the best-fit line through data points uses \( A^T A \hat{x} = A^T b \), which is a projection.
Signal Processing
Fourier analysis projects signals onto orthogonal frequency components.
Dimensionality Reduction
PCA projects high-dimensional data onto the subspace of greatest variance.

Projection onto a Line

The simplest case: projecting vector \( \mathbf{b} \) onto the line spanned by vector \( \mathbf{a} \). The formula uses the dot product:

The scalar \( \hat{x} = \dfrac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}} \) is the component of \( \mathbf{b} \) along \( \mathbf{a} \). Multiplying by \( \mathbf{a} \) gives the projection vector.

The Projection Matrix

We can write the projection as a matrix multiplication. For projection onto the line through \( \mathbf{a} \):

\( P^2 = P \)
Projecting twice gives the same result. Once you're on the subspace, you stay there.
\( P^T = P \)
Projection matrices are symmetric: projecting is a "fair" operation from both sides.

Projection onto a Subspace

When projecting onto a subspace \( W = \text{Col}(A) \) spanned by multiple columns, the formula generalizes:

This requires \( A^T A \) to be invertible, which holds when the columns of \( A \) are linearly independent.

Projection vs Decomposition

These two ideas are deeply connected but address different questions:

Projection Decomposition
Question What is the closest point in \( W \)? How do we split \( \mathbf{b} \) into parts?
Output A single vector \( \hat{\mathbf{b}} \in W \) Two vectors: \( \hat{\mathbf{b}} + \mathbf{e} \)
Key Property \( P^2 = P \), \( P^T = P \) \( \hat{\mathbf{b}} \perp \mathbf{e} \)
Formula \( P\mathbf{b} \) \( \mathbf{b} = P\mathbf{b} + (I - P)\mathbf{b} \)

The Orthogonal Decomposition Theorem

W b
\( \hat{\mathbf{b}} = \text{proj}_W \mathbf{b} \)
\( \mathbf{e} \in W^{\perp} \)

Projection with an Orthogonal Basis

If \( W \) has an orthogonal basis \( \{ \mathbf{u}_1, \mathbf{u}_2, \ldots, \mathbf{u}_k \} \), then the projection simplifies beautifully:

Each term is a separate 1D projection onto one basis vector. With orthogonal bases, the cross-terms vanish: no matrix inversion needed!

How to Compute a Projection

1
Identify the subspace \( W \). Determine the column space or span you are projecting onto.
2
Form the matrix \( A \). Arrange the basis vectors of \( W \) as columns of \( A \).
3
Compute \( A^T A \) and \( A^T \mathbf{b} \). Build the normal equation pieces.
4
Solve \( A^T A \hat{\mathbf{x}} = A^T \mathbf{b} \). This gives the coefficient vector \( \hat{\mathbf{x}} \).
5
Compute \( \hat{\mathbf{b}} = A\hat{\mathbf{x}} \). The projection is the result. The error is \( \mathbf{e} = \mathbf{b} - \hat{\mathbf{b}} \).

Example 1: Projection onto a Line

Project \( \mathbf{b} = \begin{bmatrix} 1 \\ 2 \\ 3 \end{bmatrix} \) onto the line through \( \mathbf{a} = \begin{bmatrix} 1 \\ 1 \\ 1 \end{bmatrix} \).

span(a)
\( \mathbf{b} \)
\( \hat{\mathbf{b}} = \begin{bmatrix}2\\2\\2\end{bmatrix} \)
\( \mathbf{e} \)

Example 2: Projection onto a Plane

Project \( \mathbf{b} = \begin{bmatrix}1\\0\\1\end{bmatrix} \) onto \( W = \text{Col}(A) \) where \( A = \begin{bmatrix}1 & 0 \\ 0 & 1 \\ 1 & 0\end{bmatrix} \).

Example 3: Orthogonal Basis Shortcut

Project \( \mathbf{b} = \begin{bmatrix}1\\2\\3\end{bmatrix} \) onto \( W = \text{span}\left\{ \mathbf{u}_1 = \begin{bmatrix}1\\0\\1\end{bmatrix},\; \mathbf{u}_2 = \begin{bmatrix}0\\1\\0\end{bmatrix} \right\} \). Note \( \mathbf{u}_1 \cdot \mathbf{u}_2 = 0 \): orthogonal!

Properties of Projection Matrices

1
Idempotent: \( P^2 = P \). Projecting twice has the same effect as projecting once.
2
Symmetric: \( P^T = P \). The projection matrix equals its own transpose.
3
Eigenvalues: Only \( 0 \) and \( 1 \). Vectors in \( W \) get eigenvalue 1; vectors in \( W^{\perp} \) get eigenvalue 0.
4
Complement: \( I - P \) projects onto \( W^{\perp} \). So \( (I - P)\mathbf{b} = \mathbf{e} \).
5
Rank: \( \text{rank}(P) = \dim(W) \). The rank tells you the dimension of the target subspace.

Common Mistakes

✕ Wrong Order in Formula
Writing \( A^T(AA^T)^{-1} \) instead of \( A(A^TA)^{-1}A^T \). The order \( A^TA \) vs \( AA^T \) matters: they are different matrices.
✕ Non-Orthogonal Shortcut
Using the orthogonal basis formula \( \sum \frac{\mathbf{u}_i \cdot \mathbf{b}}{\|\mathbf{u}_i\|^2}\mathbf{u}_i \) when the basis vectors are not orthogonal. This only works with orthogonal sets.
✕ Forgetting to Verify
Not checking that the error \( \mathbf{e} \) is perpendicular to the subspace. Always verify \( A^T\mathbf{e} = \mathbf{0} \) as a sanity check.

Summary of Formulas

Onto a Line Onto a Subspace
Target \( \text{span}(\mathbf{a}) \) \( \text{Col}(A) \)
Projection \( \frac{\mathbf{a} \cdot \mathbf{b}}{\mathbf{a} \cdot \mathbf{a}}\mathbf{a} \) \( A(A^TA)^{-1}A^T\mathbf{b} \)
Proj. Matrix \( \frac{\mathbf{a}\mathbf{a}^T}{\mathbf{a}^T\mathbf{a}} \) \( A(A^TA)^{-1}A^T \)
Error \( \mathbf{b} - P\mathbf{b} \) \( (I - P)\mathbf{b} \)
Orthogonal? Always (1D line) Use shortcut if basis is orthogonal

Question 1 of 5

Project \( \mathbf{b} = \begin{bmatrix}3\\4\end{bmatrix} \) onto \( \mathbf{a} = \begin{bmatrix}1\\0\end{bmatrix} \). What is \( \text{proj}_{\mathbf{a}} \mathbf{b} \)?

Question 2 of 5

Which property does every orthogonal projection matrix \( P \) satisfy?

Question 3 of 5

If \( P \) projects onto \( W \), what does \( I - P \) project onto?

Question 4 of 5

You are given an orthogonal basis \( \{ \mathbf{u}_1, \mathbf{u}_2 \} \) for \( W \). To compute \( \text{proj}_W \mathbf{b} \), what should you do?

Question 5 of 5

If \( \mathbf{b} \) is already in the subspace \( W \), what is the error vector \( \mathbf{e} = \mathbf{b} - \text{proj}_W \mathbf{b} \)?