01 / 21

Best Approximation
in a Subspace

Learn how to find the closest point in a subspace to any vector: the foundation of projection, least squares, and countless applications in data science.

Linear Algebra

What Is Best Approximation?

Given a vector \(\mathbf{b}\) in \(\mathbb{R}^n\) and a subspace \(W\), the best approximation to \(\mathbf{b}\) in \(W\) is the vector \(\hat{\mathbf{b}} \in W\) that is closest to \(\mathbf{b}\).

"Closest" means smallest Euclidean distance: the length \(\|\mathbf{b} - \hat{\mathbf{b}}\|\) is as small as possible.

Visual Intuition: Dropping a Perpendicular

\(W\)
\(\mathbf{b}\)
\(\hat{\mathbf{b}}\)
\(\mathbf{w}\)
shortest
longer

Why Does It Matter?

Least Squares
When \(A\mathbf{x}=\mathbf{b}\) has no solution, the best approximation gives the least-squares solution.
Data Fitting
Fitting a line or curve to noisy data points relies on projecting onto a subspace of model functions.
Signal Processing
Approximating a complex signal by a simpler one in a subspace of frequencies is a best-approximation problem.

The Best Approximation Theorem

For every \(\mathbf{w} \in W\) with \(\mathbf{w} \neq \hat{\mathbf{b}}\):

Why Orthogonality Makes It Work

\(W\)
\(\hat{\mathbf{b}}\)
\(\mathbf{b}\)
\(\mathbf{b} - \hat{\mathbf{b}}\)
\(\mathbf{w}\)
\(\mathbf{b} - \mathbf{w}\)
\(\hat{\mathbf{b}} - \mathbf{w}\)

Projection vs. Approximation

Orthogonal Projection Best Approximation
Question Decompose \(\mathbf{b}\) relative to \(W\) Find closest \(\mathbf{w}\in W\) to \(\mathbf{b}\)
Answer \(\text{proj}_W \mathbf{b}\) \(\hat{\mathbf{b}} = \text{proj}_W \mathbf{b}\)
Key property \(\mathbf{b} - \hat{\mathbf{b}} \perp W\) Minimizes \(\|\mathbf{b}-\mathbf{w}\|\)
Relationship They give the same vector: two views of one idea

Computing \(\text{proj}_W \mathbf{b}\)

If \(\{\mathbf{u}_1, \mathbf{u}_2, \dots, \mathbf{u}_p\}\) is an orthogonal basis for \(W\), then:

If the basis is orthonormal (each \(\|\mathbf{u}_i\|=1\)), this simplifies to:

How to Find the Best Approximation

1
Identify \(W\) and \(\mathbf{b}\). What is the subspace, and what vector are you approximating?
2
Get an orthogonal basis for \(W\). If the given basis is not orthogonal, apply the Gram-Schmidt process.
3
Compute each projection coefficient \(c_i = \frac{\mathbf{b}\cdot\mathbf{u}_i}{\mathbf{u}_i\cdot\mathbf{u}_i}\).
4
Sum up: \(\hat{\mathbf{b}} = c_1\mathbf{u}_1 + c_2\mathbf{u}_2 + \cdots + c_p\mathbf{u}_p\).
5
Verify: Check that \(\mathbf{b}-\hat{\mathbf{b}}\) is orthogonal to every basis vector of \(W\).

Example 1: Projection onto a Line

Find the best approximation to \(\mathbf{b} = \begin{bmatrix}1\\4\end{bmatrix}\) in \(W = \text{Span}\!\left\{\begin{bmatrix}2\\1\end{bmatrix}\right\}\).

\(x\)
\(y\)
\(W\)
\(\mathbf{u}=\begin{bmatrix}2\\1\end{bmatrix}\)
\(\mathbf{b}=\begin{bmatrix}1\\4\end{bmatrix}\)
\(\hat{\mathbf{b}}=\begin{bmatrix}12/5\\6/5\end{bmatrix}\)

Verify: Is It Really Closest?

Check that the residual \(\mathbf{b}-\hat{\mathbf{b}}\) is orthogonal to \(\mathbf{u}\):

The dot product is zero, confirming orthogonality. The approximation error is \(\|\mathbf{b}-\hat{\mathbf{b}}\| = \tfrac{7\sqrt{5}}{5}\).

Example 2: Projection onto a Plane

Find the best approximation to \(\mathbf{b}=\begin{bmatrix}1\\1\\3\end{bmatrix}\) in \(W=\text{Span}\!\left\{\mathbf{u}_1=\begin{bmatrix}1\\0\\1\end{bmatrix},\;\mathbf{u}_2=\begin{bmatrix}0\\1\\0\end{bmatrix}\right\}\).

1
\(c_1 = \dfrac{\mathbf{b}\cdot\mathbf{u}_1}{\mathbf{u}_1\cdot\mathbf{u}_1} = \dfrac{1+0+3}{1+0+1} = \dfrac{4}{2}=2\)
2
\(c_2 = \dfrac{\mathbf{b}\cdot\mathbf{u}_2}{\mathbf{u}_2\cdot\mathbf{u}_2} = \dfrac{0+1+0}{0+1+0} = 1\)
3
\(\hat{\mathbf{b}}=2\begin{bmatrix}1\\0\\1\end{bmatrix}+1\begin{bmatrix}0\\1\\0\end{bmatrix}=\begin{bmatrix}2\\1\\2\end{bmatrix}\)

Verify Example 2

✓
\((\mathbf{b}-\hat{\mathbf{b}})\cdot\mathbf{u}_1 = (-1)(1)+(0)(0)+(1)(1) = 0\) ✓
✓
\((\mathbf{b}-\hat{\mathbf{b}})\cdot\mathbf{u}_2 = (-1)(0)+(0)(1)+(1)(0) = 0\) ✓

Example 3: Non-Orthogonal Basis

Find the best approximation to \(\mathbf{b}=\begin{bmatrix}3\\-1\\2\end{bmatrix}\) in \(W = \text{Col}(A)\) where \(A = \begin{bmatrix}1 & 1\\0 & 1\\1 & 0\end{bmatrix}\).

1
\(A^T A = \begin{bmatrix}2&1\\1&2\end{bmatrix}\), \(\;A^T\mathbf{b}=\begin{bmatrix}5\\2\end{bmatrix}\)
2
Solve: \(\hat{\mathbf{x}}=\begin{bmatrix}8/3\\-1/3\end{bmatrix}\)
3
\(\hat{\mathbf{b}} = A\hat{\mathbf{x}} = \begin{bmatrix}7/3\\-1/3\\8/3\end{bmatrix}\)

Common Mistakes

✕ Using Non-Orthogonal Basis in the Formula
The projection formula with individual dot products only works for orthogonal bases. For a general basis, use the normal equations \(A^T A \hat{\mathbf{x}} = A^T\mathbf{b}\).
✕ Confusing \(\hat{\mathbf{x}}\) with \(\hat{\mathbf{b}}\)
\(\hat{\mathbf{x}}\) gives the coordinates in the basis; the best approximation vector is \(\hat{\mathbf{b}} = A\hat{\mathbf{x}}\). Don't stop at \(\hat{\mathbf{x}}\).
✕ Skipping the Verification Step
Always check that \(\mathbf{b}-\hat{\mathbf{b}}\) is orthogonal to each basis vector of \(W\). A nonzero dot product means an error in computation.

Summary: Methods at a Glance

Scenario Formula When to Use
Orthogonal basis $$\hat{\mathbf{b}}=\sum \frac{\mathbf{b}\cdot\mathbf{u}_i}{\mathbf{u}_i\cdot\mathbf{u}_i}\mathbf{u}_i$$ Basis vectors are pairwise orthogonal
Orthonormal basis $$\hat{\mathbf{b}}=\sum (\mathbf{b}\cdot\mathbf{u}_i)\mathbf{u}_i$$ Basis vectors are unit length and orthogonal
General basis \(A^T A\hat{\mathbf{x}}=A^T\mathbf{b}\), then \(\hat{\mathbf{b}}=A\hat{\mathbf{x}}\) Basis is not orthogonal
Verification \((\mathbf{b}-\hat{\mathbf{b}})\cdot\mathbf{u}_i=0\;\forall\, i\) Always: confirms correctness

Question 1 of 5

The best approximation \(\hat{\mathbf{b}}\) to a vector \(\mathbf{b}\) in a subspace \(W\) is defined as the vector in \(W\) that:

Question 2 of 5

If \(\hat{\mathbf{b}}\) is the best approximation to \(\mathbf{b}\) in \(W\), which statement is always true?

Question 3 of 5

Find the best approximation to \(\mathbf{b}=\begin{bmatrix}3\\4\end{bmatrix}\) in \(W=\text{Span}\!\left\{\begin{bmatrix}1\\1\end{bmatrix}\right\}\).

Question 4 of 5

You are given a non-orthogonal basis for \(W\) and need to find \(\hat{\mathbf{b}}\). Which approach is correct?

Question 5 of 5

Let \(W\) be a subspace and \(\hat{\mathbf{b}} = \text{proj}_W\mathbf{b}\). If \(\mathbf{b}\) is already in \(W\), what is the approximation error \(\|\mathbf{b}-\hat{\mathbf{b}}\|\)?