Best Approximation - Linear Algebra

SECTION 1

Historical Context & Motivation

Imagine you need to draw a straight line through a scatter of data points, but no single line passes through every point. What do you do? You find the line that comes as close as possible to all the points. This idea — getting the best answer when a perfect answer is impossible — is at the heart of best approximation. For centuries, mathematicians and scientists have tackled problems where exact solutions simply don't exist. Instead, they looked for the closest match, and this search shaped entire fields of mathematics.

1805 • Legendre & Least Squares

Adrien-Marie Legendre published the method of least squares, a technique for fitting the best line to data. This gave scientists a systematic way to handle measurement errors.

1807 • Fourier Series

Joseph Fourier showed that complicated waves could be approximated by adding together simple sine and cosine waves. This was an early form of best approximation in function spaces.

1907 • Hilbert Spaces Formalized

David Hilbert introduced abstract inner product spaces, providing a rigorous framework for defining "closeness" and proving that best approximations always exist in complete spaces.

1936 • Orthogonal Projection Theorem

John von Neumann proved the projection theorem, which guarantees that the best approximation to any vector from a closed subspace is the orthogonal projection. This became a cornerstone of modern linear algebra.

The central question that best approximation answers is: given a vector (or point, or signal) and a subspace where we must find our answer, which element of that subspace is the closest to our target? As we'll see, the answer involves a beautiful geometric idea called orthogonal projection.

SECTION 2

Core Principles & Definitions

Before we can find the best approximation, we need a few building blocks. These ideas let us measure distances and angles in vector spaces, much like a ruler and protractor work in everyday geometry.

Inner Product

An inner product (also called a dot product in ℝⁿ) is a rule that takes two vectors and returns a number. It tells us how much two vectors "agree" or point in the same direction. Written as ⟨u, v⟩.

Norm (Length)

The norm of a vector, written ‖v‖, measures its length. It's defined using the inner product: ‖v‖ = √⟨v, v⟩. We use norms to measure distances between vectors.

Subspace

A subspace is a flat subset of a vector space that passes through the origin. Think of a plane through the origin in 3D space, or a line through the origin in 2D.

Orthogonality

Two vectors are orthogonal (perpendicular) when their inner product equals zero: ⟨u, v⟩ = 0. This is the key geometric condition behind best approximation.

Orthogonal Projection

The orthogonal projection of a vector onto a subspace is the "shadow" it casts straight down onto that subspace. This shadow is the best approximation.

✦

KEY TAKEAWAY

Think of standing under a streetlight at night. Your shadow on the ground is the closest flat version of you. The best approximation works the same way: you "drop a perpendicular" from your target vector straight down onto the subspace, and where it lands is the closest point in that subspace. The error — the gap between you and your shadow — is perfectly perpendicular to the ground.

SECTION 3

Visual Explanation

The diagram below shows the geometry of best approximation in three dimensions. A vector b lives in 3D space, but the subspace W (shown as a shaded plane) is only two-dimensional. The best approximation b̂ is the point on W that is closest to b. The error vector b − b̂ is perpendicular to the entire subspace.

The vector b (purple) is projected onto the subspace W. The projection b̂ (cyan) is the best approximation, and the error b − b̂ (pink dashed) is perpendicular to the plane, as shown by the 90° angle marker.

Notice the right-angle symbol where the error vector meets the subspace. This perpendicularity is not a coincidence — it's the defining property that makes b̂ the closest point in W to the target b. If you moved to any other point on the plane, you'd be farther away. This is the same reason the shortest distance from a point to a line is always the perpendicular distance, an idea you may remember from geometry class.

SECTION 4

Mathematical Framework

Now let's write down the math. Suppose W is a subspace of an inner product space V, and b is a vector in V that may or may not be in W. We want to find the vector b̂ in W that minimizes the distance ‖b − b̂‖.

BEST APPROXIMATION THEOREM

b̂ = proj_W(b) minimizes ‖b − w‖ for all w in W

b̂ = the best approximation (the orthogonal projection of b onto W). w = any vector in the subspace W. ‖b − w‖ = the distance between b and w.

When the subspace W has an orthogonal basis {u₁, u₂, …, uₖ}, computing the projection becomes straightforward. You project b onto each basis vector individually and then add the results together.

PROJECTION ONTO AN ORTHOGONAL BASIS

b̂ = (⟨b, u₁⟩ / ⟨u₁, u₁⟩) × u₁ + (⟨b, u₂⟩ / ⟨u₂, u₂⟩) × u₂ + … + (⟨b, uₖ⟩ / ⟨uₖ, uₖ⟩) × uₖ

⟨b, uᵢ⟩ = the inner product (dot product) of b and uᵢ. Each fraction ⟨b, uᵢ⟩ / ⟨uᵢ, uᵢ⟩ is a scalar that tells you how much of b lies in the direction of uᵢ.

If the basis is orthonormal (meaning each basis vector has length 1), the formula simplifies even further because ⟨uᵢ, uᵢ⟩ = 1 for every basis vector.

PROJECTION (ORTHONORMAL BASIS)

b̂ = ⟨b, u₁⟩ × u₁ + ⟨b, u₂⟩ × u₂ + … + ⟨b, uₖ⟩ × uₖ

Each coefficient ⟨b, uᵢ⟩ is called a Fourier coefficient. It measures how much b "agrees with" the direction uᵢ.

ORTHOGONALITY CONDITION (ERROR)

⟨b − b̂, w⟩ = 0 for every w in W

The error vector b − b̂ is orthogonal (perpendicular) to every vector in W. This condition is what guarantees b̂ is the closest point.

SECTION 5

Step-by-Step Projection Process

Let's visualize the step-by-step process of finding a best approximation. The diagram below breaks down how we decompose a vector b into a part that lies in the subspace (the projection b̂) and a part that is perpendicular to it (the error). In this example, W is a one-dimensional subspace — simply a line through the origin in 2D.

In 2D, projecting vector b (purple) onto the subspace W (amber line) gives the best approximation b̂ (green). The error vector (pink dashed) is perpendicular to W, confirmed by the 90° marker.

The Projection Algorithm

Step 1 — Get an orthogonal basis. If the subspace W doesn't already have an orthogonal basis, use the Gram–Schmidt process to create one.
Step 2 — Compute each projection. For each basis vector uᵢ, calculate the scalar (⟨b, uᵢ⟩ / ⟨uᵢ, uᵢ⟩) and multiply it by uᵢ.
Step 3 — Add them up. Sum all the individual projections to get b̂ = proj_W(b).
Step 4 — Find the error. Compute the error vector as b − b̂. Its length ‖b − b̂‖ is the distance from b to the subspace.

SECTION 6

Worked Example

Let's find the best approximation to a vector b from a subspace W using a concrete numerical example.

Find the Best Approximation in ℝ³

1
Step 1 — Identify Given Values

We have b = (1, 2, 3) in ℝ³. The subspace W is spanned by two orthogonal vectors: u₁ = (1, 0, 1) and u₂ = (0, 1, 0). We want to find b̂ = proj_W(b), the vector in W closest to b.

b = (1, 2, 3), u₁ = (1, 0, 1), u₂ = (0, 1, 0)
2
Step 2 — Verify Orthogonality

Check that u₁ and u₂ are orthogonal: ⟨u₁, u₂⟩ = (1)(0) + (0)(1) + (1)(0) = 0. They are orthogonal, so we can use the projection formula directly without Gram–Schmidt.

⟨u₁, u₂⟩ = 0 ✓ Orthogonal
3
Step 3 — Project b onto u₁

Compute ⟨b, u₁⟩ = (1)(1) + (2)(0) + (3)(1) = 1 + 0 + 3 = 4. Compute ⟨u₁, u₁⟩ = (1)(1) + (0)(0) + (1)(1) = 2. So the projection onto u₁ is (4/2) × u₁ = 2 × (1, 0, 1) = (2, 0, 2).

proj_{u₁}(b) = (2, 0, 2)
4
Step 4 — Project b onto u₂

Compute ⟨b, u₂⟩ = (1)(0) + (2)(1) + (3)(0) = 2. Compute ⟨u₂, u₂⟩ = (0)(0) + (1)(1) + (0)(0) = 1. So the projection onto u₂ is (2/1) × u₂ = 2 × (0, 1, 0) = (0, 2, 0).

proj_{u₂}(b) = (0, 2, 0)
5
Step 5 — Add the Projections

The best approximation is b̂ = proj_{u₁}(b) + proj_{u₂}(b) = (2, 0, 2) + (0, 2, 0) = (2, 2, 2).

b̂ = (2, 2, 2)
6
Step 6 — Compute the Error

The error vector is b − b̂ = (1, 2, 3) − (2, 2, 2) = (−1, 0, 1). Let's verify it's orthogonal to W: ⟨(−1, 0, 1), u₁⟩ = −1 + 0 + 1 = 0 ✓ and ⟨(−1, 0, 1), u₂⟩ = 0 + 0 + 0 = 0 ✓. The distance is ‖b − b̂‖ = √((−1)² + 0² + 1²) = √2 ≈ 1.41.

Error = (−1, 0, 1), Distance = √2 ≈ 1.41

💡

Verification Tip

Always check that the error vector is orthogonal to every basis vector of W. If any dot product is not zero, go back and recheck your arithmetic. This orthogonality check is your built-in answer verifier!

SECTION 7

Strengths & Limitations

Best approximation via orthogonal projection is a powerful tool, but like any technique, it has situations where it shines and situations where you need to be careful.

Aspect	Strengths	Limitations
Uniqueness	The best approximation is always unique — there is exactly one closest point in the subspace.	Only guaranteed in complete inner product spaces with closed subspaces.
Computation	Very fast with an orthogonal or orthonormal basis — just dot products and scalar multiplication.	If the basis is not orthogonal, you first need the Gram–Schmidt process, which adds extra steps.
Geometric Intuition	The perpendicularity condition gives strong visual and geometric understanding.	In very high dimensions, visualizing the geometry becomes impossible (but the math still works).
Error Control	You can calculate the exact error ‖b − b̂‖, so you know how good your approximation is.	The error can only be reduced by enlarging the subspace W — you cannot improve within the same subspace.

Strengths and limitations of best approximation via orthogonal projection

✦

KEY TAKEAWAY

Best approximation is like using a GPS that always finds the closest road to your location. It may not put you exactly where you want to go, but it guarantees you'll be on the nearest road. The only way to get closer to your true destination is to build more roads (expand the subspace).

SECTION 8

Connection to Advanced Theory

Best approximation is not just an isolated technique. It connects to many powerful ideas in mathematics, science, and engineering. Here's how the concept you just learned relates to more advanced topics.

This Lesson	Advanced Extension	Where It Appears
Projection onto a subspace with orthogonal basis	Least-squares regression — finding the best-fit line through data points	Statistics, data science, machine learning
Orthogonal projection in ℝⁿ	Fourier series — approximating signals using sine and cosine functions	Signal processing, audio compression, music technology
Minimizing ‖b − b̂‖	QR factorization — a matrix decomposition that makes projection computationally efficient	Numerical analysis, computer graphics, engineering simulations
Error vector is perpendicular	Hilbert space theory — projection theorem in infinite-dimensional spaces	Quantum mechanics, functional analysis, partial differential equations

How best approximation connects to advanced topics

The idea you learned here — projecting onto a subspace to find the closest match — is the same idea behind fitting a line to data in statistics, compressing music files on your phone, and even how quantum physics describes the state of a particle. The language gets more sophisticated, but the core geometry of dropping a perpendicular stays exactly the same.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Explain in your own words why the error vector b − b̂ must be perpendicular to the subspace W. What would go wrong if it weren't perpendicular?

Show answer

If the error vector were not perpendicular to W, it would have a component that lies along W. You could then "slide" b̂ in the direction of that component to get even closer to b, meaning b̂ was not actually the closest point. The perpendicularity guarantees that no further improvement within W is possible — every direction you could move in the subspace would only take you farther from b. Think of it like standing directly below a chandelier: if you're not directly underneath, you could step closer, so you haven't found the shortest distance yet.

Question 2: BASIC CALCULATION

Let b = (3, 4) and u = (1, 0). The subspace W is the span of u (the x-axis). Find the best approximation b̂ = proj_W(b) and the error ‖b − b̂‖.

Show answer

Using the projection formula: b̂ = (⟨b, u⟩ / ⟨u, u⟩) × u. We compute ⟨b, u⟩ = (3)(1) + (4)(0) = 3, and ⟨u, u⟩ = 1. So b̂ = (3/1) × (1, 0) = (3, 0). The error vector is b − b̂ = (3, 4) − (3, 0) = (0, 4). The distance is ‖(0, 4)‖ = 4. This makes geometric sense: projecting (3, 4) onto the x-axis gives (3, 0), and the vertical distance is 4.

Question 3: INTERMEDIATE

Let b = (1, 3, 5) and W = span{u₁, u₂} where u₁ = (1, 1, 0) and u₂ = (−1, 1, 0). Verify that u₁ and u₂ are orthogonal, then find the best approximation b̂ and the approximation error ‖b − b̂‖.

Show answer

First, check orthogonality: ⟨u₁, u₂⟩ = (1)(−1) + (1)(1) + (0)(0) = −1 + 1 + 0 = 0. They are orthogonal. Now project b onto each basis vector. For u₁: ⟨b, u₁⟩ = 1 + 3 + 0 = 4, ⟨u₁, u₁⟩ = 1 + 1 + 0 = 2, so proj_{u₁}(b) = (4/2)(1, 1, 0) = (2, 2, 0). For u₂: ⟨b, u₂⟩ = −1 + 3 + 0 = 2, ⟨u₂, u₂⟩ = 1 + 1 + 0 = 2, so proj_{u₂}(b) = (2/2)(−1, 1, 0) = (−1, 1, 0). Therefore b̂ = (2, 2, 0) + (−1, 1, 0) = (1, 3, 0). The error is b − b̂ = (0, 0, 5), and ‖b − b̂‖ = 5. You can verify: ⟨(0, 0, 5), u₁⟩ = 0 and ⟨(0, 0, 5), u₂⟩ = 0.

Question 4: APPLIED

A student collects data points (1, 2), (2, 3), and (3, 6) and wants to fit them with a line y = c₁ through the origin. This means solving the inconsistent system [1; 2; 3]c₁ = [2; 3; 6], where a = (1, 2, 3) and b = (2, 3, 6). Use best approximation to find the value of c₁ that gives the best-fit line.

Show answer

The subspace W is the span of a = (1, 2, 3). We need b̂ = proj_W(b) where b = (2, 3, 6). Using the projection formula: ⟨b, a⟩ = (2)(1) + (3)(2) + (6)(3) = 2 + 6 + 18 = 26. ⟨a, a⟩ = 1 + 4 + 9 = 14. So c₁ = ⟨b, a⟩ / ⟨a, a⟩ = 26/14 = 13/7 ≈ 1.857. The best-fit line through the origin is y = (13/7)x. The best approximation vector is b̂ = (13/7)(1, 2, 3) = (13/7, 26/7, 39/7). This is the least-squares solution to the system.

Question 5: CRITICAL THINKING

Suppose W₁ ⊂ W₂ are two subspaces with W₁ contained inside W₂. Let b̂₁ = proj_{W₁}(b) and b̂₂ = proj_{W₂}(b). Which approximation error is smaller — ‖b − b̂₁‖ or ‖b − b̂₂‖? Explain your reasoning carefully and discuss whether adding more basis vectors to a subspace always helps.

Show answer

The error ‖b − b̂₂‖ ≤ ‖b − b̂₁‖ always holds. Here's why: since W₁ ⊂ W₂, every vector in W₁ is also in W₂. The best approximation b̂₂ minimizes ‖b − w‖ over all w in W₂, which includes all of W₁ as candidates. Since b̂₂ is chosen from a larger set, it can do at least as well as b̂₁, and often strictly better. Adding more basis vectors to a subspace always reduces (or maintains) the approximation error. However, the improvement may be tiny if the new basis vectors happen to be nearly orthogonal to the error b − b̂₁. In practice, the most useful basis vectors to add are those that align with the remaining error. This insight is the foundation of techniques like principal component analysis (PCA) in data science.

SUMMARY

Summary

The best approximation to a vector b from a subspace W is the orthogonal projection of b onto W, written b̂ = proj_W(b). It is the unique vector in W that minimizes the distance ‖b − b̂‖. The defining property is the orthogonality condition: the error vector b − b̂ is perpendicular to every vector in W, meaning ⟨b − b̂, w⟩ = 0 for all w in W.

To compute b̂, use an orthogonal basis {u₁, u₂, …, uₖ} for W and apply the formula b̂ = Σ (⟨b, uᵢ⟩ / ⟨uᵢ, uᵢ⟩) × uᵢ. With an orthonormal basis, this simplifies to b̂ = Σ ⟨b, uᵢ⟩ × uᵢ. Best approximation is the foundation of least-squares regression, Fourier analysis, and countless applications in science and engineering.