Projections & Decompositions — Projections and Decompositions - Linear Algebra

SECTION 1

Historical Context & Motivation

Have you ever shined a flashlight straight down onto a table and noticed how its shadow falls? That shadow is a kind of projection — it takes something three-dimensional and flattens it onto a surface. In math, the same idea lets us break a complicated vector into simpler parts. This concept has a long history, stretching from ancient Greek geometry all the way to today's computer graphics and data science.

~300 BC • Euclid's Elements

The ancient Greek mathematician Euclid studied perpendicular lines and right angles. His work laid the foundation for understanding how shapes can be decomposed into simpler parts.

1799 • Least-Squares Method

Carl Friedrich Gauss and Adrien-Marie Legendre developed the method of least squares for fitting lines to data. This technique relies on projecting data points onto a line to minimize errors.

1907 • Inner Product Spaces

Mathematicians like David Hilbert formalized the concept of inner product spaces, giving us a precise language for angles, lengths, and projections in any number of dimensions.

1936 • Gram–Schmidt Process

Jørgen Pedersen Gram and Erhard Schmidt refined the process for creating orthogonal (perpendicular) sets of vectors. This process uses projections repeatedly to build a clean, right-angle coordinate system.

2000s • Modern Applications

Projections and decompositions now power technologies like image compression (JPEG), search engines, facial recognition, and machine learning — all of which rely on breaking data into simpler components.

The big question these thinkers were trying to answer is: how can we take something complicated and split it into simple, independent pieces? Projections and decompositions give us exactly that power.

SECTION 2

Core Principles & Definitions

Before we dive into formulas, let's build a clear picture of the key ideas. Everything in this lesson rests on a few core principles that connect geometry (shapes and angles) to algebra (numbers and equations).

Vectors

A vector is a quantity with both direction and magnitude (size). Think of it as an arrow pointing from one place to another. Vectors can live in 2D, 3D, or even higher dimensions.

Inner Product (Dot Product)

The inner product (also called the dot product) measures how much two vectors point in the same direction. If two vectors are perpendicular, their dot product equals zero.

Orthogonality

Two vectors are orthogonal when they meet at a right angle (90°). Orthogonal means "perpendicular" in the language of linear algebra. This is the key property that makes decompositions clean and unique.

Projection

A projection of one vector onto another is the "shadow" that falls along the target direction. It tells you how much of the original vector lies along that direction.

Decomposition

A decomposition splits a vector into two or more pieces that add back together to give the original. An orthogonal decomposition uses perpendicular pieces, making the split as clean as possible.

✦

KEY TAKEAWAY

Imagine you're carrying a heavy suitcase up a ramp. Part of your effort pushes the suitcase forward along the ramp, and part fights gravity pulling it straight down. A projection separates these two parts, and the decomposition is the act of splitting the total force into the "along the ramp" piece and the "perpendicular to the ramp" piece. Together, those two pieces perfectly reconstruct the original force.

SECTION 3

Visual Explanation

The diagram below shows the most important picture in this lesson. We have a vector v (the blue arrow) and a direction vector u (the violet arrow along the x-axis). The projection of v onto u is the cyan dashed arrow, and the leftover piece — the part perpendicular to u — is the pink dashed arrow. Notice the right-angle square where they meet!

The blue arrow v is decomposed into a projection along u (cyan dashed) and a perpendicular remainder (pink dashed). The yellow square confirms the 90° angle between them.

This picture captures the entire idea. No matter what vector v you start with, you can always drop a perpendicular "shadow" onto any direction u. The shadow is the projection, and the leftover piece points straight up from u. Adding the shadow and the leftover piece always gives you back the original vector v.

SECTION 4

Mathematical Framework

Now let's put numbers to the picture. The formulas below use the dot product to measure how much one vector "aligns" with another.

DOT PRODUCT (INNER PRODUCT)

u · v = u₁v₁ + u₂v₂ + … + uₙvₙ

Multiply matching components, then add them all up. If u = (2, 3) and v = (4, 1), then u · v = 2×4 + 3×1 = 11.

SCALAR PROJECTION

comp_u v = (u · v) / ‖u‖

This gives a single number (a scalar) telling you how far the shadow of v reaches along u. Here ‖u‖ means the length of u, calculated as √(u₁² + u₂² + …).

VECTOR PROJECTION

proj_u v = [(u · v) / (u · u)] × u

This gives you the actual vector (the cyan arrow in the diagram). Divide the dot product of u and v by the dot product of u with itself, then multiply that fraction by the vector u. The result is the component of v that lies along u.

ORTHOGONAL DECOMPOSITION

v = proj_u v + (v − proj_u v)

The first piece (proj_u v) lies along u. The second piece (v − proj_u v) is perpendicular to u. Together they always reconstruct the original v. This is the orthogonal decomposition of v with respect to u.

💡

Why u · u instead of ‖u‖²?

They're actually the same thing! Since ‖u‖ = √(u · u), squaring it gives ‖u‖² = u · u. Writing it as u · u avoids the square root entirely, which keeps the arithmetic cleaner.

SECTION 5

Types of Decompositions

The basic idea — project onto one direction — can be extended in powerful ways. When you have multiple perpendicular directions, you can project onto each one separately and add the pieces together. This is the foundation of several important decomposition methods.

Left panel: projecting onto one direction leaves a perpendicular remainder. Right panel: when we have two orthogonal basis vectors (e₁ and e₂), we can project onto both, and the two projections add up perfectly to recreate v with no leftover.

Decomposition Type	What It Does	When You Use It
Single-vector projection	Projects v onto one direction u; splits v into a piece along u and a perpendicular leftover.	Finding the component of a force in one direction, or the closest point on a line to a given point.
Orthogonal basis decomposition	Projects v onto each vector in a set of perpendicular basis vectors; the pieces add up to v exactly.	Expressing any vector in standard x-y-z coordinates, or in any custom perpendicular coordinate system.
Projection onto a subspace	Projects v onto an entire plane (or higher-dimensional subspace), finding the closest point in that subspace to v.	Least-squares fitting (best-fit lines), image compression, and removing noise from data.

Common types of projections and decompositions

SECTION 6

Worked Example

Let's work through a complete example with real numbers. Suppose v = (5, 6) and u = (4, 0). We want to project v onto u and then decompose v into its "along u" and "perpendicular to u" components.

Project v = (5, 6) onto u = (4, 0)

1
Step 1 — Compute the dot product u · v

Multiply matching components and add: u · v = 4 × 5 + 0 × 6 = 20 + 0 = 20.

u · v = 20
2
Step 2 — Compute the dot product u · u

This measures the "length squared" of u: u · u = 4 × 4 + 0 × 0 = 16 + 0 = 16.

u · u = 16
3
Step 3 — Find the scaling factor

Divide: (u · v) / (u · u) = 20 / 16 = 5/4 (or 1.25). This tells us the projection is 1.25 times the vector u.

Scaling factor = 5/4
4
Step 4 — Compute the projection vector

Multiply u by the scaling factor: proj_u v = (5/4) × (4, 0) = (5/4 × 4, 5/4 × 0) = (5, 0). This is the component of v that lies along u (the cyan arrow in our diagram).

proj_u v = (5, 0)
5
Step 5 — Find the perpendicular component

Subtract: v − proj_u v = (5, 6) − (5, 0) = (0, 6). This is the part of v that is perpendicular to u (the pink arrow).

Perpendicular part = (0, 6)
6
Step 6 — Verify the decomposition

Check: (5, 0) + (0, 6) = (5, 6) = v ✓ Also check orthogonality: (5, 0) · (0, 6) = 5×0 + 0×6 = 0 ✓ A dot product of zero confirms the two pieces are perpendicular!

v = (5, 0) + (0, 6) — orthogonal decomposition confirmed!

📝

Pattern to notice

In this example, u pointed along the x-axis, so the projection simply kept the x-component and dropped the y-component. For a direction that isn't along an axis, the formula does the same thing — it finds the component along u — but the arithmetic is a bit more involved.

SECTION 7

Strengths, Limitations & Comparisons

Projections and decompositions are incredibly useful, but like any tool, they work best in certain situations. Understanding both their strengths and limitations helps you choose the right approach for a given problem.

Strengths	Limitations
Gives a unique, clean split when the directions are orthogonal (perpendicular).	If the directions are NOT orthogonal, the decomposition is not unique and harder to interpret.
Works in any number of dimensions — from 2D arrows to 1,000-dimensional data.	In very high dimensions, computing projections can be slow without efficient algorithms.
The projection is always the closest point on the target line or plane (minimizes distance).	Requires the dot product (inner product) to be defined — which it always is for standard vectors but may need care in abstract spaces.
Easy to visualize in 2D and 3D, building geometric intuition.	Geometric intuition can be misleading in very high dimensions where our 3D imagination breaks down.

Strengths vs. Limitations of Orthogonal Projections

✦

KEY TAKEAWAY

Think of orthogonal projection as a GPS that always finds the shortest route from a point to a road. The road is the direction (or subspace) you're projecting onto, and the shortest route is always a straight, perpendicular drop. This "shortest distance" property is why projections are at the heart of best-fit lines and data analysis.

SECTION 8

Connection to Advanced Theory

The projection ideas you've learned here form the stepping stones to some of the most powerful tools in mathematics and science. Let's peek at where they lead.

What You Learned	Where It Leads	Why It Matters
Projecting onto one vector	Gram–Schmidt process — repeatedly project to build an entire orthogonal basis from any set of vectors.	Creates clean coordinate systems for solving systems of equations.
Orthogonal decomposition	QR factorization — a way to factor any matrix into an orthogonal part and an upper triangular part.	Used by computers to solve large systems quickly and accurately.
Projection onto a subspace	Least-squares regression — finding the best-fit line (or curve) through a cloud of data points.	Powers predictions in statistics, machine learning, and science experiments.
Decomposing into perpendicular pieces	Singular Value Decomposition (SVD) — splits any matrix into orthogonal directions ranked by importance.	Drives image compression, recommendation systems (like Netflix), and data science.

From basic projections to advanced techniques

Every one of these advanced tools is built from the same core move: project, subtract, repeat. If you understand the projection formula and the idea of orthogonal decomposition, you've already grasped the engine that powers all of them.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

In your own words, explain why the dot product of a vector's projection (along some direction u) and its perpendicular remainder must always equal zero.

Show answer

The whole point of the decomposition is to split the original vector into two pieces that are at right angles (90°) to each other. By definition, two vectors that meet at a right angle are orthogonal, and the dot product of orthogonal vectors is zero. The projection formula is specifically designed so that the leftover piece (v − proj_u v) has no component along u. Since it has no component along u, the dot product with any vector along u must vanish.

Question 2: BASIC CALCULATION

Let v = (3, 4) and u = (1, 0). Find proj_u v and the perpendicular component.

Show answer

First, compute u · v = 1×3 + 0×4 = 3. Next, u · u = 1×1 + 0×0 = 1. The scaling factor is 3/1 = 3. So proj_u v = 3 × (1, 0) = (3, 0). The perpendicular part is v − proj_u v = (3, 4) − (3, 0) = (0, 4). Check: (3, 0) · (0, 4) = 0 ✓. Since u points along the x-axis, the projection simply extracts the x-component of v.

Question 3: INTERMEDIATE

Let v = (2, 8) and u = (3, 4). Compute proj_u v and verify that the remainder is perpendicular to u.

Show answer

Compute u · v = 3×2 + 4×8 = 6 + 32 = 38. Compute u · u = 3×3 + 4×4 = 9 + 16 = 25. The scaling factor is 38/25. So proj_u v = (38/25) × (3, 4) = (114/25, 152/25) = (4.56, 6.08). The perpendicular part is v − proj_u v = (2 − 4.56, 8 − 6.08) = (−2.56, 1.92). Verify: u · (perpendicular part) = 3×(−2.56) + 4×(1.92) = −7.68 + 7.68 = 0 ✓. The zero dot product confirms orthogonality.

Question 4: APPLIED

A boat is traveling with velocity v = (6, 2) km/h. The current flows in the direction u = (1, 1). How much of the boat's speed is in the direction of the current, and how much is perpendicular to it?

Show answer

Find proj_u v. u · v = 1×6 + 1×2 = 8. u · u = 1×1 + 1×1 = 2. Scaling factor = 8/2 = 4. proj_u v = 4 × (1, 1) = (4, 4). The speed along the current is ‖(4, 4)‖ = √(16 + 16) = √32 ≈ 5.66 km/h. The perpendicular component is v − (4, 4) = (2, −2), with magnitude √(4 + 4) = √8 ≈ 2.83 km/h. This means about 5.66 km/h of the boat's motion goes with/against the current, and about 2.83 km/h goes sideways across it.

Question 5: CRITICAL THINKING

Suppose you project a vector v onto a vector u and get proj_u v = v (the projection equals the original vector). What does this tell you geometrically? What if proj_u v = (0, 0)?

Show answer

If proj_u v = v, then the perpendicular component v − proj_u v = v − v = (0, 0), the zero vector. This means v has no component perpendicular to u, so v must lie entirely along the direction of u. Geometrically, v and u are parallel (v is a scalar multiple of u). On the other hand, if proj_u v = (0, 0), then the scaling factor (u · v)/(u · u) must be zero, meaning u · v = 0. This means v is entirely perpendicular to u — they make a 90° angle. These two extreme cases bracket all possibilities: every vector is somewhere between perfectly parallel and perfectly perpendicular to u.

SUMMARY

Lesson Summary

Summary

In this lesson, you learned that a projection is the "shadow" of one vector onto another direction, calculated using the formula proj_u v = [(u · v)/(u · u)] × u. An orthogonal decomposition splits any vector v into a piece along u and a piece perpendicular to u, so that v = proj_u v + (v − proj_u v). The dot product (inner product) is the engine that drives this calculation, measuring how much two vectors align.

The two perpendicular pieces always satisfy a key test: their dot product equals zero, confirming orthogonality. This idea extends to projecting onto entire subspaces and underpins advanced techniques like the Gram–Schmidt process, least-squares regression, and Singular Value Decomposition. Whenever you need to break a complex quantity into independent, perpendicular parts, projection and decomposition are your go-to tools.