Singular Value Decomposition (SVD)

SECTION 1

Historical Context & Motivation

Imagine you have a giant table of numbers — maybe test scores for hundreds of students across dozens of subjects. How do you find the most important patterns hiding in all that data? This is the kind of question that led mathematicians to develop the Singular Value Decomposition, or SVD for short. SVD is a way of breaking any matrix (a rectangular grid of numbers) into three simpler pieces. Those pieces reveal what the matrix is really doing under the hood.

The idea didn't appear overnight. It grew over more than a century as mathematicians tackled problems in geometry, physics, and data analysis. Let's walk through the key moments.

1873–1874

Beltrami & Jordan

Italian mathematician Eugenio Beltrami and French mathematician Camille Jordan independently studied how certain square matrices could be decomposed. Their work planted the seeds for what would become SVD.

1936

Eckart–Young Theorem

Carl Eckart and Gale Young proved that SVD gives the best possible low-rank approximation of a matrix. In simpler terms, if you want a smaller, simpler version of your data, SVD finds the closest one.

1965

Golub & Kahan Algorithm

Gene Golub and William Kahan published a reliable computer algorithm for calculating SVD, making it practical for real-world use on early digital computers.

2006–Present

SVD Powers the Modern World

SVD became the backbone of recommendation engines (like Netflix suggestions), image compression, search engines, and machine learning. Today it is one of the most widely used tools in data science.

The central question SVD answers is this: What are the most important directions and magnitudes hidden inside a matrix? Understanding those directions lets us simplify data, remove noise, and see patterns that would otherwise be invisible.

SECTION 2

Core Principles & Definitions

Before we dive into the math, let's build a mental picture. Every matrix describes a transformation — it takes input vectors (arrows with direction and length) and stretches, rotates, or flips them to produce output vectors. SVD breaks that transformation into three clean steps.

Rotation (or Reflection) — Vᵀ

First, the input space is rotated (or reflected) so that the important directions line up with the coordinate axes. The matrix Vᵀ handles this step.

Scaling — Σ

Next, each aligned direction is stretched or shrunk by a specific amount. These stretch factors are the singular values, collected in the diagonal matrix Σ (the Greek letter Sigma).

Rotation (or Reflection) — U

Finally, the stretched result is rotated (or reflected) again into the output space. The matrix U performs this last rotation.

Any Matrix Works

Unlike some decompositions that only work on square or symmetric matrices, SVD works on every matrix — tall, wide, square, or even one with all zeros.

✦ KEY TAKEAWAY

Think of SVD like analyzing a dance move. Every complicated move (matrix) can be broken down into: turn your body (Vᵀ), stretch your arms by different amounts (Σ), and turn again to face the audience (U). No matter how wild the dance looks, those three simple steps always recreate it perfectly.

SECTION 3

Visual Explanation

The diagram below shows what SVD does to a unit circle (a circle with radius 1) in two dimensions. A matrix transforms that circle into an ellipse. SVD reveals the rotation angles and the stretch factors that describe that transformation.

On the left, a unit circle with two perpendicular direction vectors v₁ and v₂. The matrix stretches them by the singular values σ₁ = 3 and σ₂ = 1.2, then rotates them into the directions u₁ and u₂. The circle becomes an ellipse whose axes are the singular values.

Notice how the dashed circle on the left becomes a solid ellipse on the right. The long axis of the ellipse has length equal to σ₁ (the largest singular value), and the short axis has length σ₂. A big singular value means the matrix stretches a lot in that direction. A small one means it barely stretches at all — or even squishes.

SECTION 4

Mathematical Framework

Now let's write SVD as an equation. If A is any m × n matrix (m rows and n columns), SVD says we can always write it as a product of three matrices.

THE SVD FORMULA

A = U × Σ × Vᵀ

A = the original m × n matrix. U = an m × m orthogonal matrix (its columns are the left singular vectors). Σ = an m × n diagonal matrix (the singular values σ₁ ≥ σ₂ ≥ … ≥ 0 sit on the diagonal). Vᵀ = the transpose of an n × n orthogonal matrix (its rows are the right singular vectors).

The word orthogonal means the columns of U (and the columns of V) are all perpendicular to each other and each has length 1. Think of them as perfectly spaced direction arrows. The word diagonal means Σ has numbers only along its main diagonal — everything else is zero.

SINGULAR VALUES

σ₁ ≥ σ₂ ≥ σ₃ ≥ … ≥ σᵣ > 0, σᵣ₊₁ = … = 0

The singular values are always non-negative and listed from largest to smallest. The number r of non-zero singular values equals the rank of the matrix — roughly, how many truly independent pieces of information it contains.

OUTER-PRODUCT EXPANSION

A = σ₁ u₁ vᵀ₁ + σ₂ u₂ vᵀ₂ + … + σᵣ uᵣ vᵀᵣ

Each term σᵢ uᵢ vᵀᵢ is a rank-1 matrix (a simple building block). The first term captures the most important pattern in A, the second captures the next most important, and so on. Keeping only the first few terms gives a useful low-rank approximation.

💡 Why 'Singular'?

The word 'singular' here does not mean 'strange.' In linear algebra, a matrix is called singular when it squishes space down to a lower dimension (its determinant is zero). The singular values tell you exactly how much squishing happens in each direction. If a singular value is zero, that direction gets completely flattened.

SECTION 5

Detailed Breakdown — The Three Matrices

Let's look more closely at each of the three matrices U, Σ, and Vᵀ and understand what they contain. The diagram below shows their sizes and internal structure for a 3 × 2 matrix A.

For a 3 × 2 matrix A, U is 3 × 3 (cyan), Σ is 3 × 2 (amber) with singular values on the diagonal, and Vᵀ is 2 × 2 (pink). The shaded columns of U and rows of Vᵀ represent the singular vectors.

Summary of the three SVD matrices

Matrix	Size	What It Contains	Role
U	m × m	Left singular vectors (columns)	Output rotation/reflection
Σ	m × n	Singular values on the diagonal	Scaling (stretch/shrink)
Vᵀ	n × n	Right singular vectors (rows)	Input rotation/reflection

SECTION 6

Worked Example

Let's work through the SVD of a simple 2 × 2 matrix. We won't compute every detail from scratch (that involves eigenvalues, which you may not have seen yet), but we'll verify that the factorization works and interpret the result.

SVD of a 2 × 2 Matrix

Step 1 — State the MatrixSuppose A = [[3, 0], [0, 1]]. This is a diagonal matrix that stretches the x-direction by 3 and leaves the y-direction unchanged.

A = [[3, 0], [0, 1]]

Step 2 — Identify U, Σ, and VᵀBecause A is already diagonal with positive entries, the SVD is especially clean. No rotation is needed at all, so U and V are both the 2 × 2 identity matrix (1s on the diagonal, 0s elsewhere). The singular values are simply the diagonal entries of A, listed from largest to smallest.

U = [[1, 0], [0, 1]], Σ = [[3, 0], [0, 1]], Vᵀ = [[1, 0], [0, 1]]

Step 3 — Verify the ProductMultiply: U × Σ × Vᵀ = [[1,0],[0,1]] × [[3,0],[0,1]] × [[1,0],[0,1]]. Since multiplying by the identity doesn't change anything, the product equals [[3,0],[0,1]] = A. ✓

U × Σ × Vᵀ = A ✓

Step 4 — Try a Harder MatrixNow consider A = [[2, 1], [1, 2]]. Through eigenvalue calculations (which you'll learn in a later course), the SVD turns out to be: σ₁ = 3, σ₂ = 1, with U and V involving 45° rotations. The matrix u₁ = [1/√2, 1/√2]ᵀ and u₂ = [−1/√2, 1/√2]ᵀ, and V has the same columns.

σ₁ = 3, σ₂ = 1 → the matrix stretches space by 3 in one direction and by 1 in the perpendicular direction

Step 5 — Interpret the ResultThe singular value σ₁ = 3 tells us the maximum stretch. The singular value σ₂ = 1 tells us the minimum stretch. The ratio σ₁/σ₂ = 3 is called the condition number. A large condition number means the matrix treats different directions very differently.

Condition number = σ₁ / σ₂ = 3 / 1 = 3

SECTION 7

Applications, Strengths & Limitations

SVD is incredibly useful because it works on every matrix and reveals the most important structure. But like any tool, it has strengths and limitations. Let's compare.

Strengths vs. Limitations of SVD

Strengths	Limitations
Works on any m × n matrix — no restrictions	Computing SVD for very large matrices can be slow (though fast algorithms exist)
Gives the best low-rank approximation (Eckart–Young theorem)	The decomposition is not unique when singular values are repeated
Numerically stable — small errors in data cause only small errors in the SVD	Interpreting what the singular vectors mean in a specific context can be tricky
Reveals the rank, range, and null space of a matrix all at once	Does not preserve sparsity — even if A has lots of zeros, U and V usually don't

Real-World Applications

Image compression: A photo is a matrix of pixel values. Keeping only the largest singular values gives a compressed version that still looks great.
Recommendation systems: Netflix and Spotify use SVD-like methods to predict which movies or songs you'll enjoy based on patterns in millions of user ratings.
Search engines: Latent Semantic Analysis (LSA) uses SVD on a term-document matrix to understand the meaning behind search queries.
Noise reduction: In scientific data, noise often hides in the smallest singular values. Dropping them cleans the data.

✦ KEY TAKEAWAY

SVD is like an X-ray for matrices. Just as an X-ray reveals the bones beneath skin, SVD reveals the essential structure beneath a sea of numbers. The largest singular values are the 'bones' — the most important patterns — while the smallest ones are like background noise you can safely ignore.

SECTION 8

Connection to Other Decompositions

SVD is not the only way to break a matrix into simpler pieces. You may encounter other decompositions in future courses. Here's how SVD compares to two common alternatives.

SVD vs. Other Matrix Decompositions

Feature	SVD	Eigendecomposition	QR Decomposition
Works on	Any m × n matrix	Only square matrices (and not always)	Any m × n matrix
Factors into	U Σ Vᵀ	P D P⁻¹ (diagonal D)	Q R (orthogonal Q, upper-triangular R)
Best for	Rank, approximation, and data analysis	Understanding matrix powers and stability	Solving systems of equations quickly
Values always real?	Yes — singular values are real and non-negative	Not always — eigenvalues can be complex	Yes — entries are real

An important connection: if A is a symmetric matrix (meaning A = Aᵀ, like a correlation matrix), then its SVD and eigendecomposition are essentially the same thing. The singular values equal the absolute values of the eigenvalues. As you advance in linear algebra, you'll see that SVD is the most general of these decompositions — it always exists and always tells you something useful.

🚀 Looking Ahead

In college-level courses, you'll learn to compute SVD using eigenvalues of AᵀA and AAᵀ. You'll also encounter the truncated SVD — keeping only the top k singular values — which is the foundation of principal component analysis (PCA), one of the most important techniques in data science and machine learning.

SECTION 9

Practice Problems

PROBLEM 1 — CONCEPTUAL

In the SVD formula A = UΣVᵀ, what does the matrix Σ represent, and what is special about its shape?

PROBLEM 2 — BASIC CALCULATION

A matrix A has singular values σ₁ = 5 and σ₂ = 2. What is the condition number of A? What does this number tell you?

PROBLEM 3 — INTERMEDIATE

Suppose a 4 × 3 matrix A has singular values σ₁ = 7, σ₂ = 3, and σ₃ = 0. What is the rank of A? What are the dimensions of U, Σ, and Vᵀ?

PROBLEM 4 — APPLIED

A 1000 × 1000 grayscale image is stored as a matrix. Its SVD reveals that the first 50 singular values are large while the remaining 950 are tiny. If you keep only the first 50 terms of the outer-product expansion (A ≈ σ₁u₁v₁ᵀ + σ₂u₂v₂ᵀ + … + σ₅₀u₅₀v₅₀ᵀ), roughly how many numbers do you need to store instead of the original 1,000,000?

PROBLEM 5 — CRITICAL THINKING

A matrix B is formed by doubling every entry of matrix A (so B = 2A). How do the singular values and singular vectors of B relate to those of A? Explain your reasoning.

SUMMARY

Summary — Singular Value Decomposition

The Singular Value Decomposition (SVD) breaks any m × n matrix A into three pieces: A = UΣVᵀ. The matrix U provides an output rotation, Σ contains the singular values (the stretch factors, listed largest to smallest), and Vᵀ provides an input rotation. The number of non-zero singular values equals the rank of the matrix.

SVD is the most universal matrix decomposition — it always exists and works on every matrix. Its outer-product expansion lets you build the best possible low-rank approximation, which powers applications from image compression to recommendation systems to noise reduction. The condition number (σ₁/σₙ) measures how unevenly a matrix stretches space. As you continue studying linear algebra, you'll find SVD is the go-to tool for understanding what any matrix truly does.