Orthogonal Diagonalization - Linear Algebra

SECTION 1

Historical Context & Motivation

Have you ever tried to understand a complicated machine by taking it apart into simpler pieces? That is exactly the idea behind orthogonal diagonalization. Mathematicians wanted a reliable way to break a matrix—a grid of numbers that describes stretches, rotations, and reflections—into its simplest possible form. Over several centuries, brilliant thinkers developed the tools that make this possible.

Note for students: This lesson assumes you have already worked with matrices, vectors, and systems of linear equations in an introductory linear algebra course. You should be comfortable with matrix multiplication, solving systems of equations, and basic algebra (including the quadratic formula and the distance formula). We will define each new term—eigenvalue, eigenvector, symmetric matrix, and orthogonal matrix—carefully as we go, so read each definition before moving on.

1750s • Euler & Squared-Variable Equations

Leonhard Euler studied equations involving squared variables, such as 3x² + 5y² = 1. He discovered that rotating your coordinate axes—like tilting graph paper—could simplify these equations dramatically, eliminating messy cross terms.

1829 • Cauchy's Eigenvalue Theorem

Augustin-Louis Cauchy proved that every symmetric matrix (a matrix that mirrors across its main diagonal) has only real eigenvalues—special scaling numbers we will define shortly. This was a huge step toward orthogonal diagonalization.

1855 • Sylvester & the Spectral Theorem

James Joseph Sylvester formalized the idea that symmetric matrices can always be diagonalized using perpendicular (orthogonal) directions, laying the groundwork for the Spectral Theorem.

1900s • Modern Applications

Orthogonal diagonalization became essential in physics (vibrations, quantum mechanics), statistics (finding the most important directions in large data sets), and computer science (image compression, machine learning).

The central question these mathematicians pursued was: Can we always find a set of perpendicular directions along which a matrix simply stretches space, without any twisting or shearing? For symmetric matrices, the answer turns out to be a resounding yes.

SECTION 2

Core Principles & Definitions

Before we dive into orthogonal diagonalization, let's build up the key vocabulary one piece at a time. Each idea below is like a building block—you need all of them to see the full picture. Read every definition carefully; some of these terms may be new if this is your first pass through them in a linear algebra course.

Matrix

A matrix is a rectangular grid of numbers arranged in rows and columns, written inside brackets. For example, [[2, 1], [1, 2]] is a 2×2 matrix (2 rows, 2 columns). Matrices are used to represent transformations—ways of moving or reshaping space.

Eigenvalue & Eigenvector

When a matrix A is multiplied by a special nonzero vector v, the result is just a number λ times v—the vector doesn't rotate, only stretches or flips. That vector is called an eigenvector and the scaling number λ (the Greek letter 'lambda') is its eigenvalue. In symbols: Av = λv. Think of it as: the matrix multiplies the vector, but only stretches it—never tilts it off its line.

Symmetric Matrix

A symmetric matrix is one that looks the same if you flip it across its main diagonal (top-left to bottom-right). In symbols, A = Aᵀ, where Aᵀ means swapping every row with the corresponding column. Example: the entry in row 1, column 2 equals the entry in row 2, column 1.

Orthogonal Vectors

Two vectors are orthogonal if they meet at a 90° angle—like the x-axis and y-axis on a standard graph. We check this using the dot product: multiply matching components and add them up. Two vectors are orthogonal when their dot product equals zero.

Orthogonal Matrix

An orthogonal matrix P is a matrix whose columns are all unit-length (length = 1) and mutually perpendicular. A remarkable property: its inverse equals its transpose, so P⁻¹ = Pᵀ. Finding the inverse of a general matrix is hard, but for orthogonal matrices you simply flip rows and columns—much easier!

Diagonal Matrix

A diagonal matrix has nonzero entries only on its main diagonal—everywhere else is zero. For example, [[3, 0], [0, 5]] is diagonal. Diagonal matrices are the simplest to work with because they just scale each direction independently, with no mixing between directions.

✦

KEY TAKEAWAY

Think of a symmetric matrix like a trampoline that stretches in certain directions. Orthogonal diagonalization finds those special stretching directions (eigenvectors) that are perfectly perpendicular, and tells you how much each direction stretches (eigenvalues). It's like discovering that a trampoline's bounce can be described by just two perpendicular springs.

SECTION 3

Visual Explanation

The diagram below shows what happens when a symmetric 2×2 matrix acts on the unit circle. The circle gets stretched into an ellipse. The eigenvectors point along the axes of the ellipse—the directions of pure stretching with no rotation. The eigenvalues tell us how far the circle stretches along each eigenvector direction.

The cyan arrow (v₁) and the pink arrow (v₂) are the two eigenvectors. Notice they are perpendicular (orthogonal). When the matrix A acts on the circle, each eigenvector simply stretches by its eigenvalue—no rotation occurs along these special directions.

This is the heart of orthogonal diagonalization. Instead of looking at the complicated way a matrix transforms every point, we find the perpendicular eigenvector directions where the matrix simply stretches space. In those special coordinates, the matrix becomes diagonal—just the eigenvalues sitting on the main diagonal, with zeros everywhere else.

SECTION 4

Mathematical Framework

Now let's look at the formulas. Orthogonal diagonalization means writing a symmetric matrix A in a very specific way using three matrices multiplied together.

ORTHOGONAL DIAGONALIZATION

A = P D Pᵀ

A = the original symmetric matrix, P = an orthogonal matrix whose columns are the unit eigenvectors of A, D = a diagonal matrix with the eigenvalues of A on the diagonal, Pᵀ = the transpose of P (swap its rows and columns), which also equals P⁻¹ for orthogonal matrices.

EIGENVALUE EQUATION

det(A − λI) = 0

This is called the characteristic equation. Solving it gives you the eigenvalues (λ). Here I is the identity matrix (ones on the diagonal, zeros elsewhere—like multiplying by 1) and det means determinant. The determinant of a 2×2 matrix [[a, b], [c, d]] is computed as ad − bc—a single number that summarizes key information about the matrix.

FINDING EIGENVECTORS

(A − λI)v = 0

For each eigenvalue λ, plug it in and solve this system of equations to find the corresponding eigenvector v. Then normalize v—divide it by its length to make it a unit vector (length = 1)—to build the orthogonal matrix P. The length of a vector [a, b] is √(a² + b²), the same as the distance formula from geometry.

ℹ

The Spectral Theorem Guarantee

The Spectral Theorem promises that every symmetric matrix can be orthogonally diagonalized. You will always find real eigenvalues, and the eigenvectors from different eigenvalues will automatically be perpendicular. This guarantee does not hold for all matrices—only symmetric ones.

SECTION 5

The Step-by-Step Process

Here is a roadmap for orthogonally diagonalizing any symmetric matrix. Follow these steps in order, and you'll arrive at A = PDPᵀ every time.

Follow the flowchart from top to bottom. Start by checking symmetry (Step 1), find eigenvalues and eigenvectors (Steps 2–3), normalize them (Step 4), and assemble the final factorization (Step 5).

Step 1 — Check symmetry: Make sure A equals its transpose Aᵀ. To find Aᵀ, swap rows and columns: row 1 becomes column 1, row 2 becomes column 2, etc. If A ≠ Aᵀ, orthogonal diagonalization may not be possible.
Step 2 — Find eigenvalues: Compute det(A − λI) = 0. For a 2×2 matrix [[a, b], [c, d]], the determinant is ad − bc. Subtracting λ from each diagonal entry and computing the determinant gives a polynomial equation in λ; solving it (often by factoring or the quadratic formula) yields the eigenvalues.
Step 3 — Find eigenvectors: For each eigenvalue, solve (A − λI)v = 0. This means substituting λ back in, then finding vectors v = [x₁, x₂] that satisfy the resulting equations. You express one variable in terms of the other to get a general solution.
Step 4 — Normalize: Divide each eigenvector by its length to make it a unit vector (length = 1). The length of [a, b] is √(a² + b²), just like the distance formula. If two eigenvectors share the same eigenvalue, they must be made perpendicular to each other using an orthogonalization procedure before normalizing.
Step 5 — Assemble P and D: Place the unit eigenvectors as columns of P. Place the matching eigenvalues on the diagonal of D (with zeros elsewhere). Then A = PDPᵀ.

SECTION 6

Worked Example

Let's orthogonally diagonalize the symmetric matrix A = [[2, 1], [1, 2]]. We will follow every step of the process carefully, explaining each calculation along the way. This example applies familiar algebraic techniques—solving systems of equations, the distance formula, and polynomial factoring—within the matrix framework introduced in this lesson.

Orthogonally Diagonalize A = [[2, 1], [1, 2]]

1
Step 1 — Verify Symmetry

To find Aᵀ, swap the rows and columns of A: the first row [2, 1] becomes the first column, and the second row [1, 2] becomes the second column. So Aᵀ = [[2, 1], [1, 2]] = A. Check: the entry in row 1, column 2 is 1, and the entry in row 2, column 1 is also 1—they match. Since A equals its transpose, A is symmetric. We can proceed.

A = Aᵀ ✓
2
Step 2 — Find the Eigenvalues

We solve det(A − λI) = 0. The identity matrix I = [[1, 0], [0, 1]], so λI = [[λ, 0], [0, λ]]. Subtracting: A − λI = [[2 − λ, 1], [1, 2 − λ]]. Now compute the determinant using the rule for 2×2 matrices: det([[a, b], [c, d]]) = ad − bc. Here a = (2 − λ), b = 1, c = 1, d = (2 − λ). So det = (2 − λ)(2 − λ) − (1)(1) = (2 − λ)² − 1. Expanding (2 − λ)² = 4 − 4λ + λ², so det = 4 − 4λ + λ² − 1 = λ² − 4λ + 3. Factoring (find two numbers that multiply to 3 and add to −4: they are −1 and −3): (λ − 1)(λ − 3) = 0.

λ₁ = 1, λ₂ = 3
3
Step 3 — Find the Eigenvectors

For each eigenvalue, we substitute it back into (A − λI) and solve (A − λI)v = 0 for the vector v = [x₁, x₂]. This means finding all vectors that the modified matrix sends to zero. For λ₁ = 1: A − 1·I = [[2 − 1, 1], [1, 2 − 1]] = [[1, 1], [1, 1]]. The equation [[1, 1], [1, 1]][x₁, x₂] = [0, 0] gives x₁ + x₂ = 0 (both rows say the same thing). So x₂ = −x₁. Choosing x₁ = 1 gives v₁ = [1, −1]. For λ₂ = 3: A − 3·I = [[2 − 3, 1], [1, 2 − 3]] = [[−1, 1], [1, −1]]. The equation gives −x₁ + x₂ = 0, so x₂ = x₁. Choosing x₁ = 1 gives v₂ = [1, 1].

v₁ = [1, −1], v₂ = [1, 1]
4
Step 4 — Normalize the Eigenvectors

To normalize, divide each eigenvector by its length. Use the distance formula: length = √(a² + b²). Length of v₁ = [1, −1]: √(1² + (−1)²) = √(1 + 1) = √2. Dividing: û₁ = [1/√2, −1/√2]. Length of v₂ = [1, 1]: √(1² + 1²) = √2. Dividing: û₂ = [1/√2, 1/√2]. Quick orthogonality check using the dot product (multiply matching components and add): û₁ · û₂ = (1/√2)(1/√2) + (−1/√2)(1/√2) = 1/2 + (−1/2) = 0. A dot product of zero confirms the vectors are perpendicular!

û₁ = [1/√2, −1/√2], û₂ = [1/√2, 1/√2], dot product = 0 ✓
5
Step 5 — Build P and D

Place the unit eigenvectors as columns of P (first column = û₁, second column = û₂): P = [[1/√2, 1/√2], [−1/√2, 1/√2]]. Place the eigenvalues on the diagonal of D in the same order (λ₁ = 1 first, λ₂ = 3 second, zeros off-diagonal): D = [[1, 0], [0, 3]]. Now A = PDPᵀ. Since P is orthogonal, Pᵀ is found by simply swapping rows and columns of P: Pᵀ = [[1/√2, −1/√2], [1/√2, 1/√2]]. You can verify by multiplying P × D × Pᵀ step by step and checking that you recover the original matrix A = [[2, 1], [1, 2]].

P = [[1/√2, 1/√2], [−1/√2, 1/√2]], D = [[1, 0], [0, 3]]

SECTION 7

Strengths, Limitations & Comparisons

Orthogonal diagonalization is a powerful tool, but it doesn't apply to every matrix. Let's compare it with regular (non-orthogonal) diagonalization to see where each method shines.

Feature	Orthogonal Diagonalization	Regular Diagonalization
Which matrices?	Only symmetric matrices (A = Aᵀ)	Any matrix with enough linearly independent eigenvectors
Eigenvectors	Always perpendicular and unit length	May not be perpendicular
Inverse of P	P⁻¹ = Pᵀ (just swap rows and columns—very easy!)	Must compute P⁻¹ through a more involved algebraic process
Eigenvalues	Always real numbers (no imaginary parts)	May be complex (imaginary) numbers
Computational reliability	Very reliable in practice—orthogonal matrices preserve lengths and angles, so small rounding errors do not grow out of control	Can be unreliable for certain matrices where small errors in computation get amplified through the steps

Orthogonal vs. Regular Diagonalization

✦

KEY TAKEAWAY

Think of regular diagonalization as breaking a recipe into steps using any kitchen tools you can find—it works, but can be messy. Orthogonal diagonalization is like using a perfectly organized, matching set of tools: everything lines up cleanly, there's no overlap, and undoing any step is as simple as reversing it. The trade-off is that this only works for symmetric matrices—your 'matching set' is guaranteed to exist only for that special type.

SECTION 8

Connection to Advanced Theory

Orthogonal diagonalization is the starting point for many advanced topics you might see in later math, science, and engineering courses. Here's a preview of where these ideas lead. Each extension listed below builds directly on the skills from this lesson.

This Lesson	Advanced Extension
Orthogonal diagonalization of symmetric matrices	Singular Value Decomposition (SVD) — a generalization that works for any matrix, not just symmetric ones. SVD is the mathematical engine behind image compression (storing photos in fewer bits), internet search engines, and streaming service recommendation algorithms.
Eigenvalues describe stretching/compressing	Principal Component Analysis (PCA) — a statistical technique that uses eigenvalues to find the directions of greatest variation in a large data set. For example, given thousands of measurements about patients, PCA can identify the two or three most informative combinations of variables, reducing complexity while preserving the most important patterns.
Real symmetric matrices → real eigenvalues	Hermitian Matrices in Quantum Mechanics — in quantum physics, observable quantities (like energy or momentum) are represented by a complex-number generalization of symmetric matrices called Hermitian matrices. Just as symmetric matrices guarantee real eigenvalues, Hermitian matrices guarantee that measurement results are always real numbers.
A = PDPᵀ factorization	Classifying Curved Surfaces — expressions like 3x² + 2xy + 3y² describe geometric shapes (bowls, saddles, cones). The eigenvalues from orthogonal diagonalization reveal whether the surface curves upward, downward, or in mixed directions—crucial in optimization and machine learning for finding minima and maxima.

From Orthogonal Diagonalization to Advanced Topics

If you continue studying linear algebra, you'll see orthogonal diagonalization pop up again and again. Mastering it now gives you a strong foundation for these exciting advanced applications.

SECTION 9

Practice Problems

Question 1: CONCEPTUAL

Why can only symmetric matrices be orthogonally diagonalized? What special property of symmetric matrices guarantees that their eigenvectors are perpendicular?

Show answer

The Spectral Theorem guarantees that symmetric matrices (A = Aᵀ) have all real eigenvalues and that eigenvectors from different eigenvalues are automatically orthogonal. This happens because if Av₁ = λ₁v₁ and Av₂ = λ₂v₂ with λ₁ ≠ λ₂, then using the symmetry property, we can show λ₁(v₁ · v₂) = λ₂(v₁ · v₂), which forces v₁ · v₂ = 0 (they are perpendicular). Non-symmetric matrices may have complex eigenvalues or eigenvectors that are not perpendicular, so this clean decomposition breaks down.

Question 2: BASIC CALCULATION

Find the eigenvalues of the symmetric matrix A = [[5, 1], [1, 5]].

Show answer

Set up the characteristic equation: A − λI = [[5 − λ, 1], [1, 5 − λ]]. Use the 2×2 determinant formula det([[a, b], [c, d]]) = ad − bc with a = (5 − λ), b = 1, c = 1, d = (5 − λ): det = (5 − λ)(5 − λ) − (1)(1) = (5 − λ)² − 1 = 0. Expanding: 25 − 10λ + λ² − 1 = λ² − 10λ + 24 = 0. Factoring (find two numbers that multiply to 24 and add to −10: they are −4 and −6): (λ − 4)(λ − 6) = 0. The eigenvalues are λ₁ = 4 and λ₂ = 6. Notice both are real numbers, as the Spectral Theorem guarantees for symmetric matrices.

Question 3: INTERMEDIATE

Orthogonally diagonalize the matrix A = [[3, 1], [1, 3]]. Find P and D such that A = PDPᵀ.

Show answer

First, find eigenvalues: A − λI = [[3 − λ, 1], [1, 3 − λ]]. det = (3 − λ)(3 − λ) − (1)(1) = (3 − λ)² − 1 = λ² − 6λ + 8 = 0. Factoring: (λ − 2)(λ − 4) = 0, giving λ₁ = 2 and λ₂ = 4. For λ₁ = 2: A − 2I = [[1, 1], [1, 1]]. Equation x₁ + x₂ = 0 gives x₂ = −x₁; choose x₁ = 1, so v₁ = [1, −1]. Length = √(1² + (−1)²) = √2. Normalized: û₁ = [1/√2, −1/√2]. For λ₂ = 4: A − 4I = [[−1, 1], [1, −1]]. Equation −x₁ + x₂ = 0 gives x₂ = x₁; choose x₁ = 1, so v₂ = [1, 1]. Length = √2. Normalized: û₂ = [1/√2, 1/√2]. Orthogonality check: û₁ · û₂ = (1/√2)(1/√2) + (−1/√2)(1/√2) = 1/2 − 1/2 = 0 ✓. So P = [[1/√2, 1/√2], [−1/√2, 1/√2]] and D = [[2, 0], [0, 4]], giving A = PDPᵀ.

Question 4: APPLIED

A data scientist collects measurements in two variables and computes the covariance matrix C = [[4, 2], [2, 1]]. (A covariance matrix is always symmetric and summarizes how the variables vary together.) Orthogonally diagonalize C to find the principal directions and their variances (eigenvalues). Which direction has the most spread in the data?

Show answer

The covariance matrix C = [[4, 2], [2, 1]] is symmetric (the off-diagonal entries both equal 2). Find eigenvalues: C − λI = [[4 − λ, 2], [2, 1 − λ]]. det = (4 − λ)(1 − λ) − (2)(2) = 4 − 4λ − λ + λ² − 4 = λ² − 5λ = λ(λ − 5) = 0. Eigenvalues: λ₁ = 0 and λ₂ = 5. For λ₂ = 5 (the larger eigenvalue, meaning more spread): C − 5I = [[−1, 2], [2, −4]]. Equation −x₁ + 2x₂ = 0 gives x₁ = 2x₂; choose x₂ = 1, so v₂ = [2, 1]. Length = √(4 + 1) = √5. Normalized: û₂ = [2/√5, 1/√5]. For λ₁ = 0: C − 0·I = [[4, 2], [2, 1]]. Equation 4x₁ + 2x₂ = 0 gives x₂ = −2x₁; choose x₁ = 1, so v₁ = [1, −2]. Normalized: û₁ = [1/√5, −2/√5]. The direction with the most spread in the data is û₂ = [2/√5, 1/√5] because it corresponds to the larger eigenvalue (λ₂ = 5). This is the core idea behind a statistical method called Principal Component Analysis (PCA), which finds the most informative directions in data.

Question 5: CRITICAL THINKING

Consider the non-symmetric matrix B = [[0, 1], [−1, 0]]. Try to find its eigenvalues using det(B − λI) = 0. What happens, and why does this show that orthogonal diagonalization is special to symmetric matrices?

Show answer

The characteristic equation is: B − λI = [[0 − λ, 1], [−1, 0 − λ]] = [[−λ, 1], [−1, −λ]]. det = (−λ)(−λ) − (1)(−1) = λ² + 1 = 0. This gives λ² = −1, so λ = ±√(−1). Since no real number squares to −1, these eigenvalues are imaginary (written as ±i in mathematics). Since the eigenvalues are not real numbers, we cannot build a real orthogonal matrix P and orthogonal diagonalization fails. Geometrically, B represents a 90° rotation—every vector gets rotated, so no vector simply stretches along its original line. There are no eigenvectors in the real plane. This confirms why the Spectral Theorem requires symmetry: symmetric matrices guarantee real eigenvalues and perpendicular eigenvectors, making the clean A = PDPᵀ decomposition possible. Non-symmetric matrices like B can have imaginary eigenvalues and no real eigenvectors, preventing orthogonal diagonalization.

SUMMARY

Lesson Summary

Summary

Orthogonal diagonalization is the process of decomposing a symmetric matrix into the product A = PDPᵀ, where P is an orthogonal matrix of unit eigenvectors and D is a diagonal matrix of eigenvalues. The Spectral Theorem guarantees that this decomposition always exists for symmetric matrices, providing real eigenvalues and mutually perpendicular eigenvectors.

To carry out the process, you verify symmetry, compute eigenvalues from the characteristic equation, find and normalize the eigenvectors (divide by their length using the distance formula), and assemble P and D. The beauty of orthogonal diagonalization is that P's inverse is simply its transpose—just swap rows and columns—making calculations clean and reliable. This concept is foundational for many advanced applications in math, science, and engineering, including data analysis techniques like finding key directions in large data sets, image compression, and quantum mechanics.